How to evaluate Arabic call transcription and analysis

Build a test set from your own calls

A demo call is recorded to look good. Your calls are not. Build a small test set from real recordings your team made, chosen to cover the conditions that trip transcription up:

• Every dialect you serve, for example Egyptian, Saudi and Emirati callers, plus any calls in English.
• Every recording setup you use, since systems differ: calls from your phone system, recordings from mobile phones, and uploaded files.
• Noisy lines: a caller in a car, a speakerphone, a busy showroom.
• Calls full of names, numbers, prices, dates and product terms, where English words sit inside Arabic sentences.
• At least one long call, since problems with timing often show up late in a recording.

Check the words that matter

Play each call and read along. Do not try to grade every word the same. A wrong filler word costs nothing, while a wrong number, a misheard product name or a misspelled customer name can send a follow-up to the wrong place.

Mark the errors that would change a decision: amounts, dates, names, commitments, and any sentence where negation was lost, since a missed word such as not or never can reverse the meaning. Word error rate is a useful research metric, but on business calls the errors you mark by hand tell you more.

Check who said what

Speaker attribution, often called diarization, decides whether every measure downstream is right. Pick moments where the speakers change quickly, where one interrupts the other, or where a third person joins, and check that each line is given to the right person.

Check whether speakers are named or only numbered. A report that says speaker 2 made a promise still leaves a manager guessing. In Callens every measure is attributed to a named person; speakers are assigned by who greets like an agent, and when the roles come out wrong, one click corrects them. Check how any tool you test assigns roles, and how quickly a mistake can be fixed.

Check the timestamps

A cited moment is only useful if it plays the right sentence. Click several timestamps early, in the middle and near the end of each long call, and check that the audio starts on the words the transcript shows. Timing that drifts over a long call makes every citation after it less trustworthy.

Callens cites the second of the recording behind every claim, and its 22 conversation measures are computed from word-level timings. That makes timestamp accuracy something to verify directly, not take on trust.

Check summaries and judgments against the recording

This is where analysis tools differ most. For each summary line and each judgment, such as the budget, the decision maker or the outcome, find the moment in the recording that supports it. Mark any line you cannot find, and any line that states as fact something the customer only hinted at.

In Callens, every model-produced line is checked against a real segment of the transcript before it is stored, and summaries arrive in the language of the call with an English rendering beside them. Read both versions: the English should say what the Arabic says, not a smoother version of it.

Record the results and decide

Keep a simple sheet per call: the dialect, the line type, the errors that would change a decision, speaker mistakes, timestamps that missed, and summary lines you could not find in the audio. Compare tools on the same calls, and involve a team leader who knows the customers, since they will spot a wrong name or product faster than anyone.

To run this test on Callens, recordings can be uploaded in the browser as MP3, M4A, AAC, WAV, WebM, OGG or FLAC, up to 200 MB and 60 minutes each. Any call exports as a PDF report, with or without the transcript, which makes side-by-side review easier.