Audio Realism Benchmark

The fundamental unit of scoring is the pairwise comparison. A listener is presented with two clips of the same prompt from two models, blind to their identity and with left/right order randomized, and selects the one that sounds more human. Each selection constitutes a single vote.

These votes are aggregated with the Bradley-Terry model, the maximum-likelihood method behind Elo-style ratings, which resolves every head-to-head win and loss into a single Elo rating per model. A higher rating means the model is chosen as more human more often.

For each prompt we also record a human reading it and enter that clip into the pool as a human baseline. Every model is then measured directly against that baseline: how often it is picked over a real person.

The corpus is a fixed, versioned set of naturalistic prompts, most up to roughly 40 words (about 15 seconds), though phone-agent prompts are single turns and tend to be shorter. They are organized into three buckets defined by speech register (how someone is talking) rather than topic; within each bucket we deliberately vary the subject matter. Because a bucket follows how something is spoken rather than what it is about, one topic can span more than one bucket: a healthcare call to book an appointment is a phone agent, while a walkthrough of what a medication does is an explainer.

Within each bucket, prompts are stratified across lexical difficulty, quantified by language-model perplexity and spanning roughly an order of magnitude, so each system is evaluated on both smooth, predictable speech and harder, less predictable passages that are more difficult to voice naturally.

Acting-heavy speech (game characters, dramatized fiction, persona-driven companion voices) is out of scope for this version, deferred to a future expressive track. Transcripts keep the hesitations, false starts, and fillers of real speech wherever the register allows. Numbers, times, and amounts are written as spoken words rather than digits or symbols, so the transcripts read exactly as a model is expected to voice them.

The scored corpus is a held-out set of 500 prompts, kept fully private and never published. The examples below are a separate public set, in the same style and from the same sources.

Transcripts are drawn from established, permissively licensed spoken-language datasets, not written for this benchmark:

No transcript is used verbatim. Before a passage enters the corpus we swap its entities and topic, changing the names, numbers, places, and subject matter so nothing is searchable back to its source. How it is spoken is preserved: the disfluencies and register stay intact, and only the surface content changes.

We distinguish disfluencies from transcription errors. Authentic disfluencies (fillers, restarts, self-corrections) are retained verbatim, as reproducing them is what the benchmark measures. Transcription noise the speaker never produced, such as non-speech tokens (for example [noise], [cough], <unk>) and garbled fragments introduced during transcription, is not carried into prompts.

Numerics are normalized to spoken form: times, currency, counts, years, and decimals are written as words (for example, "four fifteen p.m." rather than "4:15 PM"), never as digits or symbols.

Every prompt is manually reviewed before it enters the corpus, and in the same pass a native speaker records a human reading of it. That recording serves as the human anchor in scoring and confirms each prompt is natural to speak aloud.

Most providers ship many voices, so the voice can matter as much as the model. For each model we use one female and one male voice, chosen as its most naturally conversational American-English voice. Candidates must be the right gender and American-accented; among those we take the voice the provider’s docs describe as most conversational — preferring, in order, conversational/voice-agent → warm/natural → casual → neutral.

For gender, each matchup draws a random prompt and a random gender, and both models (and the human anchor) speak it with their selected voice of that gender. Every comparison is same-gender, so gender is never the difference between the two clips.

We want the benchmark to represent the text-to-speech systems people are choosing between, and we add new models and providers on an ongoing basis. We generally prioritize systems with meaningful adoption or strong realism, but this is an open, evolving process and we genuinely welcome feedback, suggestions, and new models to consider.