How far synthesized speech overruns subtitle timing: 150 measured cues

Speech generated to land on someone else's timeline (dubbing, narration over video, an accessibility layer) does not fit the slots it is given. How badly is a question with a measurable answer, and the answer shapes the design: here it is for 150 cues of a real broadcast caption track.

Context: these come from a browser extension that reads a video page's subtitle track, synthesizes speech from it locally, and plays it in sync.

Setup. Kokoro-82M v1.0 ONNX at natural pace, one voice (af_heart), against the caption track of NASA Science Live: Aurora Glow, Electric Flow & the EZIE Mission (Episode 63, March 24, 2025; listed with a download link on NASA's episode archive, and mirrored on NASA's YouTube channel), a U.S. Government work in the public domain under 17 U.S.C. §105, 1,326 cues total, 150 sampled evenly across a dense region. MacBook Pro 16-inch 2021 (M1 Max, 64 GB, macOS 26.5.2), onnxruntime-node, August 2026. Raw data: cue-fit.csv; the caption file itself is nasa-captions.vtt.

The distribution

For each cue: its caption window, the gap from its onset to the next cue's onset, and how long the model takes to speak it at natural pace.

Underlying rates: the original speakers delivered a median 17.4 characters per second, the model 13.8. About 21% slower, and that difference lands hardest on lines that were already tight.

Two consequences. Most lines need some compression, so fitting is part of the main path rather than an edge case. And the tail is the constraint: the 4% needing more than 2.5× cannot be handled by asking the model to speak faster, because its speed parameter saturates near 2.2×. Those lines have to be time-stretched after synthesis, allowed to overrun, or cut.

Two limits worth stating. This track is English captions spoken by an English voice, so it measures synthesis pace against speaker pace and contains no translation expansion. And the caption track appears to be machine-generated rather than authored by a captioner: it preserves disfluencies ("it's it's"), mis-transcribes names and words that a human would have caught, and its cue boundaries are frame-contiguous and often fall mid-phrase. That matters for these numbers, because cue boundaries are exactly what sets each line's budget. A professionally captioned track would likely have cleaner, longer cues and a gentler distribution; the 0.37-second slot for the words "and you are" is an artifact of automatic segmentation, not a captioner's decision. A system translating into a wordier target language should expect a worse distribution. These numbers also come from one track, one voice, and one model, so the percentages are indicative rather than universal. The shape (most lines need a little, a small tail needs more than the model can give) is the part likely to generalize.

The budget is the next onset, not the cue window

A natural first implementation compresses each line to fit its own caption window. That is the wrong budget, though on this track it errs in the generous direction.

The budget that matters is nextCue.startMs - thisCue.startMs, the gap to the next line's onset, because when only one voice plays at a time the next line's onset is when the current one stops. Whether that gap is longer or shorter than the cue's own window depends on the source. Where captions overlap, as rolling and some SDH tracks do, the gap is shorter and fitting to the window lets the tail get chopped. On this broadcast track the opposite holds: not one of its 1,326 cues overlaps the next, so every gap is at least as long as its window and fitting to the gap gives lines strictly more room. That shows up directly in the distribution, where 76.7% of lines overrun their window but only 67.3% overrun the real budget.

Worth noting because it is easy to assume otherwise: caption tracks rarely contain overlapping cues even when the people on screen talk over each other, because captioners serialize simultaneous speech into consecutive cues. A dubbing system's collisions therefore come almost entirely from synthesized speech overrunning its slot, not from the source timing being ambiguous.

What the overlap policies sound like

Six consecutive captions from the densest stretch of the track (34.9 characters per second against a 17.4 median), two voices, one timeline, rendered three ways. This is where collisions are worst, and it is a normal passage of an ordinary broadcast, not a contrived one. Per-clip numbers are in overlap-demo.csv:

Clips 3 and 4 are the extreme case: 1.73 seconds of speech due in a 0.57-second slot, three times more audio than there is room for.

Everything plays in full — every transition collides, so two voices talk at once for over a second at a time: overlap-naive.mp3

Each clip is cut when the next begins — one voice at a time, but line endings are lost: overlap-cut.mp3

Each clip is compressed to its budget first, then cut if still needed — what shipped: overlap-fitted.mp3

Every sample on this page is also available uncompressed: same path,.wav instead of.mp3 (24 kHz mono). The mp3s are 96 kbps, which is transparent enough for pace and pronunciation but not necessarily for the finer comparisons above, so the originals are there to check against.

The middle option is what a straightforward "stop the previous source" implementation produces, and it sounds worse than the overlap numbers suggest, because the words removed are sentence endings, which carry the grammar. The third is not free either: on the longer lines the compression is audible.

Ordering matters more than it looks

"Cut the previous clip when the next starts" assumes clips arrive in timeline order. They do not. A playback-speed change causes upcoming lines to be re-synthesized for the new window size, and a speed slider emits several intermediate values, so the same cues come back repeatedly and out of order. An older clip arriving after a newer one slips past a rule that tracks a single "previous" source and plays underneath it, producing the two-voice artifact from an entirely different cause.

The fix is to compare intervals rather than track one previous clip, and to drop a superseded clip rather than start it. The durable part is the test: a property test that drives the scheduler with randomly ordered arrivals and asserts no two started clips ever overlap. Against the pre-fix implementation it fails on 1,645 of 2,000 random orderings, a failure rate that by-hand testing had not produced, since a person exercising the feature tends to generate events in order.

The same technique later caught a subtitle extractor that only worked when a container's init segment happened to arrive before its media segments. Two captures of the same video, one complete and one empty, with arrival order the only difference. Permutation testing is now routine for anything that accumulates streaming events.

The compression order

The pipeline that came out of this data:

1. Estimate speech length before synthesizing (characters per second, calibrated per voice, since voices differ by up to 67% in pace for identical text).
2. Ask the model to speak faster for as much of the compression as it will actually deliver, capped near its saturation point.
3. Time-stretch the remainder with a pitch-preserving stretch. Pitch preservation matters because playback-rate changes are common and a rate-shifted clip sounds wrong next to video whose own audio keeps its pitch.
4. Cut on the next onset as a last resort, having made it rare.

Steps 2 and 3 are in that order because faster synthesis appeared to hold up better than post-hoc stretching at the ratios this content needs. That is a judgment about how two artifacts sound, so it is worth forming your own: matched-ratio samples of both are published alongside the voice measurements, along with the caveat that the pairs are not exactly duration-matched at the top of the range.

Step 2 carries a separate trap worth repeating: the model accepts any speed value but delivers saturating compression. Passing the requested rate to step 3 rather than the measured clip length makes step 3 under-stretch, and the line overruns anyway.

Reproducing this

The model is onnx-community/Kokoro-82M-v1.0-ONNX (Apache-2.0); the captions are from the episode linked above. The measurement is straightforward to restate:

1. Parse the caption track and keep cues from a dense region (skipping sparse intros, which distort inter-onset gaps).
2. Sample evenly across that region rather than taking a contiguous block, so one dense passage does not dominate.
3. For each sampled cue, synthesize its text at natural pace and record: the cue window, the gap to the next cue's onset, and the duration of the generated audio.
4. Report the ratio of speech duration to each of those budgets as a distribution, not a mean. The mean is not the interesting part; the tail is.

Data: cue-fit.csv · overlap-demo.csv · summary.json