Why captured subtitles don't line up: X-TIMESTAMP-MAP, stitched timelines, and four other causes

A subtitle file parses cleanly, every cue has plausible times, and the text still does not match what is on screen. Six distinct causes produce that symptom in HLS and DASH players. They have different fingerprints, and telling them apart quickly is most of the work.

Context: these come from building a browser extension that reads a video page's subtitle track, synthesizes speech from it locally, and plays it in sync. Everything below concerns clear-text caption payloads and manifest metadata that a player has already fetched in a normal session; audio and video streams are not involved.

1. The caption clock does not start at zero

Segmented WebVTT, the usual arrangement in HLS, carries a header:

X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:93900000

That relates cue time to the stream's 90 kHz MPEG-TS clock. Divide by 90,000 and the result is an offset in seconds. The trap: the offset is a position on the stream's own clock, not a value relative to the start of playback, and that clock can start anywhere.

On one stream it started at about 1,043 s. Treating the value as an absolute media offset put every cue roughly 17 minutes past the end of a video whose currentTime starts at zero. A renderer that only generates audio for cues near the playhead then produces nothing, with no error, indefinitely. Streams whose baseline happens to sit near zero work fine, which makes this look site-specific rather than systematic.

Symptom: the first cue starts implausibly far into a video that may be shorter than that, and nothing ever fires.

Approach that worked: re-base against the smallest timestamp map observed for the track rather than applying it as an absolute offset. The media's own initial timestamp is not readable from the caption side, so a relative baseline is the best available.

2. Content time versus presentation time

When extra segments are stitched into a stream server-side, the presentation timeline becomes longer than the content itself. Caption files are written in content time. After 120 s of inserted material, content time T is shown at presentation time T+120.

Symptom: alignment is fine at first, then jumps by a roughly fixed amount, and the error grows in steps rather than drifting smoothly. In DASH the structure is visible as multiple <Period> elements, and a piecewise mapping can be rebuilt from period starts and durations.

Second-order problem: some streams only reveal later insertions when the manifest refreshes, so a mapping computed once can be correct at first and wrong later. Alignment that degrades in steps over a long session is consistent with this, though a stale or wrong track can look similar.

3. Presentation time offset on the track

DASH @presentationTimeOffset on a text AdaptationSet shifts that track relative to its period. Standalone sidecar files are usually already period-relative and need no adjustment; packaged tracks frequently are not. Applying the offset to something already period-relative produces the same class of error in the opposite direction.

Symptom: a constant offset present from the first cue, affecting one track while another track on the same stream is fine.

4. The captions were fetched before anyone was watching, or are not being fetched at all

Two behaviors that look identical to "nothing is working":

• Lazy fetching. Manifests and caption tracks are commonly fetched at first play, on a seek, or on a manifest refresh, rather than at page load. Observing for a couple of seconds after load and concluding nothing was fetched is a mistake that is easy to make and expensive to act on: the natural next hypothesis is that the player fetches captions somewhere unobservable, which sends the investigation in the wrong direction.
• Application-level caching. A player that fetched the file once may serve it from memory afterwards, including across in-page navigations. On a rewatch there may be no network activity at all. A reload usually forces a refetch; toggling captions off and on often does not, because the file is already in hand.

Symptom: works after a reload and not otherwise, or works on a first watch and never on a second.

A third variant belongs here, because it looks the same from the outside: the track was fetched and you saw it, but you failed to match it to the manifest. Caption sidecars are often served from a different host than the manifest that lists them, so pairing by URL finds nothing. The reliable signal is the payload itself. TTML declares xml:lang; WebVTT can carry a Language: header. Trust the body's own declaration over any URL correspondence.

5. One URL, several different payloads

Packaged tracks are often delivered as byte-range requests against a single URL, and manifests get refetched at the same address. Any store keyed by URL alone keeps only the last body, so segments are lost silently and the result is one time window instead of a full track.

Symptom: far fewer cues than the video's length implies, and the cues that exist cover one contiguous stretch.

Approach that worked: key by something other than the bare URL for these, and accumulate rather than replace. A counter for "a distinct body replaced another at this key" is worth adding, because otherwise the loss leaves no trace.

6. Arrival order

Subtitles packaged in fragmented MP4 arrive as an init segment (metadata, no samples) plus media segments (samples, no metadata). Both are needed to extract anything, and the init does not reliably arrive first.

An extractor that combines a media segment with the init only when the media arrives afterwards will work sometimes. Two captures of the same video produced a full track and nothing at all, with arrival order the only difference; the successful one was luck.

Symptom: intermittent, video-dependent, and not reproducible on demand.

Approach that worked, and the general lesson: buffer what arrives early and retry it when its counterpart lands. Then test by permuting arrival order, since that is the reliable way to surface this class. A property test that shuffles the segment stream and asserts the extracted track contains every window catches it. The same technique surfaced an unrelated ordering bug in a playback scheduler that by-hand testing had not produced, and which failed on 1,645 of 2,000 random orderings once the test existed.

Diagnosing quickly

One more sequencing detail, since it produces a symptom that mimics several of the above: sniff the format from the content rather than the extension or content type, and check the formats that begin with [ or { (ASS, LRC, MicroDVD) before any generic JSON handling. Otherwise they are classified as JSON, parse to nothing, and present as "the track is there and empty".

Two things that made all six tractable

Make the idle path report itself. A renderer that produces nothing should say why: how many cues it considered, how many were ahead of the playhead, how many behind, where the playhead was, where the first cue starts. Success paths log themselves and errors throw; the paths that stay silent are the ones that are waiting, and in event-driven media code that is the common state. One line of that output identified cause (1) directly:

attempted: 0, ahead: all, behind: 0, playheadSec: 35, firstCueSec: 1043

Make a correction reach every consumer of the timeline. A manual sync control that computed the right offset to the second still produced no audio, because the correction re-timed playback of already-generated clips while the component deciding what to generate still compared raw cue times against the raw playhead. A correction applied to half the pipeline is hard to distinguish from no correction, and it tends to send the investigation back to the offset itself, which was not the problem.