Video Containers, in depth: what MP4, Matroska/WebM and MPEG-TS store, how timestamps work, and how to remux without breaking playback

The jobs every container does

Whatever the format, a container has to answer the same questions for a player:

• Track list. How many streams, of which type, in which language, and which are default.
• Codec configuration. The parameters needed to initialise a decoder: for H.264, the sequence and picture parameter sets; for AAC, the AudioSpecificConfig; for Opus, its identification header.
• Sample boundaries. Where each compressed frame starts and ends.
• Timing. A timescale and, for each sample, a decode time and a presentation time.
• Random access. Which samples are keyframes and where they sit in the file, so seeking does not require reading from the start.
• Interleaving. Audio and video stored close together in time so a player streaming the file does not need two read positions far apart.
• Metadata. Rotation, colour information, chapters, subtitles and tags.

Formats differ mostly in where they put the index and how much of it must be read before playback can begin. That single design choice explains most of their strengths and failure modes.

MP4 and the ISO base media file format

MP4 is a profile of the ISO base media file format (ISO BMFF), which also underlies MOV, 3GP, CMAF and HEIF images. The file is a sequence of boxes, each beginning with a 32-bit size and a four-character type. ftyp declares brands the file conforms to. mdat holds the compressed samples as raw bytes. moov holds everything else: one trak per stream, and inside each, the sample table.

The sample table is a set of compact run-length tables: stsd carries the codec configuration as a sample entry, stts the decode-time deltas, ctts the offsets from decode to presentation time, stsz the sample sizes, stsc and stco (or co64 for large files) map samples to byte offsets, and stss lists keyframes. An edit list in edts can shift or trim the timeline, typically to hide encoder delay at the start.

Because a recorder only knows the sizes and offsets once all samples are written, moov is usually written at the end. A player that streams such a file over HTTP must fetch the end first, or wait for the whole download. Moving moov to the front, known as faststart, fixes this, and every MP4 served for progressive download should have it. A short script shows where the boxes are:

import struct, sys

def top_level_boxes(path):
    with open(path, "rb") as f:
        offset = 0
        while True:
            header = f.read(8)
            if len(header) < 8:
                return
            size, kind = struct.unpack(">I4s", header)
            if size == 1:                      # 64-bit size follows the type
                size = struct.unpack(">Q", f.read(8))[0]
            elif size == 0:                    # box runs to end of file
                print(offset, kind.decode("latin-1"), "to EOF")
                return
            print(offset, kind.decode("latin-1"), size)
            offset += size
            f.seek(offset)

top_level_boxes(sys.argv[1])

If the output lists mdat before moov, the file is not faststart. The sample entry type matters too: HEVC can be signalled as hvc1, with parameter sets only in the sample entry, or hev1, which allows them in the stream. Apple's players expect hvc1, which is why ffmpeg users add -tag:v hvc1 when targeting Safari. For streaming, ISO BMFF is fragmented into moof and mdat pairs; fragmented MP4 internals covers that layout byte by byte.

Matroska and WebM

Matroska is built on EBML, a binary format of nested elements, each with a variable-length ID and size. A file starts with an EBML header naming the document type, then a Segment containing Info, Tracks (with each codec's private configuration data), a sequence of Clusters, and usually Cues, the seek index. Each Cluster carries a base timestamp and a run of SimpleBlocks or BlockGroups, each holding a frame and a small timestamp offset relative to the cluster.

The cluster structure makes Matroska robust. A recording that stops abruptly still holds complete clusters a player can decode, and only seeking suffers until the Cues are rebuilt. That is why many recording tools favour MKV. Matroska also accepts nearly any codec, multiple audio and subtitle tracks, chapters and attachments such as fonts, which makes it popular for archives and media libraries.

WebM is a restricted Matroska profile defined for the web, with the document type webm. It allows VP8, VP9 and AV1 video and Vorbis and Opus audio, and browsers that support WebM handle it natively. If you need H.264 or AAC, WebM is the wrong container.

MPEG transport stream

MPEG-TS was designed for broadcast, where receivers tune in at arbitrary moments over lossy links. The stream is a sequence of 188-byte packets, each starting with the sync byte 0x47 and carrying a 13-bit packet identifier (PID). The Program Association Table on PID 0 lists programs; each program's Program Map Table lists its elementary streams, their types and PIDs. Audio and video are carried as PES packets split across TS packets, and PES headers carry PTS and DTS values in 33 bits at 90 kHz, which wrap roughly every 26.5 hours. A Program Clock Reference at 27 MHz lets receivers lock their clocks to the sender.

There is no global index. The tables repeat every fraction of a second, so a decoder can start anywhere it finds a PAT, a PMT and a keyframe. The price is overhead: four-byte headers on every packet, padding, repeated tables and PES headers, often a few percent, more at low bitrates. TS remains the standard for broadcast, IPTV and contribution links, and was HLS's original segment format, though modern HLS and DASH increasingly use fragmented MP4; HLS vs DASH in depth covers that shift.

Timestamps, PTS and DTS

Every container records time as integers in a timebase: ticks per second set per track in MP4, a TimestampScale in nanoseconds in Matroska (one millisecond by default), and a fixed 90 kHz in TS. Converting between them rounds, and careless rounding accumulates drift.

Codecs with B-frames decode frames out of display order: a frame that displays later may be needed to predict the ones before it. The decode timestamp (DTS) orders decoding and the presentation timestamp (PTS) orders display. In MP4, stts gives decode times and ctts the presentation offsets; TS carries both in PES headers; Matroska stores presentation times and leaves decode order to the block sequence. Variable frame rate video, common from phones and screen capture, has irregular timestamps, and tools that assume a constant frame rate when remuxing it will desynchronise audio.

Inspecting the timeline

When playback misbehaves, look at the packets rather than the summary. ffprobe can list every packet of a stream with its timestamps, size and flags, and a few lines of Python turn that into a check for keyframe spacing and timestamp gaps:

import json, subprocess, sys

out = subprocess.run(
    ["ffprobe", "-v", "error", "-select_streams", "v:0",
     "-show_entries", "packet=pts_time,dts_time,flags", "-of", "json", sys.argv[1]],
    capture_output=True, text=True, check=True).stdout
packets = json.loads(out)["packets"]

keys = [float(p["pts_time"]) for p in packets if "K" in p["flags"]]
gaps = [b - a for a, b in zip(keys, keys[1:])]
print("keyframes", len(keys), "max interval", round(max(gaps), 2), "s")

dts = [float(p["dts_time"]) for p in packets if p.get("dts_time")]
jumps = [(a, b) for a, b in zip(dts, dts[1:]) if b <= a or b - a > 1.0]
print("non-increasing or jumping DTS:", jumps[:5])

Long keyframe intervals explain slow, imprecise seeking and segmenters that cannot cut where they need to. Decode timestamps that go backwards or jump usually come from concatenated recordings or a broken muxer, and should be fixed before the file is packaged for streaming.

Choosing a container

Worked example: preparing a recording for the web

You have talk.mkv from a screen recorder and want a file that starts playing immediately in every browser. Inspect it first:

$ ffprobe -v error -show_entries stream=index,codec_type,codec_name,profile,r_frame_rate,avg_frame_rate -of compact talk.mkv
stream|index=0|codec_name=h264|profile=High|codec_type=video|r_frame_rate=60/1|avg_frame_rate=60/1
stream|index=1|codec_name=aac|profile=LC|codec_type=audio|r_frame_rate=0/0|avg_frame_rate=0/0

H.264 and AAC are both valid in MP4, so no re-encoding is needed. Remux, copying the bitstreams and asking for faststart:

$ ffmpeg -i talk.mkv -map 0:v:0 -map 0:a:0 -c copy -movflags +faststart talk.mp4
$ python boxes.py talk.mp4
0 ftyp 32
32 free 8
40 moov 41203
41243 mdat 512339871

The remux takes seconds because nothing is decoded. Listing shows moov before mdat, so playback can begin after the first few kilobytes; the offsets shown are illustrative. Had the audio been Vorbis or the video VP9 with a Safari target, you would need to transcode that stream, for example -c:v copy -c:a aac -b:a 160k, and only that stream. Finally, compare durations of both files with ffprobe -show_entries format=duration and play the end of the output to confirm audio and video still line up.

Failure modes

What to do next

1. Run the box lister on files your service ships and confirm moov precedes mdat.
2. Use ffprobe to list codec, profile and frame rate for every input before deciding to remux or transcode.
3. Default to stream copy; transcode only the streams the target container or device cannot take.
4. Record live sources to MKV or fragmented MP4, then remux to faststart MP4 for distribution.
5. Tag HEVC as hvc1 when Apple playback matters.
6. For streaming, move to fragmented MP4 segments; read the CMAF architecture guide.
7. Add an automated check comparing input and output durations and stream counts after every remux.