Video Streaming Architecture in Depth

The pipeline end to end

The diagram has a write path and a read path. The write path runs once per asset (or per segment, for live): contribution brings the source in, the transcoder produces a ladder of renditions, the packager wraps them into segments and manifests, and the origin stores the result. The read path runs once per viewer request: the player asks an edge server for a manifest or a segment, the edge answers from cache or asks a shield, and only the shield's misses reach the origin.

Two properties make this work. Every response is a cacheable HTTP object, so standard CDNs can serve it. And the client holds all session state, so any edge can serve any request and a failed server costs a retry rather than a session. Personalised manifests, per-user tokens and ad insertion all chip away at those properties, so each needs a deliberate answer for how the cache key stays small.

The pipeline end to end

The diagram has a write path and a read path. The write path runs once per asset (or per segment, for live): contribution brings the source in, the transcoder produces a ladder of renditions, the packager wraps them into segments and manifests, and the origin stores the result. The read path runs once per viewer request: the player asks an edge server for a manifest or a segment, the edge answers from cache or asks a shield, and only the shield's misses reach the origin.

Two properties make this work. Every response is a cacheable HTTP object, so standard CDNs can serve it. And the client holds all session state, so any edge can serve any request and a failed server costs a retry rather than a session. Personalised manifests, per-user tokens and ad insertion all chip away at those properties, so each needs a deliberate answer for how the cache key stays small.

Ingest and contribution

Contribution is the first mile: getting the source to your encoders. For on-demand (VOD) content it is an upload of a high-bitrate mezzanine file, kept forever because every re-encode starts from it. For live content it is a continuous stream. RTMP over TCP is still everywhere because every encoder and streaming app speaks it, but it suffers on lossy long-haul links. SRT runs over UDP with retransmission inside a fixed latency window and optional encryption, which is why broadcasters prefer it for contribution across the public internet.

Treat ingest as a reliability boundary: run endpoints in more than one region, accept primary and backup feeds for important events, and validate frame rate, resolution and audio layout at the door.

Transcoding and the encoding ladder

The transcoder turns one source into an encoding ladder: the same content at several resolution and bitrate pairs. The table is an illustrative H.264 ladder, not a recommendation; the right rungs depend on your content and devices. Per-title encoding moves the rungs per asset, so a static lecture gets a cheaper ladder than fast sports at the same perceived quality (measured with a metric such as VMAF).

The single most important encoder setting for streaming is GOP alignment. A player can only switch renditions at a segment boundary, and each segment must start with a keyframe that is independently decodable. So every rendition must place keyframes at exactly the same timestamps, at a fixed interval that divides the segment duration, with scene-cut keyframe insertion disabled or constrained.

Codec choice is a cost trade-off. H.264 plays everywhere. HEVC and AV1 reach the same quality at materially lower bitrates, which cuts egress for the long tail of viewing, but cost far more compute to encode and are not decodable on every device, so most services encode them in addition to H.264 rather than instead of it. Here is an FFmpeg sketch that produces a three-rung fMP4 HLS ladder with fixed two-second GOPs at 24 frames per second:

ffmpeg -i mezzanine.mov \
  -filter_complex "[0:v]split=3[a][b][c];[a]scale=1920:1080[v0];[b]scale=1280:720[v1];[c]scale=640:360[v2]" \
  -map "[v0]" -c:v:0 libx264 -b:v:0 5000k -maxrate:v:0 5350k -bufsize:v:0 7500k \
  -map "[v1]" -c:v:1 libx264 -b:v:1 2800k -maxrate:v:1 3000k -bufsize:v:1 4200k \
  -map "[v2]" -c:v:2 libx264 -b:v:2 800k  -maxrate:v:2 856k  -bufsize:v:2 1200k \
  -map a:0 -map a:0 -map a:0 -c:a aac -b:a 128k -ac 2 \
  -g 48 -keyint_min 48 -sc_threshold 0 \
  -f hls -hls_time 4 -hls_playlist_type vod -hls_segment_type fmp4 \
  -master_pl_name master.m3u8 \
  -var_stream_map "v:0,a:0 v:1,a:1 v:2,a:2" out_%v/index.m3u8

-g 48 -keyint_min 48 -sc_threshold 0 forces a keyframe every 48 frames and nowhere else, so each four-second segment holds exactly two GOPs in every rendition; maxrate keeps peaks close to the bandwidth the manifest advertises.

Packaging: HLS, DASH and CMAF

Packaging decides how segments are wrapped and described. HLS describes renditions in text playlists: a multivariant (master) playlist lists the renditions, and each media playlist lists its segments. MPEG-DASH describes the same thing in one XML manifest, the MPD. CMAF standardises fragmented MP4 segments that both formats can reference, so one set of media files serves Apple devices through HLS and everything else through DASH, and the CDN caches a single copy. A minimal pair of HLS playlists looks like this:

# master.m3u8 -- one entry per rung; the player reads BANDWIDTH and CODECS
#EXTM3U
#EXT-X-VERSION:7
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-STREAM-INF:BANDWIDTH=5600000,AVERAGE-BANDWIDTH=5200000,RESOLUTION=1920x1080,CODECS="avc1.640028,mp4a.40.2"
out_0/index.m3u8

# out_0/index.m3u8 -- the init segment, then numbered media segments
#EXTM3U
#EXT-X-VERSION:7
#EXT-X-TARGETDURATION:4
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-MAP:URI="init_0.mp4"
#EXTINF:4.000,
data00000.m4s
#EXT-X-ENDLIST

Segment duration is the main packaging knob. Short segments let the player switch quality and start up sooner and reduce live latency, but they add requests, manifest churn and per-segment overhead, and because each segment starts with a keyframe they cost compression efficiency. Durations of two to six seconds cover most deployments. Just-in-time packaging at the origin saves storage when you support many formats and DRM schemes, at the cost of origin compute.

Origin, shield and CDN caching

The CDN is where the economics are decided. Each edge point of presence caches what its nearby viewers request. An origin shield (a mid-tier cache) sits between all edges and the origin, so a miss at fifty edges becomes at most one request to the origin. Shields should also collapse concurrent requests: when a thousand viewers ask for the same new live segment in the same instant, the shield sends one upstream request and holds the rest until it returns.

Objects that never change get long lifetimes; objects that change get short ones.

A worked example shows why origin load is decoupled from audience size. Take a live event with five video renditions plus one audio rendition, two-second segments and ten shield nodes. Each shield fetches each new segment of each rendition once, so the origin sees roughly 10 x 6 / 2 = 30 segment requests per second, plus playlist refreshes, whether one thousand or one million people are watching. The edge, by contrast, carries the whole audience: one million viewers averaging 3 Mbit/s is 3 Tbit/s, which is why large events are spread across several CDNs. Multi-CDN setups steer viewers by DNS, by manifest rewriting, or with content steering, where the player fetches a steering manifest and can switch CDNs mid-session.

The player and adaptive bitrate

The player is where adaptivity lives. It fetches the manifest, picks a starting rendition, downloads segments into a buffer, and feeds them to the decoder: through Media Source Extensions in browsers (hls.js, dash.js, Shaka Player), natively through AVPlayer on Apple platforms, and through ExoPlayer (now part of Media3) on Android. After each segment it re-decides which rendition to fetch next. Throughput-based ABR estimates bandwidth from recent downloads and picks the highest rung below a safety margin. Buffer-based ABR, such as BOLA, picks the rung from how full the buffer is, which is steadier when throughput samples are noisy. A simplified hybrid of the two:

def choose_rung(ladder_kbps, throughput_samples_kbps, buffer_s, current, seg_s=4.0):
    # harmonic mean: one slow sample counts more
    recent = throughput_samples_kbps[-5:]
    est = len(recent) / sum(1.0 / t for t in recent)
    safe = est * 0.8                      # leave headroom for variance

    if buffer_s < 2 * seg_s:              # danger zone: protect against a stall
        return 0
    target = max(i for i, r in enumerate(ladder_kbps) if r <= safe) if safe >= ladder_kbps[0] else 0

    if target > current and buffer_s < 4 * seg_s:
        return current                    # don't climb until the buffer can absorb a mistake
    if target < current and buffer_s > 6 * seg_s:
        return current                    # big buffer: ride out a dip instead of oscillating
    return target

The rules encode the trade-off every ABR algorithm makes between three things viewers notice: stalls (by far the worst), low quality, and frequent quality switches.

Live versus on-demand, and latency

Live and on-demand share the transcode and package stages but differ downstream. VOD manifests are complete and static; live media playlists grow continuously and slide a window over recent segments. Live concentrates the audience on a handful of new segments, so hit rates are high but load arrives in synchronised bursts.

Latency is the defining live metric, measured glass to glass from camera to screen. Classic HLS players often wait for about three segments before starting, so six-second segments put viewers twenty or more seconds behind real time. Low-Latency HLS splits segments into smaller parts that are announced before the full segment finishes, adds preload hints and blocking playlist reload so the player asks for the next part before it exists, and can bring latency down to a few seconds. Low-latency DASH achieves a similar result with CMAF chunks delivered using chunked transfer encoding. Interactive use cases go to WebRTC, which is stateful and hard to cache. Every second of latency removed is buffer removed, so low-latency streams rebuffer more easily; pick the latency your product truly needs.

Access control, DRM and ads

Protection has two layers. Access control decides who may fetch URLs at all: signed URLs or tokens checked at the edge, ideally scoped to a path prefix and an expiry rather than to individual segments, so the cache key stays shared. DRM decides who may decrypt: segments are encrypted (CMAF supports the cenc and cbcs schemes), and a content decryption module on the device obtains keys from a license server. Widevine, PlayReady and FairPlay cover the major platforms; using cbcs everywhere lets one encrypted copy serve all three on recent devices. If the license server fails, nobody plays.

Server-side ad insertion makes manifests per-session; keep ad segments cacheable and personalise only the lightweight playlist.

Measuring quality of experience

Server metrics alone cannot tell you whether viewers are happy, because the problems that matter happen in the player. Collect quality-of-experience metrics from clients: video start time, video start failures, rebuffering ratio (stall time divided by watch time), average delivered bitrate, rendition switch rate and exits before video start. Slice it by CDN, region, ISP, device and player version; incidents show up first as one slice going bad.

Common Media Client Data (CMCD, CTA-5004) closes the loop between player and CDN. The player attaches keys such as br (encoded bitrate), bl (buffer length), mtp (measured throughput), ot (object type) and sid (session id) to each request, as a query argument or headers, so CDN logs can be joined with session experience.

Failure modes and how to survive them

Before a large live event, load-test the shield path and rehearse a feed failover while watching a real player.

The trade-offs you will argue about

Most architecture arguments in streaming reduce to a few trade-offs. Segment length trades latency and switching agility against efficiency and request volume. More rungs give smoother adaptation but cost encoding, storage and cache fragmentation. Newer codecs cut egress but add encoding cost and device fragmentation. Low latency costs buffer and so costs stalls. Multi-CDN buys resilience and negotiating power at the price of operational complexity. Personalisation (ads, per-user manifests, watermarking) costs cacheability.

What to do next

1. Draw your own pipeline as in the diagram and mark, for every box, what happens when it fails.
2. Check GOP alignment across all renditions of one asset with ffprobe, and add that check to the encoding pipeline.
3. Audit Cache-Control headers for manifests, segments and error responses, live and VOD separately.
4. Confirm your shield collapses concurrent requests, then estimate origin load with the renditions x shields / segment-duration formula.
5. Instrument start time, start failures, rebuffer ratio and bitrate per CDN, region and device, and add CMCD.
6. Decide the live latency your product needs, then choose segment and part durations to match.
7. Read on: ABR selection algorithms, CMAF, Low-Latency HLS, DRM and per-title encoding.