Why architecture matters here
The architecture matters because VMAF is not a measurement you take — it is a control signal you build systems around, and control signals that are wrong in systematic ways cause systematic damage. Once you wire VMAF into your encoding ladder, your CDN spend, and your quality alerting, every property of the pipeline that produced the number propagates into decisions worth real money.
The clearest example is per-title encoding. The traditional bitrate ladder is a fixed table: 1080p gets 5 Mbps, 720p gets 3 Mbps, and so on, for every asset in the catalogue. This is obviously wrong in both directions simultaneously — the animated short with flat colours hits perceptual transparency at a third of that budget, and the film-grain-heavy thriller is still visibly degraded at double it. You are overspending on easy content and underdelivering on hard content, and you cannot tell which is which without a perceptual metric. With VMAF in the loop, the ladder becomes a search: encode at several bitrates, measure, and pick the lowest rung that clears your target score. The savings are large enough that VMAF's compute cost disappears into the rounding error of the bandwidth it saves.
But that same architecture means a mis-specified VMAF pipeline systematically mis-provisions your entire catalogue. Use the default 1080p model to decide about phone-delivered renditions and you systematically underestimate quality — the phone model knows a small high-DPI screen hides artifacts the TV model penalizes — so you spend bits buying quality no phone viewer can perceive. Multiply by a catalogue and that is a large, permanently recurring waste that nothing will ever flag, because the numbers are all perfectly self-consistent.
Pooling has the same character but a worse failure direction. Mean pooling on a ladder search selects the rung where average quality clears the bar, so it will happily accept a rendition that falls apart during the highest-motion scenes — exactly the scenes viewers watch most attentively. Percentile pooling inverts this, selecting on the worst moments, costing more bits and protecting the experience people actually remember. Neither is categorically right, but choosing without understanding the trade means you have chosen by accident.
The third architectural pressure is throughput. VMAF is expensive, and running it on every frame of every rendition of a large catalogue is a serious compute line item. That creates pressure to sample — score every tenth frame, or a few representative segments. Sampling is defensible, but it interacts badly with pooling, because the low-percentile scores you most care about live in precisely the brief high-motion passages that uniform sampling is most likely to skip. The compute budget therefore silently shapes what your quality signal can see, which is a coupling worth making explicit rather than discovering.
The architecture: every piece explained
The input stage is alignment, and it is where most broken VMAF numbers originate. VMAF compares a distorted frame against its reference frame, which requires that they actually correspond: same frame count, same frame order, same resolution, same colour space, same transfer function. Real pipelines violate every one of these. The encode dropped a frame during a scene cut. The reference is 4K and the rendition is 720p. The reference is full-range and the encode is limited-range. Each mismatch produces a number, and the number is nonsense — often catastrophically low in a way that looks like an encoder bug rather than a harness bug.
Scaling deserves specific attention because it is subtle. When the rendition is a lower resolution than the reference, you must upscale the rendition to the reference resolution before comparison — you are asking 'how good does this look when displayed at the reference size', which is what a viewer with a 1080p screen actually experiences. The scaler you use is part of the measurement. A bicubic upscale and a Lanczos upscale produce different VMAF scores for the identical encode, sometimes by more than a point, because the scaler is reconstructing detail that the metric then evaluates. This means the scaler must be pinned in configuration and versioned like any other part of the model, or your scores drift when someone upgrades a library.
The elementary metrics are the three feature extractors. VIF (Visual Information Fidelity) models how much of the reference's information survives in the distorted version, drawing on natural-scene statistics and a model of the human visual system's channel capacity — it is computed at multiple scales, which is how it captures both coarse and fine degradation. DLM, also called ADM (Detail Loss Measure / Additive Distortion Measure), separates the loss of genuine detail from the addition of spurious artifacts, which matters because these are perceptually different failures: a blurred face and a blocky face are both bad but bad in ways viewers describe differently. Motion is the simplest — a measure of temporal difference between adjacent reference frames — and it is a feature rather than a distortion measure. It tells the fusion model how much motion is present, because the perceptual weight of a given artifact depends heavily on whether the scene is static or moving.
The fusion model is the trained SVM. It takes the elementary features for a frame and emits a single 0-100 score calibrated against the human MOS data it was trained on. This is why the model variant is not a detail: the training data encodes a viewing context. The default vmaf_v0.6.1 assumes a 1080p television at roughly three screen-heights' viewing distance. The vmaf_v0.6.1neg variant is trained to be less fooled by sharpening enhancements that inflate scores without improving fidelity. The phone variant assumes a small screen where artifacts are less visible, so it scores the same encode higher. The 4K variant assumes a large display where they are more visible. Same pixels, different numbers, all correct for their context and all wrong for the others.
Pooling is the final architectural choice, and it is a policy decision wearing a statistics costume. Arithmetic mean gives you the average frame's quality and is dominated by the long stretches of easy content. Harmonic mean weights low scores more heavily, pulling the aggregate toward the bad frames. Low-percentile pooling — the 5th or 1st percentile — reports roughly the worst moments, which is the closest proxy for what a viewer complains about. Many mature pipelines report several poolings side by side rather than picking one, because the gap between the mean and the 5th percentile is itself the most informative signal in the whole report: a large gap means the encode is inconsistent, which is a different problem from being uniformly mediocre and calls for a different fix.
End-to-end flow
Follow one asset through a per-title ladder search. A 4K mezzanine arrives in the transcoding pipeline. The ladder generator wants to know, for the 1080p rung, the lowest bitrate that clears a VMAF target of 93 using the standard TV model with 5th-percentile pooling.
The pipeline encodes candidate renditions at several bitrates — say 2, 3, 4, and 6 Mbps — producing four 1080p files. For each candidate, the VMAF harness now needs a reference. The reference is not the 4K mezzanine as-is: since the rendition is 1080p and the target display is 1080p, the mezzanine is downscaled to 1080p to serve as the reference, and the rendition is compared against that at native resolution. The alignment stage verifies frame counts match, normalizes colour range, and confirms both are in the same transfer function.
For each frame pair, the harness extracts VIF at multiple scales, DLM, and the motion feature from the reference sequence. These features feed the SVM, which emits a per-frame score. The result is not one number but a time series — for a ninety-minute film at 24fps, roughly 130,000 scores per candidate. This series is the actual output of the metric, and everything after this point is data reduction that discards information.
The pooling stage collapses each series. The 3 Mbps candidate might show a mean of 94.2 and a 5th percentile of 81 — excellent on average, with a tail of poor frames concentrated in a few high-motion sequences. The 4 Mbps candidate shows a mean of 96.1 and a 5th percentile of 93.4. Under mean pooling with a target of 93, the 3 Mbps rung passes and you ship it. Under 5th-percentile pooling, 3 Mbps fails and 4 Mbps is selected. That is a 33% bitrate difference — and across a catalogue, a very large bandwidth bill — decided entirely by a pooling choice, with no change whatsoever to the encoder.
The selected rung's score is recorded alongside the encode's metadata, and this is where the pipeline earns its keep long after the encode ships. When quality regresses — someone tunes an encoder preset, upgrades a codec library, or changes the scaler — the regression appears as a distribution shift in scores across newly encoded assets. Catching that requires the scores to be comparable over time, which requires that the model variant, the scaler, the pooling method, and the VMAF version itself are all pinned and versioned. A VMAF number without that provenance attached is not a measurement; it is an anecdote, and it will fail you at exactly the moment you most need to trust it.