On October 29, 2026, Meta's engineering blog reported that Catching JiTTests (just-in-time tests) have moved from research to production. The method was described in a January paper, "Just-in-Time Catching Test Generation at Meta", presented at FSE 2026. The figures below are Meta's own.
A test instead of an opinion
Most AI code review tools read a diff and comment on it, as a human reviewer would. Meta's system instead "generates executable tests that demonstrate a concrete behavioral difference introduced by the change." A catch is a generated test that fails on the diff and shows how its behaviour differs from the parent commit. That is evidence a reviewer can run, not a judgment they have to trust.
Five stages
- Targeting. The system runs overnight on spare capacity and picks diffs from the previous day that a risk model rates as risky.
- Two generators. The "dodgy diff" workflow treats each diff as if it were a mutant and writes tests that tell it apart from its parent. The intent-aware workflow asks an LLM what the diff is meant to do, lists the ways that could go wrong, builds mutants for those risks and writes tests to kill them. Tests that fail on the real diff become "weak catches".
- LLM-as-Judge. Several LLMs review the test code, the failure output, the diff and the inferred intent, and classify each weak catch as a likely true or false positive.
- RubFake. A rule-based assessor adds "verified, deterministic" judgments, because the LLM judges are non-deterministic.
- Human on the loop. Only high-ranked catches reach an engineer. People set the criteria and review results in aggregate rather than approving each output.
What Meta reports
As of the post, 589 engineers, "64% of our outreach", had engaged with the findings, and those engineers "have reported" a 60 percent true-positive rate. In the first half of 2026 the system "played a key role in catching 74 serious (SEV) privacy bugs at the diff stage", and Meta says privacy SEVs fell by 20 percent over that period.
What remains uncertain. The true-positive rate comes from engineers who engaged, not from every catch. The post does not say how many SEVs the other checks would have caught anyway, so the 20 percent drop is not a controlled result. It gives no compute cost per diff, although overnight runs on spare capacity suggest the cost is not small.
Analysis: The design choice worth copying is the burden of proof. A reviewer that only comments can be wrong in ways nobody checks. A generator that must produce a failing test can be wrong only in ways a rule engine and a person can verify. A small team can start with the dodgy-diff idea alone. For Belgavi's Book Store, when Jenya changes the refund calculation for returned books, ask a model for tests that pass on the old code and fail on the new. Each failing test is then a question for Jenya: was this change in behaviour intended?
Sources and evidence
Meta's October 29 engineering post says Catching JiTTests, LLM-generated tests that must fail on a risky diff to count as a catch, now run in production through two generators, an LLM-judge ensemble, a deterministic rule-based filter and human-on-the-loop review; Meta reports a 60 percent true-positive rate among engaged engineers and 74 privacy SEVs caught at diff time in the first half of 2026, without a controlled comparison.