Prepare routing JSONL without teaching the router its own mistakes

By Sandeep Belgavi · 18 October 2026

Preserve label provenance and separate task families before evaluating routing updates.

A log entry is an observation

An execution log identifies which specialist was selected. It does not establish that the choice was correct. Training directly on the router own selections can reinforce initial mistakes. Distinguish selected_agent from reviewed_agent and preserve outcome evidence separately. A correct final answer also does not establish an efficient initial route: unnecessary transfers may have rescued a poor first decision at substantial cost.

Design a row for its actual purpose

A compact routing row can contain task_id, group_id, category, previous_agent, reviewed_agent, review_status and evidence_reference. Exclude sensitive text unless the classifier needs it and its use is permitted. JSON Lines stores one JSON object per line. That format does not by itself make the data appropriate for LLM fine-tuning. Provider training schemas, model objectives and consent requirements are separate decisions from fitting a routing count table.

Split groups before creating variants

Twenty paraphrases from one support ticket should not be randomly scattered between training and evaluation. Assign the originating ticket or family to one split before deriving variants. Depending on the deployment, stronger separation by customer, repository or time period may be needed. State which leakage the grouping prevents and what it cannot detect. Similar wording across unrelated identifiers can still reveal a shared template or copied answer.

Quarantine invalid and unreviewed records

Reject duplicate task identifiers with conflicting labels. Exclude unreviewed records from supervised updates. Check that every reviewed destination belongs to the supported role catalogue and that the reviewer can inspect the evidence. Keep a rejection report with specific reasons. Silent dropping obscures dataset quality, while accepting every syntactically valid row permits contaminated labels and unknown roles to change future behavior.

Version the transformation as well as the input

Preprocessing can change effective training data without changing the original file. Record normalization rules, category mappings, split assignment, review policy and code revision with the generated table. If performance declines, this evidence distinguishes a modeling issue from a changed label interpretation. Retain an immutable accepted dataset manifest and a way to reproduce its transformation; do not depend on a mutable folder of session exports as the only record.

Connect checks to the release gate

Dataset validation should fail before any candidate is trained when groups overlap or labels violate the contract. A later metric cannot repair contaminated evaluation evidence. Current Arena routing datasets now check declared group overlap and train reviewed count tables; they do not discover undeclared semantic duplicates automatically. A stronger implementation would combine group checks, duplicate analysis and reviewer provenance, with each check producing independently inspectable evidence.

Run the exercise

Download this Python exercise

Save the code as 10-routing-jsonl-and-leakage.py and run python 10-routing-jsonl-and-leakage.py with Python 3.11 or newer. This standard-library exercise uses synthetic data. It does not invoke a model or require an account, network connection or API key.

# Author: Sandeep Belgavi. Synthetic exercise; no network or provider calls.
import json
lines=['{"task_id":"a","group_id":"ticket-1","review_status":"accepted"}',
       '{"task_id":"b","group_id":"ticket-2","review_status":"pending"}']
rows=[json.loads(line) for line in lines]
accepted=[r for r in rows if r["review_status"]=="accepted"]
assert len(accepted)==1
assert {"ticket-1"}.isdisjoint({"ticket-2"})
assert not {"ticket-1"}.isdisjoint({"ticket-1"})
print("one reviewed row accepted; overlapping group detected")

Expected output:

one reviewed row accepted; overlapping group detected

Change one assumption

Add two accepted records with the same task_id and conflicting reviewed_agent values, then implement rejection. Place two paraphrases from one ticket in different splits. The group validator must reject them even with different task identifiers. Finally record rejected row counts without retaining unnecessary sensitive request text in the report.

Implementation scope and further reading

This is an original AICassindra teaching design. The exercise demonstrates the stated invariant; it is not a complete production implementation. The local Harness Studio extension supports saved read-only agent teams, reviewed count-based routing and bounded sequential execution. Arbitrary imported graphs, effectful tools and automatic self-promotion remain unsupported.

All harness articles · Interactive boundary labs · Our public lab code