Make the failure visible. Then fix it.
Change real inputs, inspect deterministic outcomes, and explain which boundary holds. Start with architecture repair then test retry safety, promotion gates and independent outcome grading. Your 24 checkpoints save on this device.
Your next useful step
Start your first mission.
Start mission 1 Weekly practice
Progress and drafts save on this browser. Use synthetic data only. Clearing browser storage removes them; there is no account sync.
My activity trail
Only the latest 200 events are retained on this device. These records are editable local practice records, not verified credentials.
Stop the action before it escapes.
A support agent drafts a refund and immediately invokes its payment tool. Its prompt says to be careful, but the gateway checks the budget only after generation and records the audit after delivery. Your job is to repair the prerequisite order. Edit the steps array; retain all nine stages exactly once. Run the fixture and use each failed edge to locate the misplaced control.
Read the guided walkthrough
First move budget and minimize before model. Put validation after generation, then approval before action. Audit the outcome before delivering it. The identity stage represents an authenticated principal and authorized scope, not a name supplied by the browser. In a deployed service, policy enforcement belongs in trusted code and must run for every invocation, including retries and tool callbacks.
What the result establishes
A green result proves only that this small graph respects its declared dependencies. It cannot prove that a human actually approved the exact action, that an audit write is durable, or that the output validator understands the business rule. Bind approval to an immutable action digest and principal; expire it; reject replay; and record both attempted effects and their outcomes. An audit outage after a completed effect requires reconciliation, not silently repeating the effect.
Hard mode: place audit before action and explain why a record of intent cannot stand in for a record of the outcome. Then design a second audit event before the effect. This exercise contains one audit stage for the delivered result; a production design normally needs both events and an idempotency key.
Read the reasoning: Hybrid agent architecture · Afp and prompt injection
Valid JSON can still be a bad decision.
The provider returned a syntactically correct object, but the distribution adds up to more than one. Inspect the response as a gateway would. Repair the three probabilities, then deliberately select a non-maximum option. The validator should reject that mismatch. The exercise checks a narrow Choice-style contract, not every field in a real provider response.
Read the guided walkthrough
Try proceed at 0.90, review at 0.08 and block at 0.02. The result waits for approval; it never executes an action. Next reduce proceed to 0.80 and raise review to 0.18. The same selected label now enters review because this fixture applies a threshold of 0.85. That number is an explicit teaching policy, not a claim that the model is calibrated or safe for refunds.
What the result establishes
The most important experiment uses a structurally valid distribution whose selected answer you know is wrong. The contract accepts the shape because it cannot know the ground truth. Evaluate correctness separately on labeled examples, report false positives and missed positives, and examine results by scenario. Do not turn confidence into permission or silently reinterpret an unfamiliar option as a familiar action.
Hard mode: add an execute_refund option, use a string instead of a number, or create a tied maximum. Unknown options and incorrect types fail. A tied maximum is structurally allowed here but should trigger a separate policy decision in a real workflow. Write that policy before changing the validator, so evaluation measures intended behavior.
Read the reasoning: Typed decision contract · Confidence calibration
Keep the secret out of the worker.
A refund reviewer needs an order fact and the applicable refund rule. The surrounding customer record contains unrelated fields. Run the fixture and inspect the exact simulated worker state. Only order_status and refund_rule survive. Nothing is sent to a model or logged to a server. Drafts are saved on this device until you clear them. Use synthetic values while experimenting.
Read the guided walkthrough
Remove the refund_rule field and run again. The minimizer fails closed instead of inventing a default policy. Empty strings and evidence longer than 4096 characters are also rejected. This is a field allowlist: it makes the input contract explicit and reduces accidental disclosure from broad object serialization. The omitted field count is shown without echoing the dropped values in the result.
What the result establishes
Now place a fake secret inside order_status. It remains in the output because field selection is not semantic redaction. This counterexample matters: calling an object minimized does not mean it contains no personal data. Production inputs need a purpose-specific schema, provenance, applicable retention controls, and a deliberate policy for free text. Detection of sensitive content is a separate control with its own false positives and misses.
Hard mode: insert an instruction such as ignore policy inside the order text. It remains untrusted evidence, not authorization. The browser fixture does not execute that instruction and does not measure whether a real model would follow it. Add adversarial documents to a controlled evaluation, then verify that tool permissions remain bounded even if the model produces a harmful recommendation.
Read the reasoning: Afp and prompt injection · Rag evidence verification
Who pays for the failed attempt?
An agent has two remaining calls and a queue of four requests. One caller lacks authorization and one provider attempt fails. Predict the call count before running the simulation. Unauthorized requests are denied without consuming a reservation. Authorized attempts reserve a call before provider work, and the failure does not automatically refund it. The last request should exhaust the budget.
Read the guided walkthrough
Change the initial budget to zero and then to three. Compare the trace rather than looking only at the final number. This lab models call reservations, not currency, tokens, rate limits, or vendor billing. A real cost guard needs estimated upper bounds, measured usage reconciliation, per-principal limits, and an explicit policy for provider timeouts whose billing outcome may be unknown.
What the result establishes
The sequence is deterministic and runs in one browser task. It does not demonstrate atomic enforcement under concurrency. In a deployed gateway, two workers reading the same remaining count can both believe they have permission. Use an atomic reservation in the authoritative ledger and make the reservation identity stable across retries. Never trust the remaining count supplied by a client.
Hard mode: explain why blindly retrying the failed call can exceed the budget even though no response was delivered. Then design bounded retries with a deadline and reserved capacity. A monetary refund from a provider and a release of application capacity are separate events; do not conflate them. The output intentionally distinguishes denied, exhausted, failed and received attempts.
Read the reasoning: Model routing and cost · Gcp realtime deployment
Build a query plan without opening a database.
This miniature schema exposes one table with a single partition key. Supply the full tenant_id key, choose explicit columns, and keep the limit between one and one hundred. The output is a prepared query plan with separate parameters. It never connects to Cassandra. The companion Java, Python and Node implementations perform real schema-aware checks; this lesson uses a fixed teaching schema.
Read the guided walkthrough
Replace the table with system.local, add an unknown column, repeat a column, or request a limit of 101. Each should fail before a plan is shown. Change the tenant value to a different valid identifier and observe that a plan is still produced. That is a deliberate demonstration of a missing layer: this tool-level schema check is not an authenticated tenant authorization system.
What the result establishes
Prepared values prevent a value from becoming query syntax, while allowed identifiers prevent table and column names from being chosen freely. Neither protection grants a user permission to see a partition. The trusted gateway must derive tenant scope from verified identity and compare the requested key against it. The database role should remain SELECT-only and limited to the exact tables needed.
Hard mode: explain why a small row limit is not a complete memory limit when one row contains a huge cell. The real servers apply response bounds but do not claim to prevent all driver-side allocations. Use bounded schemas and production data tests. This fixture validates a restricted string key; it does not claim compatibility with every Cassandra type or composite-key arrangement.
Read the reasoning: Typed decision contract · Evaluation and fault testing
The timeout hid a committed effect.
One logical operation is retried after a response disappears. Run the starter fixture and inspect the effect count. The first attempt already committed, so a fresh identity on the second attempt creates duplicate work. Your first challenge is to preserve the operation identity across attempts. This is an in-memory teaching ledger; it does not contact a payment service.
Read the guided walkthrough
Set stableKey to true, keep the timeout-after-commit followed by success, and run again. The second attempt replays the recorded result. Then switch the key policy back to false to demonstrate the duplicate. For the final checkpoint, make every attempt fail before commitment. A safe recovery design must distinguish no effect from an effect whose response was lost.
What the result establishes
A real idempotency contract binds an operation key to the authenticated principal and exact payload. It requires durable atomic behavior at the effect boundary, not a cache around an arbitrary network call. This simulator is sequential and does not prove crash safety or concurrent correctness. It makes the relevant states visible so you can specify those tests before implementing them.
Hard mode: explain how recovery changes if a provider forgets keys after twenty-four hours. Should a delayed worker replay an old operation? The answer requires provider-specific retention and reconciliation rules. Do not infer that a timeout, a missing chat message or a restarted worker means the original action never happened.
Read the reasoning: Hybrid agent architecture · Evaluation and fault testing Read the harness engineering series
Better on average. Worse where it matters.
A proposed harness configuration improves a small set of supplied outcomes. Before accepting it, check that evaluation source groups do not overlap training and that critical cases still pass. The fixture starts with a candidate ready for human review. These booleans are teaching inputs, not measured model predictions. Nothing is trained, promoted or deployed.
Read the guided walkthrough
Add holdout-a to trainingGroups and rerun: the gate rejects the comparison as leaked. Remove the overlap, then make a critical candidate result false. Even if you improve other cases, that critical failure blocks the candidate. The objective is to preserve the independence of the reviewer and the importance of the boundary being tested.
What the result establishes
Four visible cases cannot establish statistical improvement or serve as a secure held-out benchmark. A production loop needs independently collected outcomes, protected evaluation membership, representative coverage, repeated trials where appropriate and a rollback plan. The proposer must not edit critical labels or relax the gate. Our output deliberately says ready for review rather than automatically approved.
Hard mode: make all candidate values identical to the baseline. The gate returns no measured gain. Explain why refusing to promote can be a healthy outcome for a self-improvement loop. More changes do not necessarily produce more value, and a higher aggregate score should never erase a critical regression.
Read the reasoning: Evaluation and fault testing · Fine tuning and data Read the harness engineering series
The agent said done. Where is the record?
The worker claims success, but the simulated database has no resulting record. Run the case and examine which fact decides the outcome. Your first checkpoint demonstrates a false success claim. Next create a stored outcome under valid authorization and bound approval. Finally produce a stored outcome without authorization and observe the policy breach.
Read the guided walkthrough
Change recordExists to true while retaining authorized and approvalBound as true. The result becomes verified within this fixture. Turning authorized off then produces a breach, even if the record exists and the assistant sounds confident. A completed effect and a permitted effect are separate requirements; neither substitutes for the other.
What the result establishes
The booleans in this browser are editable assumptions. They are not authentication checks. A real grader must read independently trusted evidence, including resource ownership and the exact approved action. Model-generated claims cannot populate these fields authoritatively. A tool returning HTTP 200 also does not establish that the requested business outcome was committed correctly.
Hard mode: set claimedSuccess to false while leaving a valid authorized record in place. The grader still recognizes the stored outcome. Explain how this differs from a communication-quality check: the system may have completed the task while giving a poor answer. Keeping those measurements separate makes repairs more precise and prevents one combined score from hiding policy failures.
Read the reasoning: Typed decision contract · Evaluation and fault testing Read the harness engineering series