Desired state and a reconciler: converge, crash, recover

What this experiment demonstrates

An agent emits a desired state: three read grants for a contractor, with an expiry. It does not call the grant API itself. A deterministic reconciler reads the current state, computes the difference, and applies only the changes that close the gap, and it runs that loop once per pass. The experiment lets you break the loop in the four ways that matter and see whether it still arrives at the desired state.

Current state starts with some drift: grants that no one requested, which the reconciler must remove. You can mark the desired state expired, so the correct current state is empty. You can crash pass one partway through, make the apply non-idempotent so a repeat double-applies, and make the reconciler read a cache that lags one pass behind its own writes. Nothing calls a real API; the model is deterministic, so every run is repeatable.

Run controlled comparisons

Start with the defaults: not expired, one drift grant, two passes, a crash after one change on pass one, idempotent apply, fresh reads. The first pass adds the three grants and removes the drift but stops after the crash; the second pass recomputes the diff and finishes, and the run converges with an unconverged gap of zero.

Now change one thing. Set passes to one: the crash is never recovered, and the gap is non-zero, because a level-triggered loop needs to run again to catch up. Turn on stale reads with idempotent apply left on: the reconciler keeps seeing the pre-write state and recomputes the same diff every pass, yet the real state stays correct and the gap is zero, because an idempotent apply makes the redundant work harmless. Now also turn idempotent apply off: those same stale reads make the reconciler add grants it already created, and the duplicate-applies count rises, which is exactly why every apply must be keyed. Mark the desired state expired: the correct outcome is that every grant is removed, which is the same loop proving it handles revocation with no separate reminder.

Inspect the trace and keep evidence

The decision trace shows each pass: the current state the reconciler read, the diff it computed, and every add or remove it applied, with the crash and any duplicate apply marked. The metrics summarise the whole run: changes applied, passes, the unconverged gap, duplicate applies, drift corrected and expired grants removed. Start trace, Next step, Show all and the slider control how much of the trace is visible; the metrics and the state table always describe the complete run. Reset restores the defaults and Export JSON downloads the inputs, the metrics, the state table and the trace.

Limits and production translation

The model applies changes instantly and reads an exact state; a real reconciler calls APIs that can fail individually, and a real current state is read over the network with its own consistency. The grants here are a stand-in for any declarative end state with a time bound. What carries over is the structure: keep intent and progress in separate places, make every apply idempotent and keyed, read the current state fresh each pass, and run the loop on a timer as well as on each new proposal so expiry and drift are corrected without a new agent call.