Write properties, not examples, for the rule layer that checks agent proposals: conservation, thresholds, monotonicity and idempotency, with the shrunk counterexample they found.
The rules are the part you can prove
When an agent proposes an action, a well-designed system does not take the proposal on trust. A deterministic layer checks it first: is the order refundable, is the amount within what remains, does it need a person's approval, has this exact proposal already been applied. At Belgavi's Book Store that layer is about forty lines of Python sitting between the help-desk agent and the payments API. The agent can be wrong in creative ways; the rule layer must not be.
Teams usually test this layer with a handful of examples: one refund that passes, one that is too large, one for a cancelled order. Those tests were written by the same person who wrote the rules, so they test the cases that person thought of. The agent, though, generates inputs nobody thought of. It splits one refund into three, retries a proposal after a timeout, proposes a negative amount after misreading a credit note, or sends a currency the order never had. The input space the rule layer faces has changed shape, and example tests do not cover that shape.
Property-based testing fits this layer unusually well. Instead of writing inputs, you write statements that must hold for every input, and a library generates hundreds of inputs looking for one that breaks the statement. When it finds one, it shrinks it to the smallest failing case. This guide applies it to an agent proposal guard, shows the bug it caught in a version that passed every example test, and covers what stays out of scope.
The guard under test
The guard takes an order and a proposal and returns one of accept, needs-approval, duplicate or a rejection with a reason. Money is held in integer paise, never in floating point. Accepted refunds are recorded on the order so later proposals see them.
APPROVAL_THRESHOLD = 500_000 # above 5,000.00 a person must approve
REFUNDABLE = {"delivered", "returned"}
def check(order, p):
if p.proposal_id in order.seen:
return "duplicate"
if p.order_id != order.order_id:
return "reject:wrong-order"
if p.currency != order.currency:
return "reject:currency"
if order.status not in REFUNDABLE:
return "reject:status"
if p.amount <= 0:
return "reject:non-positive"
if p.amount > order.total - order.refunded:
return "reject:exceeds-remaining"
if p.amount > APPROVAL_THRESHOLD:
return "needs-approval"
order.seen.add(p.proposal_id)
order.refunded += p.amount
return "accept"The first version that shipped had one difference: it compared the amount with the order total instead of the remaining refundable amount. Every example test passed, because every example test proposed a single refund.
Writing properties instead of examples
A property is a sentence you would put in a design review, turned into code. For this guard the sentences are:
- Across any sequence of proposals for one order, the total accepted never exceeds the order total, and the recorded refunded amount equals the sum of accepted proposals.
- Nothing above the approval threshold is ever accepted automatically.
- Rejection for size is monotonic: if an amount is rejected as too large, every larger amount is rejected too.
- Replaying an accepted proposal returns duplicate and does not change the refunded amount.
The first property is the one that matters most, and it is stateful: it is about sequences, not single calls. The generator therefore produces whole plans, lists of up to twelve proposals with ids drawn from a small pool so that replays happen, amounts that include negatives and values far above the threshold, and an occasional wrong currency.
from hypothesis import given, settings, strategies as st
amounts = st.integers(min_value=-1_000, max_value=2_000_000)
statuses = st.sampled_from(["delivered", "returned", "shipped", "cancelled"])
plans = st.lists(st.tuples(st.integers(0, 7), amounts,
st.sampled_from(["INR", "INR", "INR", "USD"])), max_size=12)
@settings(derandomize=True, max_examples=300, deadline=None)
@given(st.integers(1, 1_500_000), statuses, plans)
def test_accepted_total_never_exceeds_order_total(total, status, plan):
order = Order("ORD-7781", total, status=status)
accepted = 0
for pid, amount, currency in plan:
if check(order, Proposal(f"P-{pid}", "ORD-7781", amount, currency)) == "accept":
accepted += amount
assert accepted <= total
assert order.refunded == acceptedThe generator deliberately weights the domain toward the cases an agent produces: small id pools for retries, out-of-range amounts, the wrong currency a quarter of the time. Generating uniformly over all integers would spend almost every example on amounts that are rejected for being huge, which tests nothing interesting.
What it found
Run against the version that shipped first, the cumulative property fails immediately. Hypothesis, version 6.170.15 in the companion, reports the shrunk failing case:
Failing test case: cumulative(
total=1,
plan=[(0, 1, 'INR'), (1, 1, 'INR')],
)An order worth one paisa, two different proposals of one paisa each, and both accepted. That is the whole bug in its smallest form: each proposal individually fits the order total, together they exceed it. Shrinking is what turns a generated failure into something a reviewer understands in five seconds: the library keeps simplifying the failing input, smaller numbers and shorter lists, for as long as the property still fails.
This is exactly the shape of input agents produce. An agent that decides to refund a damaged book and then, after a follow-up message, decides to refund the shipping charge too, issues two proposals that are each reasonable. A person writing example tests rarely writes the sequence, because a person thinks of one refund per case.
Making the tests usable in CI
Two settings matter. derandomize=True makes Hypothesis explore the same examples on every run, so a CI failure is reproducible and a green build means the same thing every time. Without it, property tests are randomised, which finds more bugs over time but can fail on one run and pass on the next; the Hypothesis documentation also describes a local example database that replays earlier failures. Many teams use derandomized runs in CI and a separate longer, randomised job on a schedule. The second setting is max_examples. Three hundred plans of up to twelve proposals run in about two seconds here; raise it in the scheduled job, not in the pull-request gate.
When a property fails, copy the shrunk example into an ordinary example test with a name that says what it means. The property keeps searching; the example keeps the specific regression pinned forever.
Properties that are worth writing for agent guards
Beyond the refund case, the same handful of property shapes covers most guards that sit in front of agents:
- Conservation: some quantity is never exceeded across any sequence, such as refunds, credits issued, stock reserved or emails sent per case.
- Threshold respect: no automatic acceptance above a limit, regardless of how the request is split or phrased.
- Monotonicity: if a request is rejected for being too large, too broad or too risky, a larger or broader one is rejected as well.
- Idempotency: replaying a proposal changes nothing after the first application.
- Order independence where it should hold: the final state after a set of accepted proposals does not depend on the order they arrived in, or, if it does, the dependence is intended and documented.
- Round trips: serialising a proposal and parsing it back gives the same decision, which catches drift between the agent's tool schema and the guard's parser.
What property tests do not cover
They test the deterministic layer, not the agent. They will not tell you that the model proposes sensible refunds; that needs evaluation on recorded cases. They do not test concurrency unless you model it: two workers applying proposals to the same order at the same time can still race in the database, and the fix there is a transaction or a conditional update, verified with a test that runs two connections. They also only check the properties you write. A guard that is perfectly conservative but rejects every legitimate refund passes the conservation property, which is why a liveness property such as "a single refund within the remaining amount and below the threshold is accepted for a delivered order" belongs in the suite too.
Finally, the generator encodes your assumptions about inputs. If the agent's tool schema allows a field the generator never produces, such as a partial refund of a digital edition, the property tests stay green while that path is untested. Review the generators whenever the tool schema changes.
A migration path for an existing guard
- Write the conservation and threshold properties first, against the current guard, before changing any rule. Treat any failure as a real bug to triage, not a test problem.
- Move money to integer minor units if it is not already; floating point makes conservation properties fail for reasons unrelated to the rules.
- Add the idempotency property and give every agent proposal a stable id from the orchestrator, not from the model's text.
- Pin each shrunk counterexample as an example test.
- Add the derandomized property suite to the pull-request gate and a longer randomised run to a nightly job.
Run the example yourself
The refund guard, the version that shipped first, and the Hypothesis 6.170.15 properties that catch it, pinned and derandomized for CI. Companion code on GitHub; its README lists the prerequisites, exact commands, expected output and limits.