Testing a Service Once an Agent Calls It: Consumer Contracts at the Tool Boundary, Recorded Model Replays, Effect Invariants and What Stays Out of CI

A caller you did not design for

A billing service has had the same callers for years: a web front end, a nightly batch and two internal services. Each was written by people who read the API documentation, and each calls the API in a small number of patterns that the integration tests already cover. Then a support agent is given a refund_order tool that wraps the same endpoint. The new caller reads a JSON Schema and a one-paragraph description, decides when to call, composes the arguments from a conversation, retries when it sees an error string, and sometimes calls twice with slightly different wording in a free-text field.

None of that breaks the service on day one. What breaks is the test strategy: the suite assumes a few known call patterns, and the agent produces a long tail of valid-but-unusual ones. Putting the model into the test suite produces slow, flaky, expensive tests that still miss most of the tail. Split the system instead along the line between what is deterministic and what is not, and give each side the test that suits it.

What stays deterministic

Four of the six rows can be tested exactly, every commit, without a model. Only the last two need a model, and those need repeated trials and confidence intervals rather than assertions. A flaky test that fails one run in twenty only teaches people to press retry.

The tool definition is the consumer contract

Consumer-driven contracts invert who writes the integration test. The consumer states which requests it sends and which parts of each response it relies on; the provider runs those expectations against its real implementation in its own pipeline, so a provider change that would break a consumer fails before it ships. Pact is the best-known tool for this, and Ian Robinson's original article describes the pattern without any tool.

When the consumer is an agent, the contract has two parts. The first is the tool definition itself: name, description, inputSchema and, if you declare one, outputSchema, which the MCP specification uses to validate structuredContent in results. A change to any of these changes what the model sees, so it is a contract change even when the HTTP API underneath is untouched. The second part is a set of interactions recorded from real agent runs: the arguments the agent actually sent and the result fields it actually read in its next step. Those recorded calls are more varied than anything written by hand, which is why they belong in the contract.

Illustrative python, checked for syntax; adapt names and storage to your system.

import json

def check_interaction(schema, call, provider):
    """Replay one recorded agent call against the real provider; return problems."""
    problems = []
    for field in schema["required"]:
        if field not in call["arguments"]:
            problems.append("missing required argument " + field)
    result = provider(call["arguments"])
    for field in call["fields_read_next"]:      # what the agent used in its next step
        if field not in result:
            problems.append("result no longer carries " + field)
    if call.get("expect_error") and "error" not in result:
        problems.append("call that used to be rejected is now accepted")
    return problems

def verify(contract_path, provider):
    contract = json.load(open(contract_path, encoding="utf-8"))
    failures = {}
    for call in contract["interactions"]:
        found = check_interaction(contract["tool"]["inputSchema"], call, provider)
        if found:
            failures[call["id"]] = found
    return failures

The fields_read_next list is the important addition. An agent does not read a whole response; it reads the two or three fields that its next step quotes. Recording which ones lets the provider remove or rename everything else freely, and fails the build only when it touches a field an agent depends on.

Recorded model replays for the runtime

The runtime around the model has plenty of deterministic logic: it parses tool calls, enforces budgets, retries, routes to approval, journals steps and formats the final answer. That logic is tested badly when every test calls a live model, because the input changes from run to run. Record instead. In a staging run, store each model request as a hash of its normalised inputs together with the model identifier and the full response. In CI, a fake model looks up the hash and returns the stored response, so the runtime sees exactly the conversation it saw before and the test can assert on every step.

Two rules keep replays honest. A request whose hash is not in the store fails the test rather than falling through to a live call, so a prompt change cannot silently turn a replay test into a live one. And fixtures are refreshed deliberately, by a job that re-records and opens a review, never automatically on mismatch. A diff in the recorded responses after a prompt change is information worth reading.

Effect invariants under generated sequences

The agent will eventually send a sequence nobody wrote a test for: a refund, a retry of the same refund after a timeout, a second refund with a reworded reason, a refund for an order that was already cancelled. Property-based testing fits this well. Hypothesis generates sequences of calls from strategies you describe, runs them against the real handler with a test database, and shrinks any failing sequence to a minimal one. The properties are business invariants that must hold for every sequence, such as the total refunded never exceeding the amount captured, and a retried call with the same idempotency key never producing a second effect.

Illustrative property test; FakeBilling stands for your handler wired to a test database.

from hypothesis import given, strategies as st

call = st.fixed_dictionaries({
    "order_id": st.sampled_from(["o-1", "o-2"]),
    "amount": st.integers(min_value=1, max_value=120),
    "reason": st.sampled_from(["late", "late delivery", "damaged"]),
    "key": st.sampled_from(["k-1", "k-2", "k-3"]),
})

@given(st.lists(call, max_size=12))
def test_refunds_never_exceed_capture(calls):
    billing = FakeBilling(captured={"o-1": 100, "o-2": 60})
    for c in calls:
        billing.refund_order(**c)
    for order_id, captured in billing.captured.items():
        assert billing.refunded(order_id) <= captured

Worked example: one refund tool, three failures caught without a model

This example is hypothetical; the runnable version, with fixtures and tests, is in agent-decision-lab. The contract holds five interactions recorded from a staging agent. In one of them the agent read refund_id and remaining_refundable from the result and quoted the second to the customer.

A provider developer renames remaining_refundable to refundable_balance as a tidy-up. The HTTP clients do not use that field, so every existing test passes; contract verification fails on the two interactions that read it, naming each interaction and the field. Next, someone relaxes validation so that a negative amount is accepted as a reversal. One recorded interaction was an agent attempt that the service correctly rejected, marked expect_error, and it now fails. Finally, the property test finds a two-call sequence the handler mishandles: two refunds for the same order under two different idempotency keys, which an agent produces when it re-plans after a timeout and builds a fresh key. Deduplication by key cannot help, the total exceeds the captured amount, and the shrunk example is two calls long. The fix belongs in the handler: check the cumulative refund against the captured amount in the same transaction, and derive the key from the order and the conversation so that a re-plan is recognised as the same request.

What stays out of CI

Whether the agent chooses the refund tool for the right complaints, and with the right amount, is a property of the model, the prompt and the traffic. It belongs in an evaluation suite that runs on a schedule and on prompt or model changes, uses enough trials to put an interval around each rate, and compares against a recorded baseline, as described in the evaluator architecture guide. The pipeline can block a release on that suite, but the suite should not run on every commit or sit among unit tests.

Migration steps for an existing service

1. Inventory the endpoints that are now wrapped as tools and write each tool definition into the repository as a versioned file, reviewed like an API change.
2. Turn on recording in a staging environment: arguments, result fields read in the next step, rejected calls, and model request hashes with responses.
3. Add provider verification of the recorded interactions to the service pipeline, failing on removed fields and newly accepted rejected calls.
4. Write two or three invariants per effecting tool and run them as property tests against the real handler.
5. Replace live-model tests of the runtime with replay tests that fail on an unknown request hash.
6. Move the remaining model-quality checks into a scheduled evaluation with intervals and a baseline.

When an agent caller is unnecessary to plan for

If the tool is called only by a fixed workflow that a person wrote, with arguments assembled by code, the caller is an ordinary client and ordinary contract tests are enough; recording interactions adds nothing a hand-written example would not cover. The extra machinery here pays for itself when the arguments are composed by a model, when the same tool is offered to several agents, or when the tool has effects that cannot be undone. A read-only lookup tool needs the schema contract and little else.

Failure modes worth testing

• A description edit changes which tool the model picks while every schema test passes; treat descriptions as part of the contract and evaluate after editing them.
• Recorded interactions contain personal data from staging conversations; scrub them at recording time, not later.
• The replay store silently falls back to a live call and a deterministic test becomes flaky; make a missing hash a hard failure.
• Property tests run against an in-memory fake that lacks the database constraint the real handler relies on; run them against the real storage engine in a container.

Practice and related reading

Further reading:

• Agent evaluator architecture: grading contracts and pass^k
• Agent-ready API contracts
• Unit testing ADK Java agents
• MCP testing patterns

Sources and scope

Checked against these sources on 20 October 2026. Examples and figures are illustrative, not measurements.

• Ian Robinson, Consumer-Driven Contracts: A Service Evolution Pattern (martinfowler.com)
• Pact documentation: consumer-driven contract testing
• Model Context Protocol specification 2026-07-28: Tools (inputSchema, outputSchema, structuredContent)
• Hypothesis documentation: property-based testing for Python