Deadline Propagation Across Service Hops: Budgets That Shrink, Clock Skew, and the Work Nobody Will Wait For

A deadline is a budget, not a timeout per hop

The SRE book gives the canonical example. Server A picks a 30 second deadline, spends 7 seconds, and calls B with 23 seconds left; B spends 4 and calls C with 19. Every call in the tree shares one absolute deadline, and each hop forwards what remains. Compare the common alternative, a fixed 10 second timeout configured on every client: A waits 10 for B, B waits 10 for C, and C may still be working on a request A gave up on eight seconds ago. The book has a phrase for this: you do not get credit for late assignments with RPCs. Work done after the caller has gone is wasted capacity, and under overload it is the capacity you needed to recover.

Deadline propagation is the practice of carrying the budget with the request. The gRPC deadlines guide describes it as supported automatically in some implementations, enabled by default in Java and Go and requiring explicit enablement in C++, so that an incoming deadline flows to outgoing calls without the error-prone approach of setting one by hand on every RPC.

What crosses the wire

The gRPC HTTP/2 protocol document defines the grpc-timeout header: a positive integer of at most eight ASCII digits followed by a unit, H for hours, M for minutes, S for seconds, m for milliseconds, u for microseconds or n for nanoseconds. If the header is omitted the server assumes an infinite timeout. The important design choice is in the deadlines guide: a deadline is a point in time, but propagating it as-is would break when the two machines disagree about the time, so gRPC converts it to a timeout from which the elapsed time has already been deducted. The receiving server adds that timeout to its own clock. Clock skew between hosts therefore never enters the calculation; only the transit time does, and you account for that with a margin.

On the server side the guide notes two behaviours. A gRPC server automatically cancels a call once the client deadline has passed, with a CANCELLED status for the handler while the client sees DEADLINE_EXCEEDED. And the application is responsible for stopping any work it spawned: a long-running handler must periodically check whether its RPC was cancelled. Plain HTTP has no standard header for this. Teams that propagate deadlines over HTTP choose a header carrying a relative budget in milliseconds, document it, and apply the same rules.

Implementation in Go and Python

Illustrative Go; context.WithTimeout and Deadline are from the standard context package, and grpc-go propagates the incoming deadline into ctx.

// Go: the incoming context already carries the client deadline (grpc-go propagates it).
func (s *server) GetOrder(ctx context.Context, req *pb.GetOrderRequest) (*pb.Order, error) {
    // Reserve a margin for our own post-processing and the network hop.
    child, cancel := withRemaining(ctx, 50*time.Millisecond, 800*time.Millisecond)
    defer cancel()
    inv, err := s.inventory.Lookup(child, req.Sku)   // outgoing call inherits the shrunk budget
    if err != nil {
        return nil, err
    }
    if ctx.Err() != nil {                             // caller gave up while we waited
        return nil, status.Error(codes.DeadlineExceeded, "caller deadline passed")
    }
    return assemble(req, inv), nil
}

// withRemaining derives a child context whose deadline is the parent deadline minus margin,
// capped at maxHop so a generous caller cannot pin our thread for long.
func withRemaining(ctx context.Context, margin, maxHop time.Duration) (context.Context, context.CancelFunc) {
    budget := maxHop
    if dl, ok := ctx.Deadline(); ok {
        if left := time.Until(dl) - margin; left < budget {
            budget = left
        }
    }
    if budget <= 0 {
        budget = time.Millisecond                       // fail fast rather than skip the call silently
    }
    return context.WithTimeout(ctx, budget)
}

Illustrative python, checked for syntax; adapt names and storage to your system.

import time

class Budget:
    """Remaining time for one request, measured on the monotonic clock of this process."""
    def __init__(self, total_ms):
        self.deadline = time.monotonic() + total_ms / 1000

    @classmethod
    def from_header(cls, headers, default_ms=2000, max_ms=30000):
        raw = headers.get("x-request-budget-ms")
        try:
            ms = min(int(raw), max_ms) if raw is not None else default_ms
        except ValueError:
            ms = default_ms
        return cls(ms)

    def remaining_ms(self):
        return max(0, int((self.deadline - time.monotonic()) * 1000))

    def check(self, stage):
        if self.remaining_ms() == 0:
            raise TimeoutError(f"deadline passed before stage {stage}")

    def outgoing(self, margin_ms=50, max_hop_ms=5000):
        """Header value and client timeout for the next hop."""
        left = min(self.remaining_ms() - margin_ms, max_hop_ms)
        if left <= 0:
            raise TimeoutError("not enough budget left for a downstream call")
        return {"x-request-budget-ms": str(left)}, left / 1000

def can_retry(budget, typical_latency_ms, margin_ms=50):
    """Retry only if the typical call fits in what is left; otherwise fail now."""
    return budget.remaining_ms() - margin_ms > typical_latency_ms

Both versions carry a relative value across the hop and convert it to a local deadline on arrival, which is the skew-proof shape the gRPC guide describes. Both check the budget before each stage, as the SRE book recommends for multi-stage handling, and both cap the outgoing budget so that a caller with a very long deadline cannot occupy a downstream thread for that long. The Python retry guard encodes a rule that is easy to forget: a retry is only worth issuing if the typical latency of the call fits in the remaining budget.

Worked example: three hops and one retry

The last row is the case that matters under overload. A retry issued when the budget cannot accommodate the call is work that will be thrown away, and under overload it is exactly the work that keeps the downstream service overloaded. The retry guard, the margin and the hop cap are three small rules that together prevent the cascade the SRE book describes as retries leading to even more overload.

Cancellation propagation

Propagating a shrinking budget is necessary but not sufficient. The SRE book describes two further cases. In hedged requests, a client sends the same request to a second instance after a delay and cancels the loser once one answers; those cancellations must propagate down the tree or both subtrees do full work. And when a critical RPC deep in the stack fails with an error that cannot succeed on retry, the initial call should be failed and its other branches cancelled, rather than running until the outer deadline expires. In Go this is the context cancellation that every function in the chain already accepts; in gRPC it is the server-side cancellation that the guide says the application must check for. Treat an outgoing cancellation as a first-class event with a metric, because a stack that propagates deadlines but not cancellations still does doomed work, just less of it.

Failure modes

• Absolute timestamps across skewed clocks. A downstream host whose clock is ahead sees an already-expired deadline and rejects everything. Pass relative budgets, as gRPC does.
• A margin that ignores queueing. The budget is checked when the request arrives, then the request sits in a queue. Check it again when work actually starts, which the SRE book calls checking the deadline left at each stage.
• Deadlines far above mean latency. The SRE example: a 100 second deadline and 5 percent of requests that never complete tie up 5,000 threads out of 1,000 and produce an 80 percent error rate. Fail fast on unavailable backends and keep deadlines within an order of magnitude of normal latency.
• Retries that start after the deadline. The retry guard exists for this; without it, every slow period produces a second wave of doomed calls.
• Noncritical backends with the full budget. A spelling-suggestion service that blackholes requests should be given a small cap or skipped, not the whole remaining budget.
• Checkpointed work killed mid-step. The SRE book allows an exception: check the deadline after writing a checkpoint, not after an expensive step, so that progress is not thrown away.

Verification

Unit-test the arithmetic with fakes: a chain of three handlers with injected latencies should produce the budgets in the table above, and a test with a skewed clock should pass because nothing in the chain reads wall-clock time from another host. Load-test the failure: slow one backend to twice the budget and confirm the upstream thread pool stays bounded, that no retries are issued, and that the metric for work finished after the deadline stays near zero. Log the remaining budget at every hop; when a request fails with a deadline error, the sequence of remaining values tells you which hop consumed the time, which is the question you will be asked.

Practice and related reading

Further reading:

• A2A timeout handling
• Load shedding in system design
• Tool-calling reliability: timeouts, idempotency and compensation
• gRPC load balancing on Kubernetes
• gRPC in ADK Java

Sources and scope

Checked against these sources on 18 October 2026. Examples and figures are illustrative, not measurements.

• gRPC documentation: deadlines and deadline propagation
• gRPC over HTTP/2 protocol specification (grpc-timeout header)
• Go standard library: package context
• Google SRE book, chapter 22: addressing cascading failures (latency and deadlines)