Operate parallel workers without losing partial failure

Pattern 05 companion: Python example · Behavioral tests · Example license

Monitor the expected set, not just returned work

A collector that received two results cannot determine coverage unless it knows how many tasks were expected. Record the worker set at admission and retain each task's terminal state. Display completed, failed, timed-out and cancelled counts with the run identifier. The fixture has completed results and failures only; timeout and cancellation are future service capabilities. Avoid adding those labels to a dashboard before the implementation can distinguish them reliably.

Measure time to first result, time to required coverage and time until all underlying work stops. These can differ substantially. A fast first answer may be useful for an operator but should be marked partial. If the product publishes an early assessment, make later updates explicit revisions rather than quietly replacing evidence. An operator responding to the first report needs to understand what information was absent when that report was produced.

Architecture and request sequence

Component diagram for Operate parallel workers without losing partial failure — Original reference architecture by Sandeep Belgavi. Arrows show the responsibilities and transitions described below.

Request sequence for Operate parallel workers without losing partial failure — Illustrative sequence for this design; failure and recovery variants are explained in the article.

Inject slow, invalid and contradictory workers

Use one worker that returns an invalid value, one that raises an exception and another that produces a valid result. Verify that valid evidence remains available while the aggregate is incomplete. Then use two valid but contradictory results and check that conflict is preserved. The supplied tests exercise these classes without external services. In a live-adapter test environment, add delayed responses and connections that ignore cancellation to test actual resource cleanup.

Include a worker that finishes after the parent deadline. The current demo has no deadline, so this belongs to the production extension. Decide whether the late result is discarded, stored for review or used in a new report revision. Test that it cannot modify an already approved action without authorization. A late observation may improve understanding, but it should not retroactively change the evidence on which an operator made an earlier decision.

Keep independence and agreement measurable

Two workers may share a telemetry backend, retrieval cache or model prompt. Their agreement is then less independent than their names suggest. Record source identity and method revision, and test shared-failure scenarios. For example, a stale cache can make both metrics and logs appear normal even when the service is degraded. A third copy of the same data does not solve that problem. The architecture needs a genuinely different observation or a freshness check.

Evaluate the aggregate policy against labeled incidents, not just individual worker accuracy. A conservative incomplete status can protect quality but increase review load. A quorum rule can improve availability while concealing a missing critical signal. Compare the tradeoff using the application's consequences of missed failures and false alerts. Preserve the original per-worker records so policy changes can be evaluated offline without rerunning every expensive observation.

Control concurrent cost across incidents

Set a global limit for each shared dependency in addition to the per-run worker limit. Ten runs with two threads each still create twenty simultaneous operations. Queue admission should account for current downstream capacity and incident priority. If a telemetry service is unhealthy, launching more parallel reads can make recovery harder. A pause mechanism should stop new fan-out while retaining completed evidence and giving operators a clear reason for delayed coverage.

Measure total worker time, provider usage where applicable, storage volume and review effort per useful report. Faster wall-clock completion can cost more because every worker runs even after an early result seems persuasive. If early termination is introduced, identify which observations are optional and verify that cancelled work actually stops. Do not count avoided waiting time as avoided compute unless the downstream operation was genuinely prevented or terminated.

Define release gates and incident fallback

Before a controlled rollout, require that every submitted task has a visible terminal record, no worker exceeds its authorized scope, aggregation is reproducible from stored inputs and failures never disappear from coverage. Test the actual shutdown path under load. A process that stops accepting work but leaves all threads blocked has not demonstrated graceful termination. Keep a manual route for operators to query the underlying observations when the collector is unavailable.

Promote aggregation changes separately from worker changes where possible. Otherwise a changed report may result from different evidence, different interpretation or both. Use a shadow run to compare policies on the same inputs and examine disagreement cases manually. Rollback should preserve previous reports and their policy revisions. The goal is an explainable collection process that remains useful during partial failure, not an aggregate label that always looks decisive.

Sources and operational evidence

The original concurrency tests demonstrate the fixture's result accounting and active-thread bound. The design article links Python's executor semantics. The operational recommendations are original proposals, with measurement prerequisites in Evals. No performance benchmark, distributed recovery guarantee or production cancellation capability is claimed from these local tests.