S3 Conditional Writes as a Coordination Primitive: Create-Once With If-None-Match, Compare-and-Swap With If-Match, a Two-Phase Lease and What an Object Store Cannot Give You

Why coordinate through an object at all

Several jobs on different machines need to agree on something small: whether a daily publication slot is still free, which of them is uploading right now, what the last published build was. The classic answers are a database row with a transaction, a lock service, or a leader election, and the Chubby lock service is the reference design for the heavy version. For a static site publisher or a nightly batch, those are a lot of infrastructure for a few bytes of state that already live next to the artefacts in S3.

S3 provides the two primitives that make a single object usable as that state. A PutObject with the If-None-Match header set to the asterisk value succeeds only if no object with that key exists, which makes creation exclusive. A PutObject with If-Match set to an ETag succeeds only if the current object still has that ETag, which makes an update a compare-and-swap. Both return a 412 Precondition Failed when the condition does not hold, and a 409 ConditionalRequestConflict when another conditional operation on the same key was in flight at the same moment, for which the documentation prescribes a retry. Reads are strongly consistent after a write in every region, so a client that reads after a 412 sees the value that beat it. Conditional writes require Signature Version 4 and apply to PutObject, CompleteMultipartUpload and CopyObject.

The compare-and-swap loop

Everything else is built from one loop: read the object and its ETag, compute the new value, write it with If-Match, and on 412 or 409 start again from the read. The loop terminates when the write succeeds or when the caller decides it has retried enough. The example below is the shape used by this site's own publication ledger, which caps automatic publications at two per day in India time and records an active run while an upload is in progress.

Illustrative python, checked for syntax; adapt names and storage to your system.

import json, secrets
from botocore.exceptions import ClientError

CONFLICTS = {"PreconditionFailed", "ConditionalRequestConflict", "412", "409"}

def read(s3, bucket, key):
    try:
        r = s3.get_object(Bucket=bucket, Key=key)
        return json.loads(r["Body"].read()), r["ETag"]
    except ClientError as e:
        if e.response["Error"]["Code"] in ("NoSuchKey", "404"):
            return None, None
        raise

def write(s3, bucket, key, value, etag):
    body = json.dumps(value).encode()
    if etag is None:                       # create exclusively: nobody else may have created it
        s3.put_object(Bucket=bucket, Key=key, Body=body, IfNoneMatch="*")
    else:                                  # update exclusively: nobody may have changed it
        s3.put_object(Bucket=bucket, Key=key, Body=body, IfMatch=etag)

def cas_update(s3, bucket, key, mutate, attempts=8):
    """mutate(value or None) -> new value, or raises to abort. Returns the committed value."""
    for _ in range(attempts):
        current, etag = read(s3, bucket, key)
        new = mutate(current)
        try:
            write(s3, bucket, key, new, etag)
            return new
        except ClientError as e:
            if e.response["Error"]["Code"] not in CONFLICTS:
                raise
    raise RuntimeError("ledger changed repeatedly; no reservation made")

def reserve_slot(s3, bucket, key, day, cap=2):
    run = {"id": secrets.token_hex(12), "day": day}
    def mutate(ledger):
        ledger = ledger or {"days": {}, "active": None, "history": []}
        if ledger["active"]:
            raise RuntimeError("another run is active or was interrupted; inspect before retrying")
        if ledger["days"].get(day, 0) >= cap:
            raise RuntimeError("daily cap reached")
        ledger["days"][day] = ledger["days"].get(day, 0) + 1
        ledger["active"] = run
        return ledger
    cas_update(s3, bucket, key, mutate)
    return run

Three details carry the correctness. The create path uses If-None-Match, because two clients that both see no object must not both create one; the loser gets a 412, re-reads, and now sees the winner's object. The update path uses the ETag from the read that produced the new value, never a cached one. And the mutate function raises to abort rather than returning the unchanged value, so a cap or an active lease is enforced before any write, not after.

The two-phase lease

A counter alone does not say who is working. The ledger therefore carries an active field: the reserve step sets it to the run's id in the same compare-and-swap that increments the day's count, and a finish step clears it, appends the outcome to a bounded history and records the digest of what was published. Finish is also a compare-and-swap, and it checks that the active id is the one this process reserved; if it is not, the process refuses to touch the ledger, because something else has already repaired or taken over.

The deliberate choice is what happens when the holder dies between the two phases. The lease stays active. There is no expiry, so no later client will overlap a still-running upload that merely looks dead, and no automatic unlock will erase the evidence of an interrupted run. An operator inspects the live objects and the interrupted run, then clears the lease by hand. That is the right trade for a publication ledger, where an overlapping upload is worse than a delayed one; it is the wrong trade for a system that must make progress unattended, which should use a store with expiry and fencing instead. The lab linked below runs several publishers against one ledger and shows what each choice does.

What the object store cannot give you

Bucket owners can also enforce conditional writes through bucket policy, so that an unconditional PutObject to the ledger key is denied. That closes the path where a script written without the headers overwrites the state everyone else is negotiating over.

Worked example: three publishers at nine in the morning

A scheduled job on a build server, a scheduled job on a laptop and an engineer running the publisher by hand all start within the same second. All three read the ledger and see one slot used today and no active run. All three compute a ledger with two slots used and themselves active, and write with the ETag they read. One PUT succeeds. The other two receive 412, re-read, and now see two slots used and an active run; their mutate functions raise, and they stop without uploading. If two of the writes had arrived at the same instant, one might have received 409 instead; it re-reads and reaches the same conclusion. Without the If-Match header, all three PUTs would have succeeded in some order, the ledger would show two slots used, and three uploads would be running against one bucket at once.

Now let the winner die after reserving but before finishing. The next day's job reads an active run from yesterday and refuses. The operator looks at the run id, finds the half-finished upload in the bucket listing, verifies the public pages, and clears the lease. The refusal cost a day; the alternative, a second uploader deleting what the first had half written, would have cost more.

Failure modes worth testing

• Unconditional writes anywhere. One script without the headers can corrupt the state. Enforce conditional writes by bucket policy and test that a plain PUT is denied.
• Stale ETag. Writing with an ETag from an earlier read than the one that produced the value re-introduces the lost update. Read and write inside the same loop iteration.
• Treating 409 as fatal. It means a concurrent conditional operation; retry from the read as the documentation says.
• No retry cap or jitter. Many clients spinning on 412 cost requests and achieve little. Cap attempts and back off.
• Expecting expiry. A dead holder blocks until someone acts. Decide in advance who acts, and alert on a lease older than the longest legitimate run.
• Multipart ETags. The ETag of a multipart object is not an MD5 of the content; treat it as an opaque token, which is all the compare-and-swap needs.
• Two keys for one decision. A counter in one object and a lease in another cannot be updated atomically. Merge them.

What to do next

1. Put the whole coordination state in one small JSON object; keep the artefacts elsewhere in the bucket.
2. Implement the read, mutate, conditional write loop with If-None-Match on create and If-Match on update, retrying on 412 and 409 with a cap.
3. Add the two-phase lease and decide, in writing, what happens to a lease whose holder died.
4. Enforce conditional writes on the key with bucket policy and keep the control prefix out of any public distribution.
5. Alert on an active lease older than your longest run, and on repeated 412 storms, which mean too many clients share the ledger.

Practice and related reading

Hands-on: Conditional-write ledger: publishers racing for two slots, an interactive lab on synthetic data.

Further reading:

• Amazon S3 architecture and operations
• The Chubby lock service
• Leader election in distributed systems
• Idempotency in system design
• Amazon DynamoDB: conditional writes and transactions
• REST API design deep dive, including ETags

Sources and scope

Checked against these sources on 19 October 2026. Examples and figures are illustrative, not measurements.

• Amazon S3 User Guide: How to prevent object overwrites with conditional writes
• Amazon S3 User Guide: Add preconditions to S3 operations with conditional requests
• Amazon S3 API Reference: PutObject (If-Match, If-None-Match, 412 and 409 responses)
• Amazon S3 User Guide: What is Amazon S3, the data consistency model