Ingesting From S3 Event Notifications: At-Least-Once Delivery, Out-of-Order Events, the Sequencer and Reconciliation

The contract you actually get

S3 event notifications are the easy way to react to new files: configure the bucket, point it at an SQS queue, SNS topic, Lambda function or EventBridge, and process each event. The Amazon S3 documentation states the contract plainly. Notifications are designed to be delivered at least once, typically within seconds but sometimes a minute or longer, and they are not guaranteed to arrive in the order the events occurred. If the destination is an SQS standard queue, that queue is itself at-least-once. A correct consumer therefore has to handle duplicates, reordering and, for safety, events that never arrive.

The sequencer

Events for object creation and deletion carry a sequencer field. For two events on the same object key, the one with the greater sequencer happened later. The documentation adds two rules that are easy to miss: sequencers cannot be compared across different keys, and they can have different lengths, so the shorter value must be left-padded with zeros before a lexicographic comparison. It also recommends what this design does: if you maintain a separate database or index of your objects, store the sequencer for each key as you process events.

Illustrative python, checked for syntax; adapt names and storage to your system.

def newer(a: str, b: str) -> bool:
    """True when sequencer a is later than sequencer b (same object key only)."""
    width = max(len(a), len(b))
    return a.zfill(width) > b.zfill(width)


assert newer("0055AED6DCD90281E6", "0055AED6DCD90281E5")
assert newer("0100", "FF")          # left-pad before comparing: "0100" > "00FF"
assert not newer("FF", "0100")

A per-key ledger

Keep one row per object key with the last applied sequencer, the event type and the version or ETag that was ingested. Every event goes through the same decision:

1. Decode the key. Object keys in notifications are URL-encoded; the documentation’s example turns red flower.jpg into red+flower.jpg. Use the decoded key everywhere, or a key with a space will never match the ledger.
2. Read the ledger row. If the event’s sequencer is not newer than the stored one, it is a duplicate or a late event: acknowledge it and do nothing.
3. For a PUT, fetch the object. S3 is strongly consistent for reads after writes, but by the time a delayed event is processed the key may hold a newer version; with versioning enabled, fetch the exact versionId from the event, otherwise compare the ETag and treat a mismatch as superseded by a later event.
4. Apply the effect to the target store and update the ledger row in one transaction, or make the effect idempotent by keying it on the object key and sequencer.
5. For a DELETE, record a tombstone with its sequencer rather than deleting the row, so a late PUT event for the old object is recognised as older.

The ledger update is a conditional write: set sequencer to S where sequencer is lower than S. Two consumers handling events for the same key concurrently then cannot both win, whichever finishes first.

Illustrative python, checked for syntax; adapt names and storage to your system.

import sqlite3
from urllib.parse import unquote_plus

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE ledger (key TEXT PRIMARY KEY, seq TEXT, kind TEXT)")

def pad(seq: str) -> str:
    return seq.zfill(32)                      # S3 sequencers are hex strings of varying length

def apply(event: dict) -> bool:
    key = unquote_plus(event["s3"]["object"]["key"])
    seq, kind = pad(event["s3"]["object"]["sequencer"]), event["eventName"].split(":")[0]
    with db:                                  # one transaction for the check and the update
        cur = db.execute(
            "INSERT INTO ledger VALUES (?, ?, ?) ON CONFLICT(key) DO UPDATE "
            "SET seq = excluded.seq, kind = excluded.kind WHERE excluded.seq > ledger.seq",
            (key, seq, kind))
        if cur.rowcount == 0:
            return False                      # duplicate or older event: acknowledge, no effect
        # ...write to the target store here, inside the same transaction if it is the same database
    return True

The upsert updates the row only when the incoming sequencer is greater, and the row count says whether this event won. Padding to a fixed width makes the comparison a plain string comparison in the database, which is what the documentation’s left-padding rule asks for.

Worked example: three events, two orders of arrival

Both arrival orders end in the same state: the object is gone. A consumer without the ledger would re-create the deleted invoice in the second order, because the PUT arrives after the DELETE it preceded.

Overwrites follow the same rule. Suppose v1 and then v2 of the same key are written seconds apart and the v2 event arrives first. The consumer fetches v2, ingests it and stores its sequencer; when the v1 event arrives it is older and is acknowledged without effect. Without versioning there is a subtler case: the v1 event arrives first, but by the time the consumer fetches the object the key already holds v2. The ETag in the event no longer matches the fetched object, so the consumer must not record v2 under v1’s sequencer. Skip it and let the v2 event, which is on its way, do the work, or fetch the exact version when versioning is enabled.

Reconciliation

At-least-once delivery covers the notification system, not your consumer. A deployment bug, a dead-letter queue nobody drained or a misconfigured filter can lose events for hours. Run a reconciliation job that lists the bucket prefix, compares each key and ETag with the ledger, and queues synthetic events for differences. For very large buckets, a scheduled S3 Inventory report can replace the listing. Reconciliation turns the pipeline from events are the truth into the bucket is the truth and events are the fast path, which is what makes it safe to replay, backfill and recover.

Failure modes

• Processing events in arrival order and trusting it, which resurrects deleted objects when a DELETE overtakes a PUT.
• Comparing sequencers as strings without padding, which orders them wrongly when lengths differ.
• Comparing sequencers across keys to order a batch, which the documentation says is meaningless.
• Fetching the current object for an old PUT event without checking the version or ETag, so an old event ingests new content under an old sequencer.
• Using the raw, URL-encoded key as the ledger key.
• No reconciliation, so the first lost event becomes a silent permanent gap.

Verification

• Replay a recorded batch of events in shuffled order, with duplicates, and compare the target store with the bucket.
• Unit-test the sequencer comparison with values of different lengths.
• Run reconciliation daily and alert when it finds differences; a non-zero count means events were lost or mishandled.
• Test keys with spaces, plus signs and non-ASCII characters end to end.

Practice and related reading

Further reading:

• Amazon S3
• Amazon SQS
• Exactly-once semantics
• The outbox pattern

Sources and scope

Checked against these sources on 22 October 2026. Examples and figures are illustrative, not measurements.

• Amazon S3 User Guide: Amazon S3 Event Notifications (delivered at least once)
• Amazon S3 User Guide: Event message structure (sequencer, versionId, eTag)
• Amazon S3 User Guide: strong read-after-write consistency
• Amazon SQS Developer Guide: at-least-once delivery for standard queues