Engineering notes
A retry can add inventory twice
By Sachidananda Singh ·
A receiver records a receipt of 10 units. Its acknowledgment gets lost, so the sender retries the same event. If the receiver adds every delivered quantity, the inventory becomes 20. The extra 10 units exist only in the record.
This example connects the messaging discussion in Supply Chains Are Distributed Systems with the released inventory event reconciliation lab. The data is synthetic. It illustrates a failure mechanism without reporting an employer incident or production result.
Give the event a stable identity
In the lab, a receipt carries an event identity, a SKU, a source sequence and a quantity change. A retry preserves that identity. The receiver stores the event in a SQLite ledger and derives the inventory quantity from accepted events.
The first delivery adds 10 units. A second delivery with the same identity and contents leaves the quantity at 10. If the same identity arrives with different contents, the receiver rejects the conflict. Silently treating a changed quantity as an ordinary retry would hide a disagreement about the underlying event.
Generating a fresh identity on each delivery would defeat this check. The identity belongs to the event being retried.
Preserve the record across restarts
An in memory set of identities disappears when a process restarts. The lab includes a test that closes and reopens a database before replaying an event. The stored identity still prevents another addition.
A production integration also needs to coordinate its acknowledgment with durable local processing. The example has no message broker or acknowledgment protocol. Its SQLite transaction protects a local write, which is only part of a complete delivery design.
Separate duplicates from missing events
Deduplication handles an event that arrives again. An event that never arrives needs a different recovery path.
In the fixed demo, the source finishes with 14 units. Before a late receipt arrives, the deduplicated consumer has 9. Its record contains each observed event once, yet it is missing a change worth 5 units.
An authoritative snapshot repairs that discrepancy before the delayed receipt arrives. The companion article explains how a source sequence watermark prevents the late receipt from being added on top of that snapshot.
Try the example
Run the repository's test suite and demo using the instructions in its README. Inspect the duplicate receipt row. The naive consumer reports 20 units and the deduplicated consumer reports 10. Then inspect the late shipment row, where both consumers still differ from the source.
The model uses additive changes and one authoritative sequence per SKU. Reservations and conditional actions need additional rules. A correct event ledger can still disagree with physical stock if the source never recorded what happened.
Prepared with OpenAI Codex assistance from the book's messaging themes and the released synthetic lab.
Code and companion reading
Run the lab and inspect its tests