Engineering notes
Recover missing inventory events with a snapshot watermark
By Sachidananda Singh ·
An inventory consumer can have an empty sequence gap list and still be wrong. If the missing event is the newest event, no later sequence has arrived to expose the gap.
The inventory event reconciliation lab makes that failure visible with a synthetic trace. Its source applies 5 changes, adding 10, removing 3, adding 4, removing 2 and adding 5. The source finishes at 14 units. A consumer that has received only the first 4 changes has 9 units and no gap inside its observed sequence range.
A quantity needs a boundary
A snapshot reports the source quantity together with a watermark identifying the last source event it includes. In this example, the snapshot says 14 units through sequence 5.
The consumer uses the snapshot as its baseline and adds only accepted events with sequences greater than 5. An event at sequence 5 is already represented in the baseline, even if its individual delivery arrives later.
Without that boundary, setting the quantity to 14 and then adding the late receipt of 5 would produce 19. The watermark tells the consumer which changes the snapshot already covers.
Keep covered events available for validation
The lab retains events covered by the snapshot in its ledger. Their deltas stop contributing to the quantity, while their identities remain available for duplicate and conflict checks.
A repeated event therefore stays harmless after reconciliation. A different event claiming an occupied sequence is rejected. An older snapshot cannot replace a newer one.
This approach depends on a coherent source contract. The snapshot quantity must actually represent all changes through its stated sequence. A watermark copied from a different SKU or taken independently of the quantity could give a precise looking boundary to an inconsistent snapshot.
Read the trace before drawing a conclusion
At the snapshot observation, the naive consumer has 19 units, the deduplicated consumer has 9 and the reconciled consumer has 14. The source has 14.
Later, the delayed receipt arrives. The deduplicated consumer reaches 14, while the reconciled consumer stays at 14 because the snapshot already includes that receipt. Both end correctly in this trace, but reconciliation repairs the discrepancy earlier.
These observations don't measure throughput or establish a general advantage for every architecture. The example deliberately supplies a trustworthy snapshot. Obtaining one can be a substantial operational problem.
Use the lab as a design exercise
Inspect the snapshot recovery and stale snapshot tests, then run the demo. Explain why the gap list is empty before the final receipt and why that receipt contributes no additional quantity after reconciliation.
Before applying the pattern elsewhere, identify who owns the source sequence and how the snapshot is produced consistently. Decide how long event identities must remain available for retries. Physical counting errors require their own detection process because reconciling two software records cannot discover an event neither record contains.
Prepared with OpenAI Codex assistance from the book's reconciliation themes and the released synthetic lab.
Code and companion reading
Run the lab and inspect its tests