Skip to content
AyoKoding

Artifact: Two-Register Update — Notification Duplicate Incident

Duplicate-notification incident, written for two audiences -- exercises co-15.

Executive version: Last quarter, a subset of customers received duplicate shipment notifications after a routine worker restart. No data was lost and no customer action was required, but it eroded trust in our SMS/email channel. We are adding a safeguard (an idempotency check) that makes this class of failure structurally impossible going forward, shipping this sprint, at no added infrastructure cost.

Engineer version: The Notification Worker consumes from an at-least-once queue and processes each message without checking whether it already ran -- a consumer-group rebalance during a deployment restart replayed roughly 340 already-processed messages, and every one fired a duplicate SMS/email. The fix: a Redis-backed idempotency cache keyed on the event's event_id, with a 48-hour TTL comfortably longer than any plausible replay window. See this topic's capstone ADR and RFC for the full design.

Last updated July 15, 2026

Command Palette

Search for a command to run...