Idempotency: the boring fix that prevents the worst automation bugs

Ask about the worst automation incident a team has had, and the root cause is very often the same shape: a job ran twice. A retry fired after a timeout that had actually succeeded. A scheduled job overlapped with a manual re-run. A webhook got delivered twice by the sending platform, which is allowed and expected, not a bug on their end. In every case, the pipeline wasn't built to be safe when that happened, and the second run duplicated data, double-charged a customer, or sent the same email twice.

Idempotency is not an edge case, it's a guarantee

Idempotent means running the operation twice with the same input produces the same result as running it once. It sounds like a nice-to-have until you consider how often 'run it twice' actually happens in production: network retries, at-least-once delivery guarantees on most webhook and queue systems, a well-meaning teammate manually re-triggering a job that looked stuck. If the pipeline assumes it only ever runs once, all of those become incidents.

• Use a stable idempotency key: an order ID, an invoice number, something that identifies the operation, not just the timestamp it ran.
• Make writes upserts, not inserts, wherever the downstream system supports it.
• Check before you act: has this exact operation already been recorded, before sending the email or issuing the charge.
• Design retries to be safe by default, not fast by default. A slightly slower retry that checks state first beats a fast one that assumes nothing changed.

It's cheap to build in, expensive to retrofit

Adding an idempotency key to a new pipeline is a small amount of extra code at design time. Retrofitting it after an incident means auditing every record that might have been duplicated, writing one-off cleanup scripts, and explaining to a client why their customer got charged twice. The asymmetry is stark enough that we treat it as a default, not an optional hardening step reserved for high-stakes pipelines.

A pipeline that only works if it runs exactly once has a bug. It just hasn't run twice yet.

Test it by actually running it twice

The most reliable way to know if a pipeline is idempotent is to deliberately run it twice against the same input in a test environment and check what happens. If the second run isn't a safe no-op, that's the fix to make before launch, not the incident to explain afterward.