At two in the morning the interface to the ERP failed halfway through a batch. Fourteen hundred invoices had posted; the rest had not; the job logged an error and retried, the way jobs do. The retry did not know the first fourteen hundred were already in. It posted them again. The morning was not spent investigating an outage — it was spent un-posting eight hundred thousand dollars of duplicate invoices by hand, while finance held the close and asked, reasonably, how this was possible.
It was possible because the integration was built for the day it works. The happy path was clean. The retry was somebody else's problem, filed under operations. But a retry is not an edge case you might hit. On any integration that runs long enough, failure is a guaranteed input — a network blip, a timeout, a restarted job, a duplicated message from the queue. The only question that matters is whether the second run is safe.
So we build the second run first. An integration that cannot survive being run twice is not finished; it is a demo that has not failed yet.
Idempotency, in plain terms
The word sounds like jargon and means something simple: running the same operation again produces the same result, not a second one. Post invoice 4471 once, you get invoice 4471 in the ledger. Post it five more times, you still have exactly one invoice 4471, because the system recognizes that it has seen this one before and does nothing the second through sixth times. The integration can fail anywhere, recover, and re-run from the top, and the books look identical to the run that never failed.
That recognition is the whole game, and it does not happen by accident. It requires that every unit of work carry a stable identity the receiving system can check against what it already holds — an idempotency key, a natural business key, a hash of the payload — and that the system check before it commits rather than blindly insert. Build on insert and every retry is a duplicate. Build on check, then upsert and retries are free.
What makes a run safe to repeat
Idempotency is the headline, but a genuinely re-runnable integration needs a few disciplines working together. These are the ones we will not ship without:
- A stable key on every record. Each unit of work carries an identity that does not change between runs. No key, no way to recognize a repeat, no safety.
- Check before you post. The system asks "have I already processed this one?" and acts on the answer. Upsert, don't insert; reconcile, don't assume.
- Deterministic ordering. The same inputs process in the same order every time, so a partial run and a full run leave the ledger in the same state.
- An audit trail that reconciles. Every attempt is logged with its outcome, so a human can prove what posted, what didn't, and what was a duplicate caught and dropped.
- A home for poison messages. The record that can never succeed is quarantined, not retried forever. One bad message should not block the queue or invent an infinite loop.
Notice that most of these are about what happens when things go wrong. That is the point. The happy path is the easy ten percent. The other ninety percent of a durable integration is the behavior under failure, which is exactly the part a demo never shows and a 2 a.m. outage always finds.
The cost of skipping it
Skip the second run and the failures do not stop — they just become manual. Every timeout becomes a cleanup. Every duplicate becomes a journal entry and an explanation. And something quieter corrodes underneath: people stop trusting the interface. They reconcile it by hand "just to be sure," which is the same as not having automated it at all, except now you are paying for the integration and the manual check both. An integration nobody trusts is a liability wearing the costume of an asset.
The same instinct runs through the rest of this series. It is why an ERP only accepts what reconciles and why, when two systems describe one customer, the mapping has to be explicit rather than assumed — a retry is only safe if the thing being retried has a stable identity on both sides.
It ties back to the ledger
The numbers tie out at close because the integrations behind them could fail and recover without leaving a mark. That is the standard: not that the interface never breaks — it will — but that breaking and re-running produces a ledger identical to the one where nothing went wrong. Built that way, an outage is a non-event. The job fails, the job re-runs, the books are unchanged, and nobody holds the close. The best integrations we have built are the ones no one remembers, because they never made anyone's morning.