Everything returned success
The multi-level approval engine I built at VNTrip moves a request up a chain of levels, and it is deliberately paranoid about the move itself: the step is guarded by optimistic locking, the whole workflow tree is snapshotted at submit so a later config change cannot rewrite an in-flight request, and the engine documents 11 distinct error codes for the ways an approval action can legitimately fail.
Then a request advanced a level, correctly, and nobody found out. The approver at the next level got no mail and no bell in the dashboard. The request sat there, approved-at-level-N, waiting for a person who did not know they were being waited on. It surfaced the way these things do — someone asking, days later, why their request was stuck. It was not stuck. It was waiting, silently, on someone we had never told. How many requests aged this way before that question was asked, I do not know, and by the end of this piece it should be clear why nothing was in a position to count them.
Two correct designs and the gap between them
The state machine had done its job; none of the 11 error codes fired, because no error occurred in anything the state machine owns.
Notification dispatch, meanwhile, is deliberately post-commit and fail-soft. That is a design I still stand behind: a notification backend being down must never block approving. Approvals move money and time; a mail server outage should cost reminders, not business. So the dispatch runs after the transaction commits, catches its own failures, and lets the flow continue.
Each design is right. Together they have a gap, and the gap is exactly this incident. Fail-soft as I had written it meant the failure was not only non-blocking but unrecorded — swallowed, not logged as an outcome anything watched. "Do not block on failure" had silently expanded into "do not tell anyone it failed", and nothing in either design was responsible for noticing. The engine's success was real. The dispatch's failure was real too. No component owned the fact that both had happened to the same request.
There was also a second, quieter way to reach the same silence. "Who approves at this level" is resolved from configuration at dispatch time, and a lookup over configuration can return an empty list — a misconfigured level, a role nobody currently holds. The original code took the empty list and faithfully delivered a notification to each of its zero members. No failure occurred at all, in any sense the code recognised. This site already has a name for that shape: an empty result is the most expensive kind of success, and delivering perfectly to no one is that rule wearing a different coat.
What changed
Three things, none of which touched the state machine.
The dispatch records its own outcome. Post-commit, still fail-soft, but every attempt now writes down what happened — sent, failed, skipped — as first-class data the system keeps rather than a line lost in application logs. Fail-soft survives; what died is fail-silent. A notification that could not be sent is no longer an event that did not happen.
An empty approver resolution fails loudly. Resolving the next level's approvers and getting nobody is now treated as the configuration error it is, not as a very short recipient list. It cannot block the approval — same principle as before — but it lands as a recorded, alertable failure instead of a successful delivery to zero people.
An end-to-end test walks the whole flow and asserts the notification. On a test environment: submit a real request, approve it level by level, and assert at each step that the next approver actually received the mail and the bell. Unit tests already covered the state machine and the dispatch separately — both passed throughout this incident, which is the whole point. The only test that can catch a failure that lives between two components is one that refuses to know where the boundary is.
The rule
Fail-soft means "do not block". It does not mean "do not tell anyone". Every place you write a catch-and-continue, you are creating an outcome that no exception will ever report — so the recording has to be built in at the same moment as the swallowing, by the same hand. And whenever the recipient of an action is resolved from configuration, "nobody" must be a loud answer. Silence downstream of success is not evidence that anyone heard.