The ticket
A support ticket, relaying a partner: searching this destination returns no hotels. The regions around it are fine. Same account, same dates, same everything — move the search one region over and the results come back.
First instinct: something is down. Nothing was down. The region row the search resolved to was valid. The query built from it was valid. Every service in the chain — resolve the region, run the search, read the cache, shape the response — returned 200 and did exactly its job. The result was empty because the data was, and an empty array is a perfectly legal answer.
What was actually wrong
The destination existed twice. Same real-world place, two region rows in the mapping data — and the hotels were attached to the other row. The search resolved to the row with nothing hanging off it, joined it against an empty set, and returned precisely what the data said: nothing. The same family has a second member we have also met: only one row, but its hotel mapping set is simply empty.
Neither row is wrong on its own. Each one passes validation. The bug is not in any row — it is in the fact that there are two, and identity questions like this are exactly why resolving one identity wrong can blank an entire search region.
Why no alarm went off
Zero hotels throws no exception. It trips no alert, because every signal we watch is tuned to failure — non-2xx rates, timeouts, exceptions — and this is a success. In every metric we had, a broken region looks identical to a search for an island that genuinely has no hotels on it.
The design-review checklist on this site's decisions page already names the class: "What does the client see while this is failing — an error, or an empty result that looks like an answer?" That question is on the list because of tickets like this one. What the client saw here was an answer.
And the cost has the worst possible shape: every search against that region while it was broken was a customer reading "no hotels" and booking somewhere else. How many searches, over how many days — I do not know. Nothing counted zero-result searches per region against a baseline, which is exactly the gap this failure hides in. There is no lost-revenue figure to publish, and that absence is itself the finding.
The fix that existed before
An engineer who had seen it before hand-copied the mappings from the populated row to the searched one, flushed the search cache, re-ran the search in a browser, and eyeballed the page. It worked — whenever that engineer was around and remembered every step. It was also a sequence of writes to production mapping data held together by memory, ending in a check whose pass condition was "looks about right".
What changed
The procedure became a runbook command, and the shape of the command is the lesson:
- Clone the mapping set to the canonical row. Labelled as a write to production data, in the index, and run under the same protocol every write in the pack follows: dry run → print the plan → confirm → apply.
- Flush the affected cache keys. The affected ones, not the whole cache — and labelled too, because a flush is a write with a blast radius, not housekeeping.
- Verify: re-run the search and count the results. The command does not end with "done". It ends with a number, and the number has to be greater than zero before anyone is allowed to declare the region fixed.
Step 3 is the one that earns its place. The old procedure's last step was a human looking at a screen; the new procedure's last step is an assertion. "Fixed" stopped being an impression and became a count.
What I took away
An empty result is the most expensive kind of success. It throws no exception, appears on no error dashboard, and pages nobody — and every search it touches is a sale lost silently. A failure that announces itself costs you an incident. A success that carries nothing costs you revenue for as long as nobody happens to look.
A status code is not an assertion about content. 200 means "I did my job", and every service in this chain did. If an empty result is a business problem, then something has to assert non-emptiness — a count, a baseline, a check at the boundary. Nothing does that by default, in any stack I have worked on.
A repair is not done until the repairing tool re-tests it. The verify step costs one extra search. Skipping it is how a fix gets declared while a cache somewhere still holds the empty answer — and how the same region comes back as next week's ticket.