Things I learned building and running production systems — each one a real problem, what it cost, and what I changed.
2 schedulers · 1 workload
· 4 min read
Two schedulers end up managing the same workload and neither knows the other exists. The concurrency nobody chose, a memory limit that belongs to the cgroup, and a crash loop hiding inside a pod that reports itself healthy — and why the fix is to pick one layer and let it do the multiplying.
200 OK · 0 hotels
· 4 min read
A support ticket: searching one destination returns no hotels while the regions around it are fine. Nothing was down — the destination existed twice in the mapping data and the hotels hung off the other row. The fix became a runbook command that has to count the results before it may say "fixed".
~40 commands · 2 companies
· 5 min read
Two teams, two companies, ~40 AI slash commands. What changed was not the automation — it was that a runbook and the tool that runs it finally became the same file, and that every dangerous one had to declare itself.
1 click · 2 bookings
· 4 min read
A booking call times out at the client and succeeds upstream. The client retries — reasonably, because from its side nothing came back — and the supplier now holds two reservations for one traveller. Why a transaction cannot undo it, why the HTTP verb has nothing to say about it, and what actually holds.
15 channels · 0 dependencies
· 6 min read
I extracted "post the error to a chat room" out of a dozen services into one published package. The interesting part was not the fifteen channels — it was realising that the library's job is deciding what is allowed to leave the process.
2 models · 1 contract
· 3 min read
Hotel, flight and car search run asynchronously — a queue, one job per supplier, a progress fraction, a 30-second deadline. Tour and transfer answer synchronously in a single process. The same public API contract sits over both, and the lesson is why it has to stay that way.
75 codes · 31 modules
· 3 min read
The same supplier integration gets resold under many brand codes — on the hotel side, 75 codes running on 31 modules. The tempting move is to copy the module per brand; what we run instead is aliases registered by code and a sparse settings table, with the failure mode that choice buys.
1.2 GB → ~200 MB
· 4 min read
An audit of a hotel engine's search path: why the worker kept getting restarted at the 3 GB mark, why the event loop stalled for 6–20 seconds per job, and the three changes that took Redis from 1.2 GB to 150–250 MB per search.
1 key · 2 tenants
· 4 min read
A key says two requests deserve the same answer, and every field left out of it says that field cannot change the result. In supplier aggregation the tenant, its supplier set and its markup all change the result — so a key built from the traveller's search parameters alone serves one tenant's prices to another, and nothing errors.
SELECT: runs · WRITE: asks
· 4 min read
The standard mysql client cannot reach the gateway database, so for a while every query was a small ceremony of workarounds. A single query script became the standard tool — and the interesting part is what got built into it: SELECT runs immediately, a write prints the hostname and the read_only flag and demands confirmation first.
XML → JSON · 0 DOM
· 3 min read
A large share of supplier APIs still speak SOAP/XML, and one search response can carry thousands of rate combinations. At that size the parser stops being a matter of taste — a declarative XML→JSON mapping against a DOM walk that builds the whole tree first, and why the template wins review after review.
1 date · ±1 day
· 4 min read
Travel runs on local civil dates, not instants. A check-in is a date at the property; a departure is a local time at the origin airport; a cancellation deadline is a local moment at the supplier. Store any of them as a UTC instant and something eventually renders a day early or late — for some users, on some dates, which is why it survives review.
callback + poll
· 3 min read
An external insurer confirms policy issuance asynchronously. The callback is the fast path — and also the path that can be lost, arrive twice, or arrive before our own transaction is visible. The scheduled poll is not a fallback; it is the actual answer. The callback just makes the answer arrive sooner.
+1 store · in sync since 2022
· 4 min read
A graph store for hotel identity and a search index for reporting both earned their place — the transactional store genuinely could not answer those queries. But each one is a copy, and a copy is a consistency debt with a repayment schedule you sign the day you add it.
3 services · 1 price
· 3 min read
Flight-delay insurance attached to a flight order: the insurance rows, the flight legs and the order total must all tell the same story, and they live in three services owned by three code paths. Each service was internally consistent. The disagreement only existed in the join — which nobody owned until we wrote the thing that computes it.
3 rounding sites · 1 rule
· 4 min read
Storing prices as integers in minor units takes an afternoon. The part that costs you is deciding where rounding happens and how many times — markup per room then summed is not markup on the total rounded once, and a traveller can end up reading a total that does not equal the lines printed above it.
level +1 · 0 notified
· 4 min read
An approval request advanced a level exactly as designed, and the approver at the next level never heard about it — no mail, no bell. The state machine was correct. The fail-soft notification dispatch was correct. The failure lived in the gap between two correct designs: "do not block" had quietly become "do not tell anyone it failed".
cert green · prod different
· 3 min read
A supplier's test environment passes the whole certification catalog, and production behaves differently — inventory that never sells out, errors that are not in the document, rate limits that exist on only one side, and latency you cannot tune a timeout against. What certification actually proves, and what the first real bookings are for.
routes, not credentials
· 4 min read
Two network ranges must both be reachable — staging and production. After a network change, routes can bind to the wrong interface, and the symptom is indistinguishable from bad credentials, a dead host, or DNS. What fixed the afternoons of guessing was not a command but an order: route, port, handshake, auth — each answer eliminates a whole layer.
1 number · 2 queries
· 3 min read
A monthly report and the operational query returned two different totals for the same business number, and both were defensibly right. The problem was never the arithmetic — it was that one definition had two implementations, and nobody owned either.
Showing 10 of 20