Skip to content
Kien

The report that disagreed with the database

· 3 min read

1 number · 2 queries

A monthly report and the operational query returned two different totals for the same business number, and both were defensibly right. The problem was never the arithmetic — it was that one definition had two implementations, and nobody owned either.

  • data quality
  • reporting
  • ownership
  • sql
On this page
  1. Two right answers
  2. Why they drift
  3. The third definition
  4. How the argument ended
  5. What I took away

Two right answers

The question arrived the way these questions always do: why does the report say one thing and the system say another? A monthly total from the reporting side did not match what the operational query returned for the same period. Someone had to be wrong.

Nobody was. The report aggregated over the search index we keep for reporting, with its own filters and its own idea of which rows count — orders in certain statuses, timestamps bucketed its way. The operational query ran against the transactional store with a different status list and a different notion of when an order belongs to a month. Each side implemented a reasonable reading of the same business phrase. Two reasonable readings, two numbers.

How far apart the two totals were, in money, I never wrote down — the argument was about which one to trust, and once that was settled nobody went back to measure the gap. What I can count is the number of implementations of the definition: two, which is exactly one too many.

Why they drift

The two queries almost certainly agreed on the day the report was written. That is what makes this failure mode so quiet. Two implementations of the same definition drift the moment either one is edited — a status added to one filter and not the other, a timezone fixed on one side only — and a report is edited far more often than a schema. Every small, locally sensible change to the report was a silent fork of the business definition, reviewed as a reporting change, never as a definition change, because nothing marked it as one.

The report was the softer artifact, so the report moved. The operational query sat still. Months later they were measurably different, and by then no commit message on either side said "this is where we forked the meaning of revenue".

The third definition

While the two queries were being compared, a spreadsheet appeared — exported from one of the sides, filtered by hand to "clean it up", and circulated as the reference. That is the moment a two-way disagreement becomes a three-way one. An export is a snapshot of one implementation at one instant; the first manual filter applied to it creates a third definition, one that exists in no codebase, has no history, and cannot be re-run. From then on, part of the debate was about rows someone had removed in a spreadsheet and could no longer name.

How the argument ended

Not by debate. Arguments about aggregates are unwinnable at the aggregate level, because both sides are correctly describing their own query. The way out was mechanical: recompute both numbers from the source rows, keep the row sets, and diff them. The diff is a finite list of concrete orders, and every order in it can be judged individually — should this row count, yes or no? Each judged row either exposes a filter that is wrong or settles a rule that was never written down. An afternoon of diffing ended a disagreement that discussion had kept alive for much longer, though how much longer, honestly, I did not track.

The fix that lasted was not the corrected query. It was picking where the definition lives: one aggregation path that both the report and the operational view call, or a derived store that is rebuilt from source on a schedule and audited against it — either works, as long as there is exactly one, and edits to it are reviewed as what they are.

What I took away

A business number needs an owner and exactly one implementation. Every additional implementation is a future argument with no referee, scheduled for whenever the two next drift.

An export with a manual filter is a new implementation. It just has no source control and no author who remembers what they filtered.

Settle aggregate disputes at the row level. Recompute both from source and diff the row sets. Aggregates can only be argued about; rows can be decided.