Skip to content
Kien
ENVI
All lessons

"It is the network" is a hypothesis until three commands make it a fact

· 4 min read

routes, not credentials

Two network ranges must both be reachable — staging and production. After a network change, routes can bind to the wrong interface, and the symptom is indistinguishable from bad credentials, a dead host, or DNS. What fixed the afternoons of guessing was not a command but an order: route, port, handshake, auth — each answer eliminates a whole layer.

  • networking
  • debugging
  • diagnostics
  • tooling
On this page
  1. Two ranges, both required
  2. The symptom lies
  3. The order
  4. What changed
  5. The rule

Two ranges, both required

Day-to-day work here needs two separate network ranges reachable at the same time: the staging databases live in one, production in the other. On a laptop both sit behind the same connection, and on a good day both just work.

After a network change — a reconnect, a sleep and wake, switching from one network to another — they sometimes do not. The routes for one range, or both, can end up bound to the wrong interface. Packets for the staging range leave through an interface that has no idea where that range lives, and nothing ever comes back.

The symptom lies

Here is what makes this failure expensive: from where you sit, it is indistinguishable from at least three other problems. The connection hangs, or it is refused. That is the same face shown by wrong credentials, by a host that is genuinely down, and by DNS handing you a stale answer. The client library reports whatever its timeout felt like reporting, and the error string names none of the layers underneath it.

So the natural response is to guess. I have re-typed passwords that were correct. I have asked whether the database was down when it was serving everyone else fine. Guessing eliminates nothing: after each guess you know exactly as much as you did before it, minus the time.

The order

What changed the shape of these afternoons was not a clever command but a fixed order — four questions, asked cheapest first, each one eliminating an entire layer:

  1. Is the route in the table, bound to the interface you think it is? One look at the routing table. If the range routes out the wrong interface, stop — you have found it, and nothing above this layer is worth touching.
  2. Does the port answer at all? A bare TCP connect, nothing more. If nothing is listening or nothing is reachable, credentials are irrelevant and so is everything else you were about to try.
  3. Does the handshake fail? You reached something — does it speak the protocol you expect, or did you reach the wrong thing?
  4. Does authentication fail? An auth error is good news. It proves the route, the port and the handshake in one line, and narrows the problem to credentials and permissions — the one layer a password can actually fix.

Each answer kills one layer. The expensive moves — rotating credentials, restarting services, filing a ticket with ops — come after the cheap ones, and usually never, because the route check already answered the question.

What changed

The order became a command. It checks both ranges, reports which one is broken — staging, production, or both — and then prints the exact fix: the privileged route commands, ready to paste, that rebind the broken range to the right interface.

Deliberately, it prints them rather than running them. Rewriting the route table needs root, and a diagnostic tool that escalates itself into an operator is a different and more dangerous kind of tool. So the split is explicit: the tool establishes the facts, and the human runs the one privileged line with their own hands and their own sudo. The human still decides; they just no longer have to remember.

I cannot tell you how many hours the old guessing cost — nobody logs a guessing session, and I will not reconstruct a number out of remembered irritation. What I can say is that the failure keeps coming back, because it is a property of how the network is arranged rather than a one-off, and that the diagnosis went from an open-ended search to a fixed, small cost you pay once and move on.

The rule

"It is the network" is a hypothesis until three commands make it a fact. So is "it is the credentials", "the database is down", and "DNS is broken" — every one of them feels certain from inside a hanging connection, and every one of them is checkable for less than the cost of acting on it. Separate the layers cheapest first, let each answer eliminate one of them, and when the fix needs privileges, hand the human the exact command instead of the keys.