The check that was true when I wrote it
The usual disclosure: an AI agent did the hands-on diagnosis below, and nearly filed the false outage below. The machine is the two-Spark cluster from earlier posts, currently serving a large mixture-of-experts model split across both boxes. This post is about the morning we proved it was healthy — over the objection of three of our own instruments.
The report
The owner said the serving pair had been wedged since the night before. Slower queries, then some that never came back at all.
That's a good report — specific, timestamped, two distinct symptoms rather than a feeling. And both halves were true. A trivial twelve-word request took 240 seconds to come back. An identical one sent earlier, with a three-minute ceiling, had returned nothing at all: it ran out of clock.
So: slow, and some never finish. Confirmed on the first two probes.
Wedged and saturated are different words
Then the agent watched the engine's internal completion counter. It moved — 834, 835, 836, 838, a steady eight steps per second, never pausing.
A wedged server doesn't do that. A wedged server's counters are flat.
This fork matters because the two diagnoses have opposite remedies. Wedged means restart it. Saturated means a restart buys you an hour of looking fixed and then hands the problem straight back — plus twenty minutes of model loading on a box this size. Accept the word "wedged" along with the true symptoms it arrived with, and you restart, watch the queue drain, report success, and are wrong again by dinner.
The actual cause was one line of scheduler configuration. The serving stack we track upstream ships a policy — while a sequence is decoding, don't mix another request's prefill into that step — and it was set to its most extreme value: admit nothing while anything is generating. Not "admit slowly." Nothing. The concurrency the config claimed was dead letter; the server was effectively single-file. The twelve-word request wasn't slow. It was queued behind someone's novel, on a machine whose cache sat at thirteen percent with room for everything.
To be fair to the setting: it's a deliberate upstream workaround for a real defect, trading throughput for the stability of the request in front of you. A legitimate trade — that nobody downstream knew existed.
The counter that reads zero in both worlds
While proving the pair healthy, the agent nearly filed an outage on it.
The two boxes split the model, so they talk constantly over a dedicated high-speed link. The link's network byte counters, read during active generation: zero. Zero in, zero out, ten straight seconds, while the thing was demonstrably producing tokens. That reads as the second box has dropped out and the first is quietly serving alone — a real failure mode for this topology, one we have a ledger entry about.
It was wrong. That link carries its traffic over a protocol that bypasses the kernel's network stack entirely, so those counters read zero on a healthy link and a dead link alike. The hardware's own counters, same ten seconds: 135 MB/s, symmetric, continuous. Eleven and a half terabytes since boot.
The question that saved it — ask it about every reading you're about to act on: if I were wrong, would this output look any different? Here: no. So it was never evidence, in either direction — it would have been exactly as worthless showing traffic.
The one that stings
When this model went live two days earlier, we verified the second box was genuinely carrying its half. The evidence, written into the ledger: the worker process held 114 GiB. And whoever wrote it did the thing we preach — recorded the contrast: a collapsed single-box fallback reads about 3 GiB, measured minutes earlier on the same machine. The check could demonstrably fail. Textbook. Our own standard, met.
The morning of the false alarm, the healthy pair's worker read 3.14 GiB.
Nothing was edited. Nobody broke anything. The 114 GiB was completely real — it was host memory during weight loading, which streams through and releases as the weights settle into the accelerators. A transient, faithfully recorded, and filed as if it were a standing property — under a heading that literally means "re-run this later to confirm."
Our own written rule for that ledger says a safe-condition must be a re-runnable check, not a snapshot claim. We wrote the rule. We then wrote a snapshot claim underneath it — with a contrast attached, which is exactly what made it look rigorous enough to stop questioning. Anyone re-running the check as written gets the number we ourselves labeled as the failure signature, on a healthy system, with our own documentation vouching for the alarm.
What the lesson actually is
I'd assumed the danger of a decaying check was that it goes quiet — stops noticing. This one got louder. It doesn't fail to alarm; it alarms on a healthy system.
And a false alarm is not the harmless direction, however much it feels like the safe kind of wrong. It sends someone to restart a service that didn't need restarting, on a box where a restart costs twenty minutes. And it burns the one thing you can't rebuild quickly: your willingness to believe the next alarm.
Three instruments called this pair broken in one morning — the memory footprint, the network counters, and the word "wedged." All three pointed at a real problem (requests genuinely weren't finishing) and all three named the wrong thing.
So, an amendment to our own rule:
"Prove it can fail" is a test you schedule, not a box you tick. We proved that ledger check could fail on the day we wrote it. The proof had a shelf life, and nothing in our process gave it an expiry date. A check earns its authority from the last time somebody demonstrated it discriminates — not from the first.