Eleven things that told us they were fine
The usual disclosure: AI agents did most of the hands-on work below, and made several of the mistakes below. We had a hardware weekend — cards moved between machines, two used workstations joined the rack, three brand-new 30B-class models got benchmarked. The interesting part isn't the hardware. It's that the weekend produced eleven separate cases where the obvious check said everything was fine, and everything was not fine. This is a running theme on this blog (a green test makes four claims, prove it can fail) — but I've never had eleven specimens land in one sitting before, so here they are, as a field guide.
1. The model server that served from RAM, with a green healthcheck
After a GPU swap, a config file still pinned the old card's device ID. CUDA enumerated zero devices — and the inference server did not fail. It logged one initialization error, loaded the entire model into system RAM, and served. The service manager said active. The log said "model loaded". It was bound to the right address. The health endpoint returned 200. It returned real, correct completions. Nothing anywhere was red — it was just an order of magnitude slower than it should have been.
The weights are ~9 GB and they have to be somewhere. What actually caught it was watching where they went: broken, the host process was huge and the GPU was empty; fixed, the host process was small and the GPU held the model. A refinement we learned the hard way a day later: host memory size alone doesn't discriminate — the server maps the model file, so host memory climbs with uptime on a perfectly healthy box, and a naive "RSS too big = broken" gate flunked a machine that was fine. GPU memory is the half that discriminates. VRAM is a function of correctness; RSS is a function of uptime.
2. A broken instrument producing a confident red
A port sweep reported three services down. The probe used bash's /dev/tcp — and the shell
running it was zsh, which doesn't have /dev/tcp. Every probe failed identically, regardless of
target. Three "outages", zero outages.
One line exposed it: point the same probe at a service you can see working. When the known-good target also read "down", the finding stopped being about the services. A red from an instrument that cannot succeed is not a finding — run the positive control before you believe a red.
3. A four-hour false alarm, manufactured by a timezone
"Monitoring has been dead for four hours." It had run six minutes earlier. One machine in the mix logs local time; the rest log UTC; the offset between them is exactly the size of a plausible outage. Nobody would ship a clock that's four hours wrong — but a mixed fleet of correct clocks produces the same reading.
4. Putting words in the owner's mouth — in the permanent record
Closing out a card swap, an agent wrote that the removed GPU went "to the spare shelf (owner, on site)". The owner never said that. The agent pattern-matched it from a previous entry where he had said it. The card was actually installed in another machine — caught only because it showed up enumerating somewhere else and was recognized.
The rule that came out of it: a removed part's destination is an observation, not an inference. If nobody stated where it went, the record says "disposition unstated". An honest gap in the record beats a plausible fabrication, every time, because the gap invites a question and the fabrication answers it wrongly.
5. The logs said dead. The owner's eyes said frozen.
Two machines went silent mid-build. The logs supported abrupt power loss: no shutdown record, journal ending mid-sentence. The owner walked over and looked: both powered, link lights on. They were frozen, not dead — which is a completely different fault with a completely different fix.
Here's the trap: both faults produce byte-identical logs. A journal that ends mid-sentence is what power loss looks like and what a hard freeze looks like, because in both cases the last entries never reached disk. Only a human looking at the chassis could tell them apart. Some diagnostics cannot be done from a terminal, and knowing which ones is part of the job.
6. The heartbeat nobody built
Ruling out a network blip during that freeze was possible for a dumb reason: two machines on the segment log a rejected broadcast packet every couple of minutes, like clockwork. That unbroken cadence across the exact failure window was a passive liveness monitor for the network segment — one that nobody designed, nobody maintains, and nobody knew existed until it was needed. Worth auditing your own logs for: you may already be running instruments you never built.
7. The interrupted build that "resumed"
A power cut mid-compile left truncated object files on disk. The build tool judges staleness by timestamp, saw the half-written files as up to date, and linked against them — compiling cleanly all the way to the final link, then dying in a pile of undefined references that looked like a source-code problem. After an unclean stop, artifacts with a plausible timestamp are not artifacts. Clean and rebuild; the hour you save resuming isn't yours to keep.
8. Same weights, different runtime, different correctness
One of the new models produced corrupted output on one inference stack: control tokens leaking into the text, the prompt echoed back, generation never terminating. The same weights file, moved to a different runtime, produced clean, terminating output every time.
First write-up blamed the model. That was wrong, and the way it got un-wronged is the method worth keeping: hold the model constant and vary the runtime (fault follows the runtime), then hold the runtime constant and vary the model (a different model on the same runtime is clean). Two controls bracket the fault: this specific model's chat template, on this specific stack. The distinction matters operationally — "bad model" means don't use it anywhere; "bad template handling on stack X" means the model is fine one runtime over. We nearly routed around a perfectly good model forever.
9. Three models, three incompatible ways to turn "thinking" off
Reasoning models spend tokens thinking before they answer. Three models in the same test wanted that disabled three different ways: a launch flag, a chat-template parameter, a magic string in the system prompt. A config that works on one is silently inert on the next — and the failure mode is not an error. It's an HTTP 200 with an empty answer, because the model spent its entire token budget thinking and never got to speak.
Verify by output, never by config presence. If your acceptance check is "the setting is in the config file", you have a check that cannot fail. Ours is now "a short prompt returns non-empty content with a normal stop reason" — which is also the check that catches it when a runtime accepts your parameter and quietly drops it, which we also found happening.
10. The benchmark that ranked two models correctly — and uselessly
Nine benchmark cells, matched quantization, matched method, every number validated against the memory-bandwidth ceiling. By the book. Two of the dense models landed within 4% of each other: call it a tie.
Except the benchmark ran every model bare, and nobody deploys these models bare. One of the two ships a working speculative-decoding drafter — a small helper model that accelerates the big one — measured at ~1.6× in the same harness. The other has no drafter at all. In deployed configuration the "tie" is one model running two-thirds faster than the other. The benchmark wasn't wrong; it measured a configuration nobody would run. A rigorous method holds some configuration constant — and if it's not the configuration you'll deploy, rigor is doing the misleading.
11. "Which model is fastest?" isn't even a well-formed question
The follow-up made it worse. Whether that drafter exists depends on the runtime: model A has a published drafter on one stack and none on the other; model C has one on the other stack and none on the first. In both directions, the accelerable model is decided by where you run it. Any "fastest model" claim that doesn't name the stack is incomplete in a way that can invert the ranking.
The method that kept the weekend honest
One habit separated real numbers from noise all weekend: compute the ceiling first. A model's generation speed is bounded by memory bandwidth — bytes that must move per token, divided by bytes-per-second the hardware can move. Every one of the nine benchmark cells landed at 89–93% of its computed ceiling. That's how you know the setup is right without trusting any single number: nine cells agreeing with arithmetic is hard to fake.
It also told us where to dig. One machine came in at 69% of its ceiling — that gap is a finding (its runtime has no drafter path and a fatter quantization, and it's now scheduled for a role change). And it settled two perennial arguments for free: two machines with identical GPUs but different CPU, RAM, PCIe generation, and CUDA versions benchmarked within 1% of each other — because generation is GPU-bandwidth-bound and essentially nothing else moves it. Meanwhile a mixture-of-experts model with 3.5B active parameters ran ~5× faster than a dense 27B on identical hardware, exactly as the arithmetic predicts. The expensive-looking variables didn't matter; the architecture did.
What eleven specimens in one weekend teaches
None of these checks was stupid. Health endpoints, port probes, service status, build timestamps, benchmark harnesses — this is the standard toolkit, correctly applied. Every one of them answered the question it was actually asked, and the question it was actually asked was narrower than the question we thought we were asking.
The test that keeps earning its keep: if this were broken, would the output look any different? The healthcheck in №1 — no. The config-presence check in №9 — no. The benchmark in №10 answered a question nobody deploys. Ask it of your own green checks before the weekend when eleven of them answer at once.