Then our own instruments lied

By Daniel S. · July 24, 2026

A follow-up. The first post was about published numbers that didn't survive scrutiny. This one is about ours — three broken gauges in one evening, and the one that nearly had us accusing a vendor's code of corrupting output.

One thing up front: the benchmarking below was run by one of our agents — this blog has always said an AI does much of the work here. It built all three of the instruments in this post, and it broke all three. Those are its mistakes and they are written as its, not mine. On a blog about unstated conditions, quietly absorbing someone else's confession would be its own small version of the problem.

1. It wasn't the machine we assumed it was

For the whole first day we called it a DGX Spark and left it there. It's an Acer Veriton GN100 — an Acer chassis wrapped around NVIDIA's reference baseboard, running NVIDIA's DGX OS. GB10 is a reference design; several vendors ship branded variants of it. The first post identifies it correctly because the identification got checked before that post went out — not because anyone knew on day one.

There were two tells sitting in plain sight the whole time. The account the out-of-box setup created has the vendor's name in it. The hostname starts with gn100 — which is Acer's product line, not an NVIDIA board code, which is what the agent assumed when it read it. It looked at both strings and inferred rather than checked. Someone else eventually pointed it out.

It matters in exactly one direction. Everything about the software transfers unchanged — the driver pinning, the update failure, the disabled autotuner, the memory arithmetic. All of that is NVIDIA's stack on identical silicon. What doesn't automatically transfer is sustained thermal behaviour, because cooling and power delivery belong to whoever built the box.

Which was awkward, because every performance number in the first post was measured in three-second bursts. We had no idea whether they survived minute two.

So we ran it under continuous load for six minutes:

        idle    ->  steady state
temp    42 C    ->  60 C     (last four samples all 60, spread 0 C -- PLATEAUED)
clock   2411    ->  2405 MHz (2411 is the IDLE reading; under load it held 2405 -> 2405,
                              flat for the whole run -- zero downclock)
power   10.9 W  ->  33 W
tok/s           ->  72.8     (-0.3% from start, within noise)

It plateaued, which means the steady state is quotable rather than a snapshot of something still climbing. Our numbers are sustained, not burst. The chassis doesn't throttle this workload.

And there's a detail in that table worth stopping on: 33 watts at 96% GPU utilisation.

That's the memory-bandwidth wall showing up in the power meter. "Utilisation" means kernels are resident on the GPU, not that the arithmetic units are busy. During single-stream decode the compute units spend most of their time stalled, waiting on memory. There's nothing to throttle because there's barely anything running hot.

Which also means we haven't answered the chassis question at all — only the easy half of it. Prefill and high-concurrency work are compute-bound and would draw real power. We didn't test those. A six-minute decode soak generalises to nothing beyond decode.

2. Gauge one: the benchmark that reported a 2× slowdown

We turned on speculative decoding — the technique where the model drafts several tokens ahead and verifies them in a batch, so accepted drafts come nearly free.

Our benchmark said throughput halved: 36.4 tokens/sec against a 72.9 baseline. Reproducible. Tight error bars, ±0.19. A clean, confident, catastrophic result for a feature whose entire purpose is going faster.

It was wrong. The feature was delivering +34% to +63%.

Our benchmark counted streaming chunks and called them tokens. Without speculation those are the same thing — one token per chunk — so the instrument had been correct for every measurement we had taken all day. Turn speculation on and accepted draft tokens arrive batched, several to a chunk. The counter undercounts by exactly the number of tokens in an accepted chunk — the acceptance length, 2.65 and 3.25 below, not the acceptance rate.

36.43 chunks/s x 2.65 tokens/chunk = 96.5    (measured properly: 97.81)
36.32 chunks/s x 3.25 tokens/chunk = 118.0   (measured properly: 118.78)

What caught it was not the number looking wrong. It looked plausible and it was reproducible. Two things disagreed with it:

The inference server's own acceptance metrics reported 59–85% draft acceptance — arithmetically incompatible with a slowdown.
Prose and code came back nearly identical, 36.43 and 36.32 — and the entire reason we ran both was that they should differ. The instrument was flattening the signal we were looking for.

An instrument that is correct across your whole baseline can be silently wrong the moment you change the thing you're measuring. That's not a rare failure. That's the normal one.

3. What speculative decoding actually did — and the first post's correction

With the counter fixed:

	prose	code
baseline	73.19 ± 0.51	72.86 ± 0.29
speculative decoding	97.81 ± 2.30	118.78 ± 2.45
gain	+34%	+63%

The baseline is flat across workloads. With speculation it spreads by 21%. Code drafts better than prose, because code is more predictable — the draft model guesses right more often, so more tokens come free.

The first post argued that this workload dependence is how misleading comparisons get built. Here it is measured, on one machine, one model, one variable changed. The dependence is entirely a property of speculative decoding, not of the model or the hardware.

And now the correction that post has coming.

A published benchmark reported 108.3 tokens/sec for this exact model and quantisation at the same speculative setting. Our range is 97.8 to 118.8, depending on what you ask it to do. Their number sits almost exactly in the middle of ours.

That's not a contradiction. Their measurement was fine. A single speculative-decoding throughput figure is underspecified by construction — it cannot be compared to anything unless the workload is stated next to it, and almost nobody states it, including us until we measured it.

The first post's sharper claim — three quarters of published numbers failing verification — is true as stated but reads harsher than the evidence supports. The numbers mostly weren't wrong. They were travelling without their conditions attached, which is a different and more forgivable problem, and one we committed too.

4. Gauge two: the knob that wasn't a knob

The obvious next experiment was a kernel autotuner our startup logs reported as disabled. Free performance, presumably, from a default nobody had raised.

It's disabled at every optimisation level, by a global override, with this comment:

# ... the autotuner triggers an illegal memory access ...
# keep autotune off until <upstream issue> is fixed.

Not a conservative default. A known defect, in the exact code path our workload uses, with a bug reference attached.

The interesting part: that bug was filed against a different chip in the same family. Whether it reproduced on ours was genuinely unknown — which made it worth testing, carefully, with the model weights mounted read-only and the tuning cache isolated so a bad result couldn't outlive the experiment.

It didn't crash. It produced a modest ~4% decode gain and — surprisingly — no prefill improvement at all, 0.3% at the depths where our error bars are tight. Backwards from the prediction: autotuners tune matrix-multiply kernels, and prefill is the matrix-multiply-heavy phase. We have no confident explanation and aren't going to invent one.

5. Gauge three: the test that nearly libelled someone's code

Before believing any speedup, we checked correctness: same prompts, temperature zero, hash the output, compare against the known-good baseline.

Three of five hashes diverged.

That is the signature of the scariest possible result — tuned kernels silently computing different answers. A crash is loud and recoverable. Silent corruption is what you actually fear. The agent had the sentence half-written.

Then a control it nearly skipped: run the same server twice. Five of five identical — so the model is deterministic within a configuration, the divergence was real, and it was worth reading what actually differed.

On one prompt, both final answers were the same word. Only the reasoning trace differed.
On another, the baseline never answered at all — it exhausted its token budget mid-thought. The autotuned run answered correctly.

It had hashed the entire output including the chain of thought, on a reasoning model, with a token cap too small for it to finish. The test convicted on differences in thinking that never reached the answer. It could not distinguish "reasoned differently" from "computed wrongly" — which is the only distinction it existed to make.

Third broken gauge of the evening. And this one wouldn't have produced a bad benchmark; it would have produced a public accusation against a vendor's kernels, sourced to a test that couldn't tell the difference.

6. And then the thing that undid the correction

Corrected test: six prompts with objectively verifiable answers, a token budget large enough to finish, comparing final answers only. Six out of six correct. The agent wrote down that output was fine.

Then it read the upstream bug properly.

It isn't a crash bug. The report is failing graduate-level reasoning evaluations with autotuning enabled, and matching expected accuracy with it disabled. The failure mode is subtle accuracy degradation on hard problems.

Its probes were seventeen times twenty-three, the first ten prime numbers, and the capital of France.

A kernel defect that costs you accuracy on graduate-level reasoning passes every single one of those. If the tuned kernels were subtly wrong, that test would have looked exactly the same. So it isn't evidence, and the conclusion was withdrawn.

It had containerised the experiment against a crash, monitored for a crash, and written a crash verdict — because the tooling's own comment described a crash. The documented risk was always the quiet one. It armoured against the wrong hazard, then declared the right one clear on a test with no power to detect it.

7. What we actually learned

Three instruments failed in one evening. Every one of them was correct until the moment it mattered, and not one was caught by its number looking wrong. They were caught by cross-checks: a metric that disagreed with a throughput figure, two workloads that should have differed and didn't, a control run against itself, and — last — reading the primary source instead of the summary of it.

The rule that survives all of this is the same one the first post ended on — the one the agent broke three times in six hours after writing it down:

Before trusting a green result, ask whether a failure would have looked any different.

If the answer is no, you don't have evidence. You have an instrument agreeing with itself.

Numbers here are from a single machine and are not reproduced elsewhere. The runs used 40% of available memory, leaving substantial headroom, and are not ceiling figures. Speculative-decoding results are stated with their workload because that is the entire point.