The Lab

What's actually running on my local-AI fleet — the hardware, the models in service, and what's on the bench right now. A weekly-ish snapshot, not a dashboard.

Snapshot: August 18, 2026 10 machines 152 GB discrete VRAM 288 GB unified memory 11 models in service

On the bench this week

Three new 30B-class models, nine bench cells — matched quant and method across three boxes, every cell landing at 89–93% of its memory-bandwidth ceiling; add each model's speculative-decoding drafter (where one exists) and the ranking between the two dense models flips — the full story is the latest post.
A 120B judgment tier on hardware we already own — a 120B MoE proven in service beside a 30B on the same two-GPU box, against thresholds written down before any number existed. The same arc measured RAM-channel bandwidth and reversed the upgrade plan: 96 GB of RDIMMs ordered instead of a third GPU.
Agent-run fleet ops — three agents on the local message bus audited the serving fleet and ran the 120B bench arc end to end, correcting each other along the way; every error they caught was a frame mismatch, never a bad measurement.
GPU shuffle week — five cards new or moved across four boxes, and two used Z6 workstations joined the rack for bring-up.
The two-3060 server got a coder brain — a 30B MoE coder replaced its dense 14B. Two upstream hazards named for this exact dual-GPU topology were probed before promotion (both failed to reproduce), and two config traps were caught from the model card before the restart, not after.

Mac Studio — M1 Max

32 GB unifiedMLX

Serving Nemotron-3.5-Lightning-30B-A3B (MoE, 4-bit) — the current arm, provisional

Bench Stack-vs-stack trial: same weights, different runtime, different correctness — this box's runtime lost one of those, and its role is under review

60–65 tok/s (MoE, 3.5B active) · no drafter path on this stack

DGX Spark cluster — 2× GB10

2× 128 GB unifiedvLLM · tensor-parallel

Serving DeepSeek-V4-Flash (284B MoE, NVFP4) — now tracking the vendor's active release lane instead of our pinned fork

Bench Lane move re-verified by output, not status: the instruction-following class that failed on the old arm now passes

KV pool 2.41M tokens (was 1.50M) · 2.3× concurrency at 1M context

ThinkStation P620

2× RTX 5060 Ti · 32 GBllama.cpp

Serving Muse-Glimmer-30B (dense, Q4) with a working speculative drafter — plus gpt-oss-120B (MXFP4) as the judgment tier, experts on CPU so both fit at once

Role The capacity tier — a 120B and a 30B co-resident on one box

120B: 26 tok/s co-resident · 35.6 tok/s tuned split

HP Z6 G4 — 5060 Ti pair

2× RTX 5060 Ti · 32 GBllama.cpp · CUDA 13.3

Serving Qwen3.8-27B (dense, Q4) at 65K context

Role Used Xeon workstation, just joined — also the RAM-channel experiment host (6 channels, 2 populated — DIMMs on the way)

Bench Its cells in the nine-cell 30B matrix matched a different CPU/RAM/PCIe/CUDA box with the same GPUs within 1% — generation is bandwidth-bound and nothing else moved it

Nemotron 30B MoE: 124 tok/s bench

HP Z6 G4 — 3060 pair

2× RTX 3060 · 24 GBllama.cpp · CUDA 13.3

Serving Nemotron-3.5-Lightning-30B (MoE, Q4) at 65K context

Role Used Xeon workstation, just joined — bench box turned serving node

Bench The budget half of the nine-cell 30B matrix — same models, same method, cheaper silicon

Nemotron 30B MoE: 94 tok/s bench

Inference server

2× RTX 3060 · 24 GBllama.cpp

Serving Qwen3-Coder-30B-A3B (MoE, Q4) at 40K context, layer-split across both cards — promoted from Qwen3-14B this week

Role The pair is what makes the coder tier possible: the 30B's weights alone are 17 GB and can't fit one 12 GB card — measured 21.7 GB on-GPU across both

91 tok/s bench (3.3B-active MoE) · bigger model, better at code, faster decode

Windows workstation

RTX 4070 Ti Super · 16 GBOllama

Serving Qwen2.5-Coder-14B (Q5_K_M) — the coding slot

Bench Routed arm of the code-gen trial (drives 4 other boxes)

Ada, FP8-capable · solo card by design

Daily driver

RTX 5060 Ti · 16 GBllama.cpp · CUDA 13.3

Serving Qwen3-4B (Q4_K_M) resident — answers the fleet's shared asks on the local bus

Role Gaming rig that moonlights as a test arm

131 tok/s on the 4B fleet benchmark

Mini node

RTX 3070 · 8 GBOllama

Serving nomic-embed-text (all fleet embeddings) + Qwen3-8B

Role The embeddings node — an Ampere card replaced the old Pascal 1080 Ti; less VRAM, more card, and the pair that fits was chosen by measurement

Card was carried as "retired" in our own records — it wasn't

This is a lab, not a service — no uptime promises, and models rotate as experiments demand. Numbers are my own measurements on my own hardware; generation speeds are labelled by condition (live server vs bench vs batched) because those aren't the same number. Refreshed roughly weekly. Curious about the methodology? It's the same discipline as the blog: verify before claiming.