The Lab

What's actually running on my local-AI fleet — the hardware, the models in service, and what's on the bench right now. A weekly-ish snapshot, not a dashboard.

Snapshot: July 31, 2026 7 machines 91 GB discrete VRAM 160 GB unified memory 8 models in service

On the bench this week

Spec-driven code-gen trial — the same five specs run against 14B, 32B and 35B tiers, repair rounds as the measurement; three checker arms, execution-graded (no LLM judges).
Speculative decoding (multi-token prediction) — measured 1.38× on a 35B MoE; now testing whether it changes run-to-run output agreement, 417-document corpus.
Dense vs MoE at the 27–35B class — swapping a dense 27B 6-bit against the deployed 35B MoE on the same Mac to isolate the architecture term.

Mac Studio — M1 Max

32 GB unifiedMLX

Serving Qwen3.6-35B-A3B (MoE, 4-bit) — ~20 GB resident, 262K context

Bench Dense-27B 6-bit trial; tier sweep vs 30B / 9B

MoE: 35B total / ~3B active per token

DGX Spark — GB10

128 GB unifiedvLLM

Serving Qwen3.6-35B-A3B (NVFP4) with MTP speculative decoding, k=3

Bench MTP output-agreement study (4 arms, 417 docs)

104.5 tok/s batched — 1.38× over plain decode

ThinkStation P620

2× RTX 5060 Ti · 32 GBllama.cpp

Serving Qwen3-32B (Q4_K_M) — 18.4 GiB split across both cards

Role The capacity tier — only box that holds a dense 32B

20.6 tok/s generation (32B, bench)

Inference server

RTX 5060 Ti · 16 GBllama.cpp

Serving Qwen3-14B (Q4_K_M) — ~9.7 GB, thinking-mode pinned off

On disk Qwen3-8B as instant fallback

45.5 tok/s live server (14B)

Windows workstation

RTX 4070 Ti Super · 16 GBOllama

Serving Qwen2.5-Coder-14B (Q5_K_M) — the coding slot

Bench Routed arm of the code-gen trial (drives 4 other boxes)

Ada, FP8-capable · solo card by design

Daily driver

RTX 5060 Ti · 16 GBllama.cpp · CUDA 13.3

Bench Qwen3-14B (Q4_K_M) — code-gen trial arm, bounded-thinking config (--reasoning-budget)

Role Gaming rig that moonlights as a test arm

131 tok/s on the 4B fleet benchmark

Mini node

GTX 1080 Ti · 11 GBOllama

Serving Qwen3-8B + Qwen2.5-3B; nomic-embed-text (all fleet embeddings)

Role The smallest tier — embeddings + tiny-model baseline

Pascal, still earning its slot

This is a lab, not a service — no uptime promises, and models rotate as experiments demand. Numbers are my own measurements on my own hardware; generation speeds are labelled by condition (live server vs bench vs batched) because those aren't the same number. Refreshed roughly weekly. Curious about the methodology? It's the same discipline as the blog: verify before claiming.