Speculation, width-laddered
MTP, K=4 (--num-drafts 3); self-disables above 32 concurrent sequences. The draft width is chosen per concurrency (1:3,2:1,4:2,8:2,16:1) rather than fixed, so verification cost never exceeds the win.
Slide 1 of 0
Verification steps
The concurrency claim on the front page, restated as commands you can run on your own box, with every axis it depends on named so it can be falsified.
Metrale Engine 1.0.0-beta-preview sustains higher mean decode throughput than vLLM + MTP (vLLM 0.27.1) on unsloth/Qwen3.8-27B-NVFP4, served on one NVIDIA GB10 Grace Blackwell, 121.7 GB unified, at every concurrency C = 1, 2, 4, 8, 16, 32, 64, 128 — ISL 128, OSL 1024, temperature 0, seed 42, mean tok/s over 3 timed reps (1 warmup discarded). Margins run 1.012× to 1.333×.
Scope
Every noun in the claim is a knob someone could have turned. The fastest way to waste a week is to test something we never said.
Result
Aggregate decode tokens per second, both engines, every rung. Log2 X because the rungs double; the table carries the exact figures and the spread the chart only implies.
Reference
Six lines decide whether anything after them is comparable; ten pins are what make the comparison fair. If your box differs on any of them you are measuring something else — fine, but say so.
Fingerprint
Parity — driven by one script, not two
vLLM at its best, not its defaults: vLLM 0.27.1 registers this checkpoint's MTP weights, so the baseline runs vLLM's own Qwen3_5MTP, K=4 (num_speculative_tokens=3). The no-speculation leg is published beside it and is not what we claim against.
Steps 1 and 2
Every check here has been the reason a run was thrown away in this campaign. The container serves models; the binary measures them. You need both, from the tree the record names.
nvidia-smi # GB10, driver 580+free -g # ~121 GB unified, not nvidia-smiexport PATH=/usr/local/cuda/bin:$PATHnvcc --version # must report CUDA 13.0docker run --rm --gpus all \ nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smidf -h ~/.cache/huggingface # weights land here, tens of GB
nvidia-smi reports memory as Not Supported on GB10: the 121 GB is a unified pool, so free is the instrument. CUDA ships outside PATH; without the export the cargo build dies in a build script rather than anywhere informative. The docker line proves the NVIDIA Container Toolkit is wired up, not just installed. Measure on an idle box: the gate refuses to self-start below 85% free host memory and tolerates at most one foreign compute process.
git clone https://github.com/Metrale/metrale-inference.git metrale-enginecd metrale-enginedocker build -f docker/gb10/Dockerfile -t metrale-gb10 .sudo apt-get install -y build-essential pkg-config \ cmake clang libclang-devcargo build --release -p metrale-server --bin met./target/release/met --version./target/release/met benchmark list concurrency-sweep./target/release/met sync-recipes
Both builds run from the repository root with CUDA on PATH from Step 1. The first cargo build takes 15–30 minutes and leaves 3–5 GB under target/. Every record under .benchmarks/ names the commit it measured as git_sha, with a .sig beside it; check that commit out to rebuild the exact tree. The list line prints every parameter of the sweep with its default, the schema the next steps override; if it prints, the toolchain is sound. sync-recipes populates the recipe index the gate's self-start reads in Step 6.
Steps 3 and 4
Same box, same checkpoint, same client, back to back. The baseline is pinned by digest, not by tag; the subject is the whole certified configuration, rendered from the record the harness wrote.
docker run --rm --gpus all --network host \ vllm/vllm-openai@sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967 \ --model unsloth/Qwen3.8-27B-NVFP4 \ --max-model-len 2048 --max-num-seqs 128 \ --gpu-memory-utilization 0.85 \ --kv-cache-dtype fp8 --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
num_speculative_tokens 3 is K=4, the same draft width the subject runs. Context 2048 and batch cap 128 are the pinned pair; changing either invalidates the comparison in both directions. The cap is load-bearing: vLLM sizes its KV blocks and scheduler budget from max_num_seqs, and a cap-32 pair inverted the ordering on the same box the same day (RESULTS.md, method note).
METRALE_PREFILL_CODISPATCH=1 METRALE_FP8_ROWWISE=1 \METRALE_MTP_DCUT_RATIO=1.0 \METRALE_MTP_K_LADDER=1:3,2:1,4:2,8:2,16:1 \met serve unsloth/Qwen3.8-27B-NVFP4 --host 0.0.0.0 --port 8888 \ --model-name unsloth/Qwen3.8-27B-NVFP4 --max-seq-len 2048 \ --max-batch-size 128 --gpu-memory-utilization 0.85 \ --kv-cache-dtype fp8 --enable-prefix-caching true \ --ssm-cache-slots 8 --ssm-checkpoint-interval 32 --speculative \ --num-drafts 3 --mtp-quantization bf16 --scheduling-policy fifo \ --tool-call-parser qwen3_coder --disable-tool-grammar true \ --disable-thinking --request-timeout 0 --ssm-h-dtype f16-pool \ --gdn-fused-norm --ssm-batched-recurrent --ssm-tail-midchunk false \ --mtp-gate force --prefill-varlen-batch --no-tui
Do not trim this. 6 of these flags (--ssm-h-dtype, --gdn-fused-norm, --ssm-batched-recurrent, --ssm-tail-midchunk, --mtp-gate, --prefill-varlen-batch) are kernel and scheduling knobs whose defaults are the opposite of the certified values; serving without them measures a different engine. An abridged command run in this campaign landed under the published ladder, the gap widening with concurrency exactly as those knobs predict. With the full command the same box reproduced every rung.
Step 5
Two instruments, not interchangeable. The campaign driver produced every number on the result slide; the gate's subcommand measures the gate's workload and is the only one that can mint a record.
python3 -m venv .venv && .venv/bin/pip install aiohttp.venv/bin/python bench/ladder38/harness_w55_conc_ladder.py \ --url http://127.0.0.1:8888 --model unsloth/Qwen3.8-27B-NVFP4 \ --label metrale --out metrale_ladder.json \ --concs 1,2,4,8,16,32,64,128 \ --reps 3 --isl 128 --osl 1024 --warmup 1
Every knob is a required argument; the driver defaults nothing silently, so the command is the methodology. Swap --url and --label for the vLLM leg and run them back to back. Budget about an hour per leg: C=128 alone streams 131072 tokens per rep. The driver prints its sha256 on the first line and writes it as driver_sha256; the published engine legs carry 1c77e1d8e9 and the copy in the repository today hashes 4649c9b001. Record the hash you ran.
met benchmark run concurrency-sweep \ --url http://127.0.0.1:8000 --model unsloth/Qwen3.8-27B-NVFP4 \ --param concurrencies=1,4,8,16 --param isls=512 --param osl=320 \ --skip-coherence-probe --format json > vllm.jsonmet benchmark run concurrency-sweep \ --url http://127.0.0.1:8888 --model unsloth/Qwen3.8-27B-NVFP4 \ --param concurrencies=1,4,8,16 --param isls=512 --param osl=320 \ --format json > metrale.json
Drives an endpoint that is already serving; it neither loads a model nor touches the GPU, so the binary that gates our pull requests is the binary that measures vLLM. stdout carries the record and stderr the progress. The prompt corpus is byte-identical to the driver's, so aggregate tok/s is comparable across the two; percentile rules are not, so never compare one instrument's TTFT against the other's.
Step 6
The same subcommand in its other mode. It serves the benchmark's own recipe on a free port, waits for a cold NVFP4 load, measures, tears down, and writes a signed record; --url and --pull-request-gate are mutually exclusive by design, so an experiment can never become evidence by accident.
./target/release/met benchmark run concurrency-sweep \ --pull-request-gate --yes./target/release/met benchmark certify --dry-run # the plan./target/release/met benchmark certify --yesmet benchmark --pull-request-gate-check # what CI then runs
certify runs every gate the commit still owes and says whether the tree is certified when they are done; it stops on a failed verdict unless given --keep-going. Each run writes .benchmarks/<id>/<date>-<sha>.json carrying the metrics, the verdict, the hardware fingerprint, the exact command and the commit sha, with a .sig beside it. A record is voided by content: a change under crates/, kernels/, Cargo.toml, Cargo.lock, vendor/, 3rdparty_patches/, rust-toolchain.toml or jinja-templates/ invalidates it, a dirty tree fails, and an entry declaring no thresholds fails rather than passes.
Floors CI holds for unsloth/Qwen3.8-27B-NVFP4
| rung | min tok/s | max J/token, GPU rail |
|---|---|---|
| C=1 | 22 | 1.70 |
| C=2 | 38 | 1.00 |
| C=4 | 67 | 0.62 |
| C=8 | 110 | 0.44 |
| C=16 | 180 | 0.30 |
| C=32 | 260 | 0.23 |
| C=64 | 360 | 0.19 |
| C=128 | 440 | 0.16 |
| peak | 440 | — |
| completion tokens | ≥ 650 | — |
| vacuous cells | ≤ 0 | — |
Absolute per-metric bounds committed beside the kernels, not a percentage band, re-cut only from fresh reps on the same instrument. The same certify command also runs the gates that judge something other than speed — bfcl-subset, agentic-webserver, ttft-warm-gate, ttft-cold-gate, serve-matrix — whose records are charted on the benchmarks page.
Mechanism
Ask this before the numbers. A speedup with no stated mechanism and no known ceiling is a configuration artefact waiting to be found.
MTP, K=4 (--num-drafts 3); self-disables above 32 concurrent sequences. The draft width is chosen per concurrency (1:3,2:1,4:2,8:2,16:1) rather than fixed, so verification cost never exceeds the win.
Prefill overlaps decode rather than stalling it, and row-wise fp8 moves fewer bytes on the bandwidth-bound path. Decode at this scale is a memory-traffic problem, so this is where a real win has to come from.
The baseline's C=128 (358.57 tok/s) falls below its own C=64 when speculation stays on at high concurrency; the subject has already switched it off. The margin is a scheduling decision, which is why it reproduces.
Batch cap 128 on both engines at every rung, independent of the concurrency driven. A matched cap-32 pair inverted the ordering on the same box the same day; re-measured at cap 128 the ordering held with non-overlapping distributions. If you re-cap to survive a hazard, say so beside the number and re-pin both engines.
Self-audit
Heiser's benchmarking-crimes taxonomy, run against our own campaign. Three rows stay open, because a checklist with nothing open is marketing.
Evidence
A reproduction that needs a conversation with us is not one. Everything below is already in the repository, and the campaign log records what failed as well as what held.
Provenance
The first question counsel asks about a serving engine is the licence; the second is what a machine refuses. Both, with the mechanism that keeps them honest.
Every number in this deck is read from the same generated records the front page renders. If the ladder is re-run and a rung is lost, these slides say so on the next build.