Metrale Engine, the inference layer

Open source
Rust and CUDA
Verified on DGX Spark

More inference from the silicon you already own.
One signed binary, verified on every release.

Metrale Engine is the open source inference layer of the Metrale platform. Rust and CUDA in one 75 MB binary, with no Python or PyTorch in the request path, OpenAI and Anthropic compatible APIs, and a signed gate record behind every release. It runs on a single accelerator today and scales across nodes with expert parallelism.

Get running → Talk to us Star on GitHub ↗

Every published rung won, C=1 to C=128, 1.012× to 1.333× against vLLM + MTP. Same box, same checkpoint, same client. Read the campaign log ↗

Published DGX Spark ladder 399 signed records · 2026-10-05 ↗

the matched vLLM configuration at the top rung: 1.333×
aggregate tok/s at the top rung: 478.11
rungs won: 8/8

concurrency ladder DGX Spark · GB10

unsloth/Qwen3.8-27B-NVFP4 vs vLLM + MTP

✓ C=1 1.196×

✓ C=8 1.012×

✓ C=32 1.027×

✓ C=128 1.333×

✓ 8/8 rungs 1.012–1.333×

engine 2276010 2026-09-26

Verified

Every number is a receipt.

Metrale Engine is benchmarked on every release against a committed baseline, and every record is signed. The performance on this page comes from those records, stamped with the engine commit and the date. If a number is not in the repository, it is not on this page.

engine b39a6650 · 2026-10-12

Against matched vLLM at C=128

1.333×

vLLM + MTP runs vLLM with its own speculative decoding at the same K as the engine, on the same box, checkpoint, client and prompts.

Aggregate throughput at C=128

478.11tok/s

unsloth/Qwen3.8-27B-NVFP4 on one DGX Spark. mean tok/s over 3 timed reps (1 warmup discarded).

Rungs won, C=1 to C=128

8/8

Margin 1.012× to 1.333× against the matched configuration at every rung, with the rungs we lost on the way in the campaign log.

Faster than vLLM at every concurrency, C=1 to 128

unsloth/Qwen3.8-27B-NVFP4 on one GB10. mean tok/s over 3 timed reps (1 warmup discarded). The matched baseline runs vLLM's own MTP speculative decoding at K=4, same as Metrale Engine, on the same box, checkpoint, client and prompts. Margin ranges 1.012×–1.333× against that matched configuration at each rung.

Metrale Engine vLLM + MTP vLLM, no speculation config differs

Throughput in tok/s. Ratio is Metrale Engine over the matched vLLM + MTP configuration at that rung. The unmatched no-speculation vLLM leg is shown, not scored.
C	Metrale Engine	vLLM + MTP	vLLM, no speculation	Ratio
1	23.59	19.72	11.04	1.196×
2	41.02	37.11	21.34	1.105×
4	74.21	71.61	41.20	1.036×
8	125.95	124.48	78.18	1.012×
16	203.36	197.03	137.11	1.032×
32	291.01	283.48	219.50	1.027×
64	386.63	361.39	312.26	1.070×
128	478.11	358.57	390.42	1.333×

Exact configuration and provenance

Checkpoint: unsloth/Qwen3.8-27B-NVFP4
Hardware: NVIDIA GB10 Grace Blackwell, 121.7 GB unified — dgx2 (spark-43fa)
Workload: ISL 128 / OSL 1024, 3 timed reps + 1 warmup, temperature 0, seed 42
Parity: disabled on both engines via chat_template_kwargs.enable_thinking=false; presence_penalty and frequency_penalty pinned to 0.0 on both engines

Metrale Engine Metrale Engine 1.0.0-beta-preview

Round 11 throughout, except C=2. C=2 is the 2026-08-18 hardening run (c2_metrale_dgx2_20260818.json), not round 8: the certified table originally carried round 8's 38.95 there, which won by 1.004x -- a margin inside either engine's run-to-run spread. Both legs were re-measured back to back on dgx2 on 2026-08-18 (Metrale Engine 15:05, vLLM+MTP 15:20) and that PAIR replaced the earlier pair, which is why the Metrale Engine number rose to 41.02 while the paired vLLM number fell to 37.11 and the margin moved to 1.105x with non-overlapping distributions. A round-11 C=2 exists at 40.42 (l38_r11_c2.json) but has no same-day vLLM leg, so it is not publishable as a pair. See RESULTS.md, C=2 HARDENED.

Buildbce152b18

Certified at 4012c9b7e1, which differs only in doc comments and gate machinery -- no executable change. Merged to main as 60370b9532.

SpeculationMTP, K=4 (--num-drafts 3); self-disables above 32 concurrent sequences

EnvMETRALE_PREFILL_CODISPATCH=1 METRALE_FP8_ROWWISE=1 METRALE_MTP_DCUT_RATIO=1.0 METRALE_MTP_K_LADDER=1:3,2:1,4:2,8:2,16:1

Command

met serve unsloth/Qwen3.8-27B-NVFP4 --host 0.0.0.0 --port 8888 --model-name unsloth/Qwen3.8-27B-NVFP4 --max-seq-len 2048 --max-batch-size 128 --gpu-memory-utilization 0.85 --kv-cache-dtype fp8 --enable-prefix-caching true --ssm-cache-slots 8 --ssm-checkpoint-interval 32 --speculative --num-drafts 3 --mtp-quantization bf16 --scheduling-policy fifo --tool-call-parser qwen3_coder --disable-tool-grammar true --disable-thinking --request-timeout 0 --ssm-h-dtype f16-pool --gdn-fused-norm --ssm-batched-recurrent --ssm-tail-midchunk false --mtp-gate force --prefill-varlen-batch --no-tui

vLLM + MTP vLLM 0.27.1

The apples-to-apples reference: ctx 2048, batch cap 128, util 0.85, fp8 KV, prefix caching, thinking off and MTP K=4 -- every axis matched to Metrale Engine. vLLM at its best, not a handicapped baseline.

Buildvllm/vllm-openai:latest @ sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967

SpeculationvLLM's own Qwen3_5MTP, K=4 (num_speculative_tokens=3)

EnvHF_HUB_OFFLINE=1

Command

vllm serve --model unsloth/Qwen3.8-27B-NVFP4 --served-model-name unsloth/Qwen3.8-27B-NVFP4 --host 0.0.0.0 --port 8001 --max-model-len 2048 --max-num-seqs 128 --gpu-memory-utilization 0.85 --enable-prefix-caching --dtype bfloat16 --kv-cache-dtype fp8 --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

vLLM, no speculation vLLM 0.27.1

Not matched to Metrale Engine: bf16 KV cache (Metrale Engine and the MTP leg use fp8); ctx 4096 (the matched legs use 2048); no speculative decoding. Shown for completeness. It is not the published denominator.

Unmatched fingerprint: bf16 KV, ctx 4096, no speculation. It beats vLLM+MTP at C=128 (390.42 vs 358.57) because MTP verification costs vLLM more than it gains 128-wide. Plotted for completeness. The published ratio uses the matched MTP leg, not this one.

Buildvllm/vllm-openai:latest @ sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967

Speculationnone

EnvHF_HUB_OFFLINE=1

Command

vllm serve --model unsloth/Qwen3.8-27B-NVFP4 --served-model-name unsloth/Qwen3.8-27B-NVFP4 --host 0.0.0.0 --port 8001 --max-model-len 4096 --max-num-seqs 128 --gpu-memory-utilization 0.85 --enable-prefix-caching --dtype bfloat16 --kv-cache-dtype bfloat16

Per-rung detail

ITL is inter-token latency as AIPerf defines it — (request_latency − TTFT) / (OSL − 1), the numerator ending at the final response chunk. It is the same quantity usually called TPOT; the record keys keep that name (tpot_p50_ms) because renaming a recorded key would orphan every measurement already committed.

Series	C	tok/s	median	spread	TTFT p50	ITL p50	source file
Metrale Engine	1	23.59	23.55	2.78%	682.54 ms	41.84 ms	l38_r11_c1.json
Metrale Engine	2	41.02	41.37	4.15%	795.2 ms	47.53 ms	c2_metrale_dgx2_20260818.json
Metrale Engine	4	74.21	74.29	2.79%	1296.3 ms	51.76 ms	l38_r11_c4.json
Metrale Engine	8	125.95	125.83	2.92%	2331.47 ms	59.01 ms	l38_r11_c8.json
Metrale Engine	16	203.36	203.45	1.23%	4252.58 ms	72.67 ms	l38_r11_c16.json
Metrale Engine	32	291.01	292.50	2.43%	8281.81 ms	98.91 ms	l38_r11_c32.json
Metrale Engine	64	386.63	387.48	1.68%	16052.93 ms	148.81 ms	l38_r11_c64.json
Metrale Engine	128	478.11	480.05	2.05%	31760.85 ms	234.6 ms	l38_r11_c128.json
vLLM + MTP	1	19.72	19.72	1.89%	286.71 ms	50.37 ms	vllm_fp8_mtp_reference.json
vLLM + MTP	2	37.11	37.18	2.94%	390.84 ms	53.22 ms	c2_vllm_mtp_dgx2_20260818.json
vLLM + MTP	4	71.61	72.05	2.82%	604.84 ms	54.51 ms	vllm_fp8_mtp_reference.json
vLLM + MTP	8	124.48	124.31	1.25%	936.14 ms	60.8 ms	vllm_fp8_mtp_reference.json
vLLM + MTP	16	197.03	197.85	3.51%	1620.35 ms	76.25 ms	vllm_fp8_mtp_reference.json
vLLM + MTP	32	283.48	284.20	1.27%	3066.96 ms	107.45 ms	vllm_fp8_mtp_reference.json
vLLM + MTP	64	361.39	361.18	0.21%	5278.74 ms	167.48 ms	vllm_fp8_mtp_reference.json
vLLM + MTP	128	358.57	358.61	0.44%	8500.7 ms	269.54 ms	vllm_fp8_mtp_reference.json
vLLM, no speculation	1	11.04	11.05	0.32%	155.95 ms	90.4 ms	vllm_latest_reference.json
vLLM, no speculation	2	21.34	21.34	0.04%	235.35 ms	93.52 ms	vllm_latest_reference.json
vLLM, no speculation	4	41.20	41.20	0.03%	419.59 ms	96.77 ms	vllm_latest_reference.json
vLLM, no speculation	8	78.18	78.18	0.02%	729.44 ms	101.71 ms	vllm_latest_reference.json
vLLM, no speculation	16	137.11	137.11	0.04%	1176.06 ms	115.56 ms	vllm_latest_reference.json
vLLM, no speculation	32	219.50	219.45	0.09%	2240.16 ms	143.58 ms	vllm_latest_reference.json
vLLM, no speculation	64	312.26	312.31	0.12%	4009.21 ms	200.87 ms	vllm_latest_reference.json
vLLM, no speculation	128	390.42	390.15	0.31%	6710.78 ms	320.84 ms	vllm_latest_reference.json

Harness bench/ladder38/harness_w55_conc_ladder.py. 5 harness revisions appear above: 6412b12d4d — vLLM legs. Does not send the penalty keys.; 1c77e1d8e9 — Metrale Engine legs (rounds 7-11). Adds explicit presence_penalty/frequency_penalty = 0.0.; 1f10d4887b — The 2026-08-18 C=2 hardening pair (c2_metrale_dgx2_20260818.json and c2_vllm_mtp_dgx2_20260818.json). Both raw files carry this one driver sha, which is what makes them a pair: same harness revision, same day, same box.; 664302439b — The ENERGY leg (vllm-mtp-energy), 2026-09-21. harness_w56_conc_ladder.py -- the w55 harness plus power_window.py, which is the half that records gpu_rail_energy_j; w55 has no power sampling at all, which is why the throughput-only legs above carry no joules. This is the sha256 of the copy committed at bench/ladder38/harness_w56_conc_ladder.py and asserted equal to the raw file's driver_sha256 at the time it was added, so the two are checkable against each other -- and gen-ladder.mjs re-hashes that file at every build as harness_repo_sha256.; 55a5963e4b — The ENERGY leg's wide rungs (vllm-mtp-energy C=32/64/128), 2026-09-28: the same harness_w56_conc_ladder.py at a later revision, the copy committed at bench/ladder38/harness_w56_conc_ladder.py since 7cd54d45. Each raw file's driver_sha256 equals it (the harness hashes itself at startup). It differs from 664302439b in comments and in detecting in-band SSE error frames and truncated streams (both turn a would-be silent success into a counted error), not in the request body; power_window.py, which does the integral, differs only in comments. The only difference is those two keys. vLLM already defaults both to 0.0, so its sampling is byte-identical either way; the keys exist to stop Metrale Engine's non_thinking preset injecting presence_penalty=1.5. Both engines therefore ran identical sampling.

Full campaign log, including every rung we lost on the way and the three claims we retracted: bench/ladder38/RESULTS.md. Generated 2026-09-28T22:12:33Z from the committed measurements.

The top of the ladder is the part that matters.

Agentic work arrives as fleets of tool calling agents sharing a context bus, and the engine underneath them is judged where the requests pile up, not at a single stream.

From C=64 to C=128, Metrale Engine adds +23.7% throughput while vLLM + MTP adds -0.8%.

An engine that flattens under load caps how many agents you can run on the hardware you have. Holding the curve is what turns one accelerator into a fleet.

What this does and does not say

One accelerator: NVIDIA GB10 Grace Blackwell, 121.7 GB unified. same box, same checkpoint, same client, back-to-back legs.
One checkpoint: unsloth/Qwen3.8-27B-NVFP4, a dense 27B hybrid, 48 GDN + 16 attention layers. Other models are measured in the dashboard and claimed nowhere else.
A matched baseline: vLLM + MTP is vLLM 0.27.1 with vLLM's own Qwen3_5MTP, K=4 (num_speculative_tokens=3), measured 2026-08-17 to 2026-08-18. It is a dated snapshot, not a live series.
One workload: ISL 128 and OSL 1024, 3 timed reps after 1 warmup, temperature 0, seed 42. Your prompts will differ.
Drawn, not scored: vLLM, no speculation differs on bf16 KV cache (Metrale Engine and the MTP leg use fp8), ctx 4096 (the matched legs use 2048), no speculative decoding. It is on the chart for completeness and is not the denominator of any ratio here.
Nothing beyond that: One box, one checkpoint, one workload. This says nothing about other hardware until we have run it there.

Benchmark dashboard

Every benchmark family, every signed run.

Agentic, BFCL, TTFT, high-ISL TTFT, decode, concurrency and cost, one tab each. Every chart point opens its record, its signature and the steps to reproduce it.

Signed records

399 of 399 gate records carry a detached signature and the public key that made it, committed beside the record. The newest is from 2026-10-05.

How to verify a record ↗

The release gate

A Metrale Engine image ships only after the serve matrix passes: every model boots, stays coherent (greedy determinism, no token leakage, tool reliability), and holds throughput within 10% of its committed baseline.

A release that ships slower than the committed baseline fails our gate.

What verified means ↗ gate_results.py ↗

Reproduce it

The campaign log holds every rung, every raw file and the harness sha, including the rungs we lost on the way and the claims we retracted.

Read the campaign log ↗

Beat these numbers or catch a regression, open an issue and we will feature it. Open an issue ↗

release gate DGX Spark · GB10

✓signed gate records399 of 399

✓benchmarks with a record17 of 42

✓concurrency sweeps passing46 of 46

newest record 2026-10-05

Models

Every model on this page is a recipe you can run today.

Pick a vendor, then a family. Each row is one recipe in the public registry, with its checkpoint, container image and serve settings fixed, so the page cannot list a model we do not ship. Copy the command and run it as is.

35recipes
6vendors
13families

Qwen Qwen3.6

12 recipes

Qwen3.6 27B FP8 Qwen/Qwen3.6-27B-FP8

FP8 single 27B

metralectl run qwen3.6-27b-fp8
Qwen3.6 27B FP8 MTP Qwen/Qwen3.6-27B-FP8

FP8 single 27B

metralectl run qwen3.6-27b-fp8-mtp
Qwen3.6 27B NVFP4 nvidia/Qwen3.6-27B-NVFP4

NVFP4 single 27B

metralectl run qwen3.6-27b-nvfp4
Qwen3.6 27B NVFP4 Prefill Record nvidia/Qwen3.6-27B-NVFP4

NVFP4 single 27B

metralectl run qwen3.6-27b-nvfp4-prefill-record
Qwen3.6 27B NVFP4 Unsloth unsloth/Qwen3.6-27B-NVFP4

NVFP4 single 27B (mixed precision above layer 55), dense hybrid (48 GDN linear-attn + 16 softmax-attn layers)

metralectl run qwen3.6-27b-nvfp4-unsloth
Qwen3.6 35B A3B FP8 Bf16head Qwen/Qwen3.6-35B-A3B-FP8

FP8 single 35B

metralectl run qwen3.6-35b-a3b-fp8-bf16head
Qwen3.6 35B A3B FP8 MTP Qwen/Qwen3.6-35B-A3B-FP8

FP8 single 35B

metralectl run qwen3.6-35b-a3b-fp8-mtp
Qwen3.6 35B A3B FP8 Nvfp4head Qwen/Qwen3.6-35B-A3B-FP8

FP8 single 35B

metralectl run qwen3.6-35b-a3b-fp8-nvfp4head
Qwen3.6 35B A3B FP8 Nvfp4head Experts NVFP4 Qwen/Qwen3.6-35B-A3B-FP8

FP8 single 35B

metralectl run qwen3.6-35b-a3b-fp8-nvfp4head-experts-nvfp4
Qwen3.6 35B A3B FP8 Nvfp4head Experts NVFP4 Gate Up Qwen/Qwen3.6-35B-A3B-FP8

FP8 single 35B

metralectl run qwen3.6-35b-a3b-fp8-nvfp4head-experts-nvfp4-gate-up
Qwen3.6 35B A3B NVFP4 nvidia/Qwen3.6-35B-A3B-NVFP4

NVFP4 single 35B

metralectl run qwen3.6-35b-a3b-nvfp4
Qwen3.6 35B A3B NVFP4 Declared nvidia/Qwen3.6-35B-A3B-NVFP4

NVFP4 single 35B

metralectl run qwen3.6-35b-a3b-nvfp4-declared

What a recipe pins

Checkpoint: The exact model and quantization the engine loads.
Container image: The engine image the recipe runs, so a deployment reproduces on another box.
Serve settings: Context length, batch size, KV precision and speculative decoding, set per model.

EP=2 is expert parallelism across two DGX Spark units. Those recipes take two machines and start from the command rather than the Run button.

Browse the recipe registry ↗

Get running

Install the launcher, check the machine, start the flagship recipe.

Three commands on Linux, macOS or Windows. The installer verifies every download against the release checksums, and metralectl will print any command before it runs it.

bash ~/.local/bin

Install the launcher

curl -fsSL https://metrale.ai/install.sh | sh

Downloads a prebuilt metralectl, checks it against the release SHA256SUMS and installs it to ~/.local/bin. No Python, no Rust toolchain.
Check the machine

metralectl doctor

Reports whether Docker, the NVIDIA container runtime and the local agent are ready, and says what to fix.
Start the flagship recipe

metralectl run qwen3.6-35b-a3b-fp8-mtp

Pulls the engine image, loads Qwen/Qwen3.6-35B-A3B-FP8 and serves an OpenAI compatible API.

Docker, and the NVIDIA container runtime for GPU recipes. metralectl list, show and run --print work without either.

Prefer to inspect first

Nothing here has to be taken on trust.

metralectl run qwen3.6-35b-a3b-fp8-mtp --print

Prints the docker command the recipe implies, and runs nothing.
cargo install metralectl

Builds the launcher from source on crates.io instead of downloading a binary.
uvx metralectl list

Lists the recipes with no install step at all, through uv.

What the installer does

Refuses any download whose checksum is not in the release SHA256SUMS.
Installs one binary to ~/.local/bin, or %LOCALAPPDATA%\Programs\metralectl on Windows.
Sets the local agent up to start at login, as your user and never as root. METRALECTL_NO_AGENT=1 skips it.
Reverses cleanly on Linux and macOS. The same one liner with --uninstall removes the binary and the agent service.

Read the deployment guide ↗ Documentation ↗ Launcher README ↗

Community

Built in the open.

The engine went from one public post to a community running it on their own hardware. Every line of it is in the repository, and so is every gate record behind a number on this page.

Star on GitHub ↗ 5 stars on GitHub

The repository

Kernels, scheduler, gate records and the campaign log, in one public tree. What ships is what you can read.

Open the repository ↗

Discord

We are in there every day, shipping fixes, taking model requests and tuning kernels in the open. Bring what you are running.

Join the Discord ↗

Discussions and issues

Design questions go to Discussions. Bugs and regressions go to Issues, and a reproducible report gets featured rather than buried.

Start a discussion ↗

Good first issues ↗r/LocalLLaMA ↗

Contribute

Your machine is the test fleet.

Metrale Engine grows from the machines it runs on. Every path below is real and linked, and a regression report is worth as much to us as a kernel.

MIT OR Apache-2.0

Dual licensed under MIT OR Apache-2.0, at your option. MIT ↗Apache-2.0 ↗

Verify

Run the serve matrix

Boot the matrix on your own DGX Spark and report what you see. Regressions and wins both get featured.

Deployment guide ↗

Models

Add or tune a recipe

Recipes are the model source of truth. Add a model, tune a quantization, open a pull request against the engine recipes.

The recipe registry ↗

Kernels

Kernels in Rust and CUDA

Hand tuned attention, MoE, GDN and Mamba-2 for Blackwell. Register level work, no generic fallbacks.

Good first issues ↗

Docs

Docs, triage, ideas

Improve the guide, triage issues, or tell us what you are running and where it fell short.

Discussions ↗

Roadmap

What we are building next.

Everything shipped links to an issue, a pull request or the Discord thread where the work happens. Anything not yet committed carries its status, and we do not round it up.

Three node DGX Spark topology

Next up

Three DGX Spark units in one rig for models that will not fit across two. More memory, more experts, more concurrency headroom. We are wiring up the topology now.

Discuss the topology in Discord ↗

Intel Arc Pro B70

In talks

Active conversations with Intel about bringing the engine to the Arc Pro B70. Nothing is signed yet, and this card will say so until it is.

Follow along in Discord ↗

AMD Strix Halo

Runs through SCALE

Native gfx1151 through SCALE. AMD provided a Strix Halo desktop and we brought the engine to it, custom kernels and all.

The gfx1151 kernels ↗

Bigger model support

Tracking

Large MoE NVFP4 ports across expert parallel topologies, DeepSeek and Kimi class, tracked in the open.

Open issues ↗

Our position

AI worth having should run on hardware you own.

An accelerator at the edge, the workstation under a desk, or a rack you operate. Metrale builds one engine for that whole range, in Rust so the entire path from HTTP to kernel dispatch can be read by one engineer, and verifies it on the silicon we can put our hands on. NVIDIA and AMD provide the machines we develop on, and the community running the engine is the test fleet.

Questions

The questions we get asked.

Short answers, each one backed by something on this page or in the repository.

What is Metrale Engine?

An open source LLM inference engine written in pure Rust and CUDA. It serves an OpenAI compatible API from a single binary, with no Python and no PyTorch in the serving path. One codebase covers the range, from edge class accelerators through workstations to expert parallel deployments across nodes.

What hardware does Metrale Engine run on?

NVIDIA DGX Spark (GB10) is verified today. AMD Strix Halo (gfx1151) runs the same CUDA source compiled through SCALE by Spectral Compute, one codebase and no HIP port.

Is Metrale Engine faster than vLLM on a DGX Spark?

On the published concurrency ladder, yes at every rung from C=1 to C=128 against the matched vLLM and MTP configuration. The margin is widest at the top, where Metrale Engine keeps scaling from C=64 to C=128 and the matched configuration flattens. Same box, same checkpoint, same client, same prompts, greedy sampling with matched penalties. The dashboard on this page draws every rung from the signed records, and the campaign log in the repository includes the rungs we lost on the way.

How do I install it?

One command. curl -fsSL https://metrale.ai/install.sh | sh on Linux and macOS, or irm https://metrale.ai/install.ps1 | iex in PowerShell on Windows. It downloads a prebuilt metralectl, verifies it against the release checksums and installs it under your own user. If you would rather not pipe a script into a shell, cargo install metralectl builds it from crates.io.

Which models can I run?

Every model on this page maps to a recipe in the recipe registry, which is the single source of truth, so the site cannot list a model that has no recipe. Qwen has the most recipes, and Gemma, Nemotron, Mistral, MiniMax and DeepSeek have recipes too.

What does verified mean here?

An image ships only after the serve matrix passes. Every model boots, stays coherent under greedy determinism with no token leakage and reliable tool calls, and holds throughput within ten percent of its committed baseline. A release that ships slower than its baseline fails the gate.

Why does concurrency matter more than single stream speed?

Because agentic systems do not send one request at a time. A fleet of tool calling agents sharing a context bus arrives as many concurrent streams, so the engine is judged where the requests pile up. On the published ladder Metrale Engine keeps gaining throughput from C=64 to C=128 while the matched vLLM configuration does not, and an engine that flattens under load caps how many agents a given box can run.

What licence is Metrale Engine under, and can I use it commercially?

Dual licensed under MIT OR Apache-2.0, at your option. Commercial use is allowed under either licence. If you run it in production and want support or a platform licence, email us.

Is there a paid tier of the engine?

No. There is one engine and one licence, and the binary you install from this page is the binary we verify and benchmark. Metrale Control and Metrale Economics are the platform around it for fleets, and they are licensed per GPU.

Does Metrale Engine run multi node?

Yes. EP=2 expert parallelism across two DGX Spark units ships as recipes, marked EP=2 in the model list. A three node DGX Spark topology is being wired up now.

Who do I talk to about running it in production?

Email [email protected], or use the reach out section below. Tell us the hardware and the workload and we will scope it with you.

Something not answered here? Talk to us → or find us in Discord ↗

Reach out

Put the engine on your workload.

We run the ladder on your models, on your hardware, and hand you the record.

Book a demo → [email protected] Find us in Discord ↗

Production

Evaluating the engine for production or already running it. Tell us what you serve and we will scope the deployment with you.

Partnerships

Frameworks, benchmarks and standards bodies. If it advances inference on hardware people own, we want the conversation.

Silicon

Hardware you want Metrale Engine running on. Tell us about it and we will scope the bring up.

More inference from the silicon you already own.One signed binary, verified on every release.

Every number is a receipt.

Faster than vLLM at every concurrency, C=1 to 128

Metrale Engine Metrale Engine 1.0.0-beta-preview

vLLM + MTP vLLM 0.27.1

vLLM, no speculation vLLM 0.27.1

Per-rung detail

The top of the ladder is the part that matters.

Every benchmark family, every signed run.

Signed records

The release gate

Reproduce it

Every model on this page is a recipe you can run today.

Qwen Qwen3.6

Install the launcher, check the machine, start the flagship recipe.

Install the launcher

Check the machine

Start the flagship recipe

Built in the open.

The repository

Discord

Discussions and issues

Your machine is the test fleet.

Run the serve matrix

Add or tune a recipe

Kernels in Rust and CUDA

Docs, triage, ideas

What we are building next.

Three node DGX Spark topology

Intel Arc Pro B70

AMD Strix Halo

Bigger model support

AI worth having should run on hardware you own.

The questions we get asked.

Put the engine on your workload.

Production

Partnerships

Silicon

More inference from the silicon you already own.
One signed binary, verified on every release.