Most LocalAI backends wrap existing engines like llama.cpp or vLLM. That's the right default. But 18 backends are custom C or C++ ports written from scratch. Why? Because wrapping the upstream engine would mean shipping something they couldn't ship: a multi-gigabyte Python install, a non-portable CUDA-only stack, or a model with no C++ implementation at all.
One file, predictable memory
Deploying a Python inference stack means resolving a dependency tree at install time, on the target machine, against whatever CUDA and glibc it has. Deploying a ggml port means copying a shared library and a GGUF file.
The clearest measurement: vllm.cpp, their C++20 port of vLLM's V1 serving architecture. Installing vLLM produces a 9.1 GiB virtualenv. Installing vllm.cpp produces a 66 MiB binary. The engine implements the same things as the Python original: paged KV cache, continuous batching, prefix caching, the scheduler, the sampler. No Python, no PyTorch, no ggml at inference.
Throughput parity with vLLM
Does the smaller footprint cost throughput? On an NVIDIA GB10 running Qwen3.6-27B in NVFP4, greedy, closed loop, against vLLM in its production graphed configuration (not --enforce-eager):
| Concurrency | vllm.cpp tok/s | vLLM tok/s | Ratio |
|---|---|---|---|
| 1 | 86.05 | 82.32 | 1.045x |
| 2 | 159.68 | 158.03 | 1.011x |
| 4 | 292.34 | 290.31 | 1.007x |
| 8 | 508.77 | 505.46 | 1.007x |
| 16 | 801.76 | 789.16 | 1.016x |
| 32 | 1095.01 | 1076.25 | 1.017x |
They're ahead at all six points, but five of those six are ties. Run-to-run noise is 0.5%, and concurrency 2 through 32 land between 0.7% and 1.7%. The honest reading: only the single-stream case (4.5%) is clearly outside noise. Output is token-for-token identical to vLLM at every point. Peak host memory: 24.88 GiB vs 28.18 GiB.
A tie against a mature CUDA stack is a good result for a 66 MiB binary. The footprint saving isn't paid for in throughput.
Against llama.cpp on CPU from the same GGUF file, prefill runs 1.18x faster (223.8 vs 177.3 tok/s), decode is a tie inside llama.cpp's own spread, and tokens are byte-identical to greedy decode. Against MLX-LM on Apple M4, prefill time-to-first-token is 1.5% ahead, but warm total throughput is 97.6% of MLX-LM — a real 2.4% gap entirely in decode.
Sometimes the port is simply faster
depth-anything.cpp is a port of ByteDance's Depth Anything 3. It gives metric depth in metres from one photo, plus per-pixel confidence, camera intrinsics and extrinsics, and a back-projected point cloud. On CPU it is faster than PyTorch running the same model.
| Engine | Quant | Model MB | Load ms | Infer ms | Peak RAM MB | vs PyTorch |
|---|---|---|---|---|---|---|
| PyTorch | f32 | 251 | 674 | 494 | 16.9 | 1.00x |
| C++/ggml | q8_0 | 142 | 40 | 319 | 4.4 | 1.31x |
Same model, 1.31x the speed, 27% of the memory, and a load that finishes in 40 ms instead of 749 ms, on a Ryzen 9 9950X3D at 504x336 with 16 threads. The quantized q4_k build is a 99 MB file and stays near-lossless. Output correlates 1.0 with the reference forward pass, component by component, across 37 parity tests.
The reason it's faster has nothing to do with writing better matmul kernels than PyTorch. Two positional embeddings — the DPT head's UV embedding and the backbone's bicubic position embedding — were being recomputed on every forward pass with single-threaded scalar sin, cos, and bicubic loops, even though they depend only on input geometry and are identical every call. Caching them removed about 95 ms of host-side overhead per forward, which is most of the gap. PyTorch builds the same embeddings with vectorized operations and never paid that cost.
The general shape of these wins: heavy GEMMs are close to a wash, because everyone calls into the same class of BLAS kernels. The difference sits in host-side work that a Python reference implementation never bothered to optimize, and in not loading an interpreter and a framework to do inference.
On GPU the picture flips back to parity: with the ggml CUDA backend and flash attention on a GB10, depth-anything.cpp ties PyTorch's tuned cuDNN at 47.3 ms per forward, and wins only the cold start, loading 1.75x to 2.9x faster.
Parity is the gate, speed is the follow-up
face-detect.cpp and voice-detect.cpp replaced LocalAI's Python insightface and speaker-recognition backends. Both are cases where they do NOT claim a CPU speed win, and both shipped anyway.
face-detect.cpp runs the whole insightface buffalo chain: SCRFD detection, five-landmark similarity-transform alignment to 112x112, and the ArcFace embedding, out of one self-contained GGUF with no Python and no onnxruntime. Detector boxes and landmarks match insightface to within 1 pixel, and the recognition embedding matches to cosine 1.000000, held at any thread count. On CPU it is slower than onnxruntime: SCRFD detect runs at about 0.83x at one thread and 0.69x at eight, ArcFace embed at about 0.61x and 0.84x. onnxruntime's MLAS convolution kernels sit at the FMA-port peak, and a custom AVX2 Winograd path narrowed the gap without closing it. On GPU, routing the same convolutions through cuDNN takes SCRFD from 14.8 ms to 6.4 ms and lands at torch-cuDNN parity.
voice-detect.cpp is the same story with a memory result attached. A WeSpeaker verification peaks at about 62 MB in their binary against about 334 MB for the CPU-only Python, torch, and onnxruntime path — roughly 5.4x lower — with an identical verdict and embedding cosine 1.000000. End to end on CPU the two land within 10-15% of each other, trading the lead by model and thread count. On GPU the conv encoders match the reference.
For a biometric pipeline, matching the reference exactly matters more than being faster than it. An embedding that differs in the fourth decimal place changes verification decisions at a threshold, and every enrolled template in a deployment would have to be recomputed. Parity is what makes the replacement a drop-in rather than a migration.
The method
Every port follows the same sequence, and the order is important.
- Convert the weights first into one GGUF with the tokenizer, vocabulary, and any auxiliary model embedded. Deploying the model is copying a file.
- Port the graph second, and gate it component by component against reference tensors dumped from the original implementation. depth-anything.cpp has 37 ctest cases covering preprocessing, backbone, attention, the DPT head, depth, pose, the ray head, the ray-to-pose solver, and exporters. parakeet.cpp gates on transcript agreement with NeMo at WER 0. face-detect.cpp gates on box and landmark distance in pixels and embedding cosine. A port that is fast and slightly wrong is worthless; without a per-component gate you find out it's wrong months later.
- Optimize third, with a profiler, and only after parity holds. In parakeet.cpp the decisive win was caching a prediction-network LSTM forward pass that was 97% of transducer decode time and mostly redundant. In depth-anything.cpp it was two cached positional embeddings. Neither was a kernel rewrite, and neither would have been findable without a working baseline to profile.
- Expose a flat C ABI last. LocalAI dlopens the shared library through purego and calls that ABI directly. No subprocess, no gRPC hop to a Python server, no interpreter in the serving path.
What it costs
Maintenance, mostly. Each engine is a repository with its own CI, its own benchmark suite, its own GGUF conversion script, and its own parity baselines. Upstream keeps releasing new checkpoints that need converter work.
GPU kernels are the weak spot. ggml's generic CUDA convolution and attention kernels trail NVIDIA's tuned cuDNN on conv-heavy models. That's why face-detect.cpp needs an explicit cuDNN path to reach parity, and why parakeet.cpp's GPU margin over NeMo is a median 1.25x while its CPU margin is wider.
Porting also doesn't scale to everything. llama.cpp, vLLM, whisper.cpp, MLX, and diffusers stay wrapped, because those projects are large, fast-moving, and already excellent at what they do. LocalAI writes an engine when a model has no C++ implementation, when the Python dependency is heavier than the model, or when the thing they need doesn't exist yet. Everything else they install from somebody else.
Each engine keeps its own benchmark suite, its parity gates, and its methodology in its own repository, including runs that didn't work. The full list is the "Backends built by us" table in the LocalAI README.
