Neutrino-1 8B: 8B-Parameter Model Fits in 3.88 GB, Runs on Laptops

Fermion Research released Neutrino-1 8B, an 8.19B-parameter decoder-only transformer that ships as a single 3.88 GB file. The model uses a proprietary ternary-family weight format that is eight times smaller than fp16. Every one of its 252 transformer linears is stored bit-packed and decoded inside the matrix kernels at runtime. This approach changes serving economics: single-stream decode is bound by memory bandwidth, so a 3.88 GB working set decodes faster than a 16 GB fp16 model on the same hardware. The model fits beside its KV cache on an 8 GB GPU or a 16 GB laptop.

Weight Format and Efficiency

The container is 3,875,404,812 bytes. Transformer linears account for 67.2% of the file (2,605 MB), token embeddings are 32.1% (1,245 MB), and normalization weights are negligible. Across the 6.95B coded weights, 62.63% are zero, with the remainder split 18.68% positive and 18.69% negative—sign-balanced to a hundredth of a point. The balance varies by depth: early feed-forward layers have 70-72% zeros, while attention projections stay near 62% across all 36 layers.

Benchmarks

Measured on standard public harnesses with thinking disabled (July 2026):

  • MMLU (5-shot): 72.1
  • MMLU-Redux (generative): 67.8
  • IFEval prompt-strict: 77.2
  • IFEval instruction-strict: 80.2
  • BFCL v3 (macro): 68.9
  • GSM8K (flexible extraction): 53.4
  • GSM8K (stated format): 51.7

Speculative Decoding

Neutrino-1 8B pairs with a 0.6B draft model (328 MB) that proposes six tokens per forward pass. The 8B verifies and accepts only tokens matching its own argmax, guaranteeing identical output to plain greedy decoding. On an H100, the drafted rate reaches 763 tok/s for counting prompts (1.93x over 396 tok/s plain), 613 tok/s for factual short answers, and 426 tok/s for code. The pair consumes 4.20 GB of memory (3.88 GB verifier + 328 MB draft) plus shared cache (1 GiB at 4k context). On a 16 GB Apple M5, both models load into one MLX process under a 6 GiB cap, achieving 25.71 tok/s on factual prompts versus 22.00 plain.

Deployment

One container serves all platforms without conversion. Commands:

# Quick chat
pip install fermion-research
fermion chat

# Local OpenAI-compatible server
fermion serve
# Point any OpenAI client to http://127.0.0.1:8000/v1

# GGUF via llama.cpp fork
llama-completion -m neutrino-8b-fv5.gguf -ngl 99

# MLX on Apple Silicon
python -m fermion_mlx --model neutrino-8b_v4.bin --mode chat --tokenizer .

The model is a derivative of Qwen3-8B and is Apache 2.0 licensed. The pip package is Apache 2.0; the llama.cpp fork is MIT. Weights and engine are in one public repository, no waitlist.

Architecture Details

  • 36 decoder layers, hidden width 4096
  • Grouped-query attention (4:1), 32 query heads, 8 key-value heads, head width 128
  • KV cache: 144 KiB per token (fp16), 0.60 GB at 4k context, 4.83 GB at 32k
  • Rotary position encoding (base 1,000,000)
  • RMSNorm (eps 1e-6)
  • Context length: 40,960 tokens
  • Vocabulary: 151,936 tokens
  • Untied input embedding and output head

Why It Matters

Most 8B models require 16 GB VRAM and datacenter GPUs. Neutrino-1 8B runs on 8 GB consumer cards and laptops without quantization loss. The ternary format reduces memory footprint by 8x while preserving full precision output. Speculative decoding with a tiny draft model further improves throughput. This makes running state-of-the-art LLMs locally practical for developers.