Pi's Minimalism Beats Bloat in Databricks Benchmark

Databricks ran a benchmark comparing coding agents on their multi-million-line codebase. The results surprised many: Pi, a minimalist harness with only 4 tools and a system prompt under 1,000 tokens, achieved the highest pass rate when paired with Opus 4.8, at a significantly lower cost than Claude Code and Codex. This is a direct challenge to the industry trend of building ever-larger, more complex harnesses.

The Databricks study, titled "Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase," was designed to avoid bias from oversaturated external benchmarks. They created tasks based on what their engineers actually do daily. The key finding: "the harness a model is called from dramatically impacts cost and quality." In many cases, "simple harnesses like Pi performed best on our workloads."

Context Discipline: 3x Less Context, 2x Lower Cost

The study separates model from harness, revealing that the same model with the same thinking effort through different harnesses showed "cost per task differed significantly (more than 2x in some cases), while quality remained the same." Pi's advantage comes from what Earendil calls "context discipline." Pi sends about 3x less context per turn, manages context better, and finishes tasks in fewer runs. This is a concrete, measurable benefit.

This validates Earendil's own observations at the model level: running complex workflows on Haiku 4.5 was often more expensive than Sonnet 4.6, because the agent required more turns to complete the task. Now we see the same effect at the harness level. Stronger, more expensive models with a performant harness can be cheaper than the converse.

Shopify Builds AutoResearch on Pi's Extensibility

Pi's minimalism doesn't mean inflexibility. It is designed for extensibility and self-editability. Shopify's David Cortés built pi-autoresearch as a Pi extension simply by asking Pi to "create an extension for Autoresearch...". Pi reads its own extension documentation and builds a new workflow.

Autoresearch is an autonomous loop for optimization with coding agents. When you ask for a change, it runs experiments to find what works and what causes regressions. It can discard regressions and keep self-improving as long as the target is measurable.

The results at Shopify were dramatic: unit tests running "300 times faster," React component mounting "20% faster," reduced build times across multiple projects, and improvements to pnpm performance. Pi doesn't ship these tools out of the box. Instead, it makes it simple for you to build them. Instead of assuming the vendor knows your workflow, Pi assumes you know best.

Why Minimal Wins Now

A year ago, native harnesses had a structural advantage because models were built around them. That argument has weakened. Frontier models are now competent at understanding a terminal-style coding environment. Anthropic recently cut Claude Code's system prompt by 80%, a clear sign of this shift. The question is no longer about how native the harness is, but how it handles context to avoid redundancy and act with clean primitives.

Pi provides this: less prompt overhead, cheaper runs, fewer unnecessary abstractions. Because it is extensible, you don't lose power; you gain selectivity. You add complexity only when it "earns its keep."

Local models are also developing fast, and Pi's context discipline is especially valuable there. Local models usually have lower context windows, and prefill can take a long time. Pi preserves a stable prompt prefix, avoiding minute-long re-prefilling. This makes Pi an ideal harness for local models.

What This Means for Developers

If you're evaluating coding agents, don't assume that more features mean better results. Pi's minimalism is a deliberate design choice that pays off in cost and performance. The Databricks benchmark provides concrete evidence: simple harnesses can outperform complex ones. The Shopify case shows that extensibility allows you to build exactly what you need, without bloat.

Consider testing Pi on your own workloads. Start with the default setup and see if it meets your needs. If you need more, build an extension. The tools are there. The performance data is compelling.

This article is based on a post from Earendil, the company behind Pi.