Why My Benchmark Stopped at N=22: A Debugging Story in Nine

Why My Benchmark Stopped at N=22: A Debugging Story in Nine Bugs

A developer uncovers nine bugs in a cross-language A2A benchmark, including CPython's 4,300-digit string conversion limit, an LLM that refused to repeat work, and a parser that couldn't handle nanoseconds. The fix revealed that the benchmark was comparing pipelines, not languages.

3 min readJul 16, 2026

Why My Benchmark Stopped at N=22: A Debugging Story in Nine Bugs

The 4,300-Digit Wall

The benchmark compared A2A agent performance across Python, Go, Node, and Rust. At N=24, Python returned N/A. The error: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit. CPython 3.11 added this default cap as a denial-of-service mitigation. The 24th Mersenne prime (2^19937−1) has 6,002 digits. The fix? Delete the unnecessary str() call. Go had the same dead weight (val.String()) but didn't crash.

The Parser Bet on LLM Phrasing

The harness parsed Python's elapsed time with regex r"It took ([\d\.\-e]+) seconds". But Gemini never said "It took"—it said "Calculating the first 5 Mersenne primes took…" or "The calculation took…". Python data survived only via a fallback that read structured tool artifacts. The primary path was a bet on a language model's phrasing habits.

Nanoseconds Broke the Parser

After deleting dead formatting, Go's small-N runs became so fast that time.Duration output changed units: Elapsed time: 836ns. The parser had branches for µs, ms, and s, but never met a nanosecond. The fix made the code too fast for its own benchmark.

The LLM Got Bored

Reruns caused three missing Go datapoints. The captured response: "I already did that. Do you want to do it again?" The harness reused deterministic context IDs; ADK keeps per-context session history. On rerun, Gemini declined to redo work. Unique per-run IDs fixed it.

The Chart Lied

The chart title: "A2A Round-Trip Time (including LLM/Tool calling)". Only half true—two agents route through Gemini, two are direct HTTP handlers. Median RTT: 2.6ms and 4.6ms for direct, ~1.6s and ~1.8s for Gemini. A ~400× gap presented as language comparison was actually pipeline comparison.

Nine Bugs, Four PRs

The full list: (1) CPython int→str crash, (2) regex on LLM prose, (3) nanosecond parsing, (4) LLM refusal to repeat, (5) Node/Rust reporting "Found first 100 Mersenne primes" (they only have 26), (6) Rust's %.2f ms rounding to 0.00ms (breaks log scale), (7) chart title misleading, (8) benchmark stopped at N=22 as workaround, (9) the workaround committed as run_benchmark_1_22.py.

The Takeaway

run_benchmark_1_22.py sat in the repo like a fossil of an uninvestigated crash. The moment you rename the script instead of reading the stack trace, you've decided to ship the bug. Benchmarks are production code—they crash, lie about counts, round away smallest measurements, and compare different architectures on one axis.

If a machine consumes output, never route it through prose. Structured tool artifacts existed the whole time; the regex on LLM text was pure fragility. An LLM in the measurement path adds failure modes including boredom.

To reproduce: docker run --rm -e GEMINI_API_KEY=your_key -v "$PWD/out:/out" xbill9/bugsmash runs all four fixed agents and writes charts to ./out. The bugs, numbers, and "I already did that" refusal are real and archived in the repo.

Editor's Take

I've been guilty of the same sin: renaming a failing script to 'workaround' instead of reading the error. The CPython 4,300-digit limit caught me off guard too—I hit it in a data pipeline last year. The nanosecond parser break is my favorite: it's a perfect example of optimizations creating new edge cases. Honestly, this article should be required reading for anyone building automated benchmarks with LLMs.

— DevDigest Editorial

Key Takeaways

•Always use structured tool artifacts instead of parsing LLM prose for measurements.
•Add unit tests for parser edge cases (e.g., nanoseconds, different time units).
•Use unique context IDs per run to prevent LLM session history from affecting results.

Why It Matters

This story exposes how easy it is to build a benchmark that's quietly wrong in multiple ways. Every developer who writes performance tests should read this to avoid the same pitfalls—especially when mixing LLMs into measurement pipelines.

#go#benchmarking#debugging#Python#llm

Get the weekly digest

Every Sunday - top tech stories, industry breakthroughs, and developer tools delivered to your inbox.

No spam, unsubscribe anytime.