389 Tests Passed, One NIST Assertion Failed
A developer gave an AI agent a calculator written in Rust. The idea was simple: let the model interpret requests, and let the calculator handle arithmetic deterministically. But when they mutated a single * to + in the multiple-regression standard-error calculation, 389 of 390 library tests still passed. The sole failure was an assertion against NIST's certified results for the Longley regression dataset.
That failure was more revealing than a completely broken build. The calculator was deterministic—it returned the same wrong answer consistently. Deterministic doesn't mean trustworthy.
The Mutation: * to +
The developer used cargo-mutants to replace multiplication with addition:
// Before:
residual_std_dev * sum_sq.sqrt()
// After mutation:
residual_std_dev + sum_sq.sqrt()
The mutation preserved valid Rust, valid types, and a plausible numeric result. Yet 389 tests passed. The only test that caught it compared against NIST's Statistical Reference Datasets (StRD), specifically the Longley dataset.
NIST carried 500 digits through its calculations for linear procedures, so floating-point errors wouldn't distort the benchmark. The Longley dataset is small but numerically challenging—perfect for catching subtle bugs.
Why NIST Matters
The developer chose NIST not for its authority, but because its reference data comes from outside the code under test. The StRD project (started in 1999) provides datasets with certified expected values. NIST explicitly says these datasets are for evaluating software, not for establishing traceability. They are independent witnesses, not oracles.
Contract and Resource Boundaries
The audit didn't stop at numerical correctness. The developer found two contract violations:
- Schema vs. deserializer mismatch: The JSON Schema forbade additional properties, but the Rust deserializer silently accepted unknown fields at the request root and inside expression nodes. A caller validating against the schema saw a stricter instrument than one talking directly to the binary.
- Unbounded resource consumption: The optimizer interface accepted grid resolution and iteration counts without upper limits. A request for one billion sample points caused the binary to hang until an external watchdog killed it after one second.
Both issues were fixed: the deserializer now rejects unknown fields, and the optimizer enforces ceilings of 100,000 iterations and 1,000,000 grid points. The same billion-point request now returns a typed resource_limit response.
Mutation Testing the Fixes
After repairing the optimizer limits, the developer ran cargo-mutants again on the two new validators. Eleven mutations tried to remove checks, replace them with unconditional success, or alter boundary comparisons. All eleven were caught by the tests.
This completed the loop: claim → attack → fail → repair → mutate → replay.
A Five-Layer Challenge Stack
The developer proposes five questions before trusting any narrow tool in an agent workflow:
- Is the contract explicit and executable? Schema, parser, runtime, limits, and failure modes must agree.
- Is there an independent reference? NIST StRD for stats; other domains might use standards specs or reference implementations.
- Which properties should survive across many inputs? Property-based tests check laws, not examples. One property in this calculator checks that translating all values by a large offset leaves sample variance unchanged.
- Can the tests detect plausible corruption? Mutation testing exposes behavior changes the suite can't distinguish.
- Are failures bounded, typed, and replayable? Division by zero, resource exhaustion, and malformed JSON should return distinct errors, not crash.
The Result: Authority Revocable by Evidence
The experiment didn't produce a calculator that can be trusted forever. It produced a calculator whose authority can be revoked by evidence. The source code can be cross-examined, and new evidence can always move the tool from trusted back to distrusted.
For developers building AI agents that delegate to deterministic tools, this is a blueprint: use independent reference data, mutation-test your tests, and enforce contracts at runtime. A green test suite isn't proof—it's only evidence until a better test proves otherwise.
