Opus 5 Scores 24% on SlopCodeBench: The Unsaturated Benchmark That Measures Code Quality Over Time

Last Friday, I ran three Claude models (Opus 4.8, Sonnet 5, and Opus 5) through a subset of SlopCodeBench, a new long-horizon coding benchmark from UW Madison's @GOrlanski lab. The headline: Opus 5 achieved a 24% strict pass rate on a 17-checkpoint subset, barely beating Opus 4.6's 17% from the original paper. None of the models completed a single challenge without defects.

What Makes SlopCodeBench Different

Unlike typical coding benchmarks that reveal the entire problem upfront, SlopCodeBench's challenges have multiple checkpoints. The model must evolve the codebase as new requirements are divulged. For example, the circuit_eval challenge starts with a CLI that parses single-bit circuits, then adds evaluation, then vector signals with slicing and concatenation. This incremental disclosure mimics real-world software engineering where requirements change over time.

The strict pass criteria means a model must pass all tests inherited from previous checkpoints. If it breaks something in checkpoint 4, it can't recover unless it inadvertently fixes it later—which didn't happen in practice.

The Benchmark Subset

I selected three problems from the SlopCodeBench repository:

  • circuit_eval (easy, 8 checkpoints)
  • database_migration (medium, 5 checkpoints)
  • dynamic_config_service_api (hard, 4 checkpoints)

All models ran in parallel with fresh context windows per checkpoint, using the Claude Code harness. The evaluation harness ran black-box tests against the produced entrypoint after each checkpoint.

Results: Opus 5 Wins, but Barely

Opus 5 scored 4 strict passes out of 17 (24%): the first three checkpoints of circuit_eval and checkpoint 1 of database_migration. Opus 4.8 and Sonnet 5 each got one strict pass (6%)—the same database_migration checkpoint 1.

For circuit_eval, Opus 5 aced the first three checkpoints but accumulated defects at checkpoints 4 and 5, failing all subsequent ones. Sonnet started cheapest but became the most cost-effective by the end of problem 1, as the work shifted to maintenance.

Code Quality Metrics: The Slop Meter

SlopCodeBench computes 41 code quality metrics after each checkpoint, grouped into:

  • Size: lines, functions, classes
  • Complexity: cyclomatic complexity mean/max, nesting depth
  • Duplication: cloned lines ratio
  • Decomposition: single-use functions, unused variables
  • Rule violations: lint errors, ast-grep slop hits
  • Dependency graph: propagation cost, cyclic dependencies

Key findings:

  • Duplication: Opus 4.8's duplication rose from 4.6% to 16.8% over eight checkpoints. Opus 5 stayed flat at ~2.5%.
  • Complexity: Opus 4.8's worst function hit cyclomatic complexity 93. Opus 5 had the lowest mean complexity but wrote 5x more functions than Opus 4.8.
  • Verbosity: Lines flagged as verbose increased from 65% at checkpoint 1 to 80% by checkpoint 8 for all models.
  • Slop triggers: 93% of Opus 5's code lines triggered at least one slop rule, compared to 98% for Opus 4.8 and 89% for Sonnet 5.

The Cost of Correctness

Opus 5 wrote five times the number of functions/callables as Opus 4.8 over the same challenges. Much of that was additional tests, but production code volume was still ~1.8x. The cost per dollar bought correctness, but not enough to reach the final checkpoint.

What This Means for Developers

SlopCodeBench provides the first unsaturated benchmark for long-horizon coding tasks. The 24% pass rate for Opus 5 indicates that even frontier models cannot be trusted to run lights-off without human steering. Code quality degrades over time—complexity grows, duplication spreads, and verbosity increases. While metrics like duplication show some improvement between model generations (Opus 5 flat vs. Opus 4.8's 12-point rise), the overall picture is clear: today's models still struggle to maintain codebases incrementally.

Next Steps

I'll run Fable and 5.6 Sol on the full benchmark soon. For now, the takeaway is pragmatic: use AI for single-shot tasks, but review generated code thoroughly for long-running projects. The slop meter is a useful directional signal, but not yet a definitive oracle for maintainability.

Note: The source article is from a GitHub markdown file. All metrics and quotes are from that source.