Reading a strategy paper is one thing.
Reproducing it is another.
QuantCodeEval evaluates coding-agent systems that reproduce quantitative strategies from finance papers. Thirty tasks. Sixteen systems. 2,400 attempts. We evaluate the strategy behavior implemented in code — not only an aggregate backtest metric.
Reproducing a strategy means getting every stage right.
A single Sharpe ratio won't tell you if any of them is.
A strategy is built in stages. Each can be silently wrong — and end-to-end metrics can't tell you which.
So we verify each stage with property checkers — not a single score.
Structural integrity every backtest must respect — point-in-time alignment, no look-ahead, survivorship handling.
Paper-specific strategy semantics — formulas, timing, model components, and portfolio rules — under the declared operational setting.
An attempt passes only when every checker in the task contract is green. Missing checker output makes the suite incomplete.
Even the best system passes fewer than one in three attempts.
Claude Code + Opus 5 leads pass@1 and pass@5.
Claude Code + Opus 5 reaches 31.3% pass@1 and 46.7% pass@5. Across the full benchmark, every system fails most attempts, and five independent attempts still leave more than half of the tasks unsolved.
| # | System | pass@1 | pass@5 |
|---|---|---|---|
| 01 | Claude Code + Opus 5 native scaffold | 31.3% | 46.7% |
| 02 | Codex + GPT-5.6-sol native scaffold | 27.3% | 43.3% |
| 03 | Claude Code + Opus 4.8 native scaffold | 27.3% | 40.0% |
| 04 | OpenCode + Kimi-K3 shared OpenCode scaffold | 26.0% | 40.0% |
| 05 | Codex + GPT-5.5 native scaffold | 24.7% | 36.7% |
| 06 | Claude Code + Opus 4.7 native scaffold | 24.0% | 43.3% |
| 07 | OpenCode + Opus 4.7 shared OpenCode scaffold | 22.7% | 40.0% |
| 08 | OpenCode + GPT-5.5 shared OpenCode scaffold | 21.3% | 33.3% |
| 09 | OpenCode + Sonnet 4.6 shared OpenCode scaffold | 18.7% | 36.7% |
| 10 | OpenCode + Qwen3.7-Max shared OpenCode scaffold | 18.0% | 30.0% |
| 11 | OpenCode + MiniMax-M3 shared OpenCode scaffold | 14.7% | 26.7% |
| 12 | OpenCode + GLM-5.2 shared OpenCode scaffold | 14.0% | 30.0% |
| 13 | OpenCode + GLM-5.1 shared OpenCode scaffold | 12.0% | 23.3% |
| 14 | OpenCode + DeepSeek V4-Pro shared OpenCode scaffold | 12.0% | 20.0% |
| 15 | OpenCode + Qwen 3.6 Plus shared OpenCode scaffold | 10.7% | 20.0% |
| 16 | OpenCode + MiniMax M2.7 shared OpenCode scaffold | 3.3% | 10.0% |
pass@1 averages over five independent attempts per task. pass@5 is the fraction of tasks where at least one attempt cleared every property checker. Resource columns report per-attempt medians.
Failures concentrate in three finance-critical requirements.
Among the 37 paper-specific properties with failure rates above 50%, errors concentrate in temporal consistency, finance-specific implementation, and trading direction.
Lag, expanding window, warmup, out-of-sample timing. Most often: contemporaneous predictors where the paper specified a lag.
Paper-specific formulas silently replaced by library defaults — cumulative residual becomes a rolling sum, tr(Σ̂) becomes tr(Σ).
Signal sign flipped on the way to a trade: high-minus-low portfolio implemented as low-minus-high. Rare, ruinous.
The best observed A-complete@1 and B-complete@1 rates are achieved by different systems. Agents can follow many self-contained rules, but preserving paper-specific semantics across stages remains a major source of error.
Implementation is the dominant failure stage.
Human review of SOTA-5 failures assigns each failed property to the earliest observable breakdown — Retrieval, Comprehension, Mid-process drift, or Implementation. The all-system view is a broader diagnostic analysis.
Frequencies are computed over 1,529 failed-property rows in the human-reviewed SOTA-5 analysis. The remaining 6.3% are grouped as Others and excluded from the four bars above.
Frequencies are computed over 7,017 failed-property rows in the all-system diagnostic analysis. The remaining 6.0% are grouped as Others and excluded from the four bars above.
Six representative tasks across the benchmark.
The full benchmark spans 30 papers across six strategy families. Open a task to inspect its paper, declared task setup, property checks, and outcomes across all 16 systems.
Empirical Asset Pricing via Machine Learning
Asset pricing: A tale of two days
Time Series Momentum
Empirical Investigation of an Equity Pairs Trading Strategy
Universal Portfolio Shrinkage
Volatility-Managed Portfolios
Reproduce the reported results from the release.
Ten tasks include bundled public data. The other twenty use one-command fetchers for WRDS-sourced inputs. The release also includes task contracts, checkers, traces, canonical result tables, and one command to regenerate the paper figures.
10K+ Hugging Face downloads · live all-time count@misc{quantcodeeval2026,
title = {QuantCodeEval: Benchmarking Quantitative Strategy Code
Reproduction from Finance Papers},
author = {Anonymous Authors},
year = {2026},
note = {Under double-blind peer review}
}