Each task combines a finance paper, a fixed instruction, and the data needed to reproduce its strategy. A coding-agent system reads these inputs and writes a backtestable Python implementation. The attempt passes only when every checker in the task contract returns true.
The QuantCodeEval paper is the authoritative reference for every definition, formula, statistic, and per-task detail. This page summarizes the methodology described there; in case of any wording difference, the paper is canonical.
The paper remains the primary methodological source. The fixed instruction resolves omitted or ambiguous execution choices and specifies the typed data and function interfaces used for evaluation.
For 22 tasks, it instantiates only the data mappings, interfaces, and evaluation conventions needed for execution. Eight tasks additionally use a bounded operational adaptation, fixed before model evaluation and documented in the release. The instruction defines the evaluation scope without prescribing an implementation.
The finance paper that defines the strategy's methodological logic. The release records the source identity and structured task materials.
A typed contract — function signatures plus clarification of paper details that are scattered or under-specified. What, not how.
Ten tasks include bundled public inputs. The other twenty provide one-command fetchers for data available through WRDS.
For each paper, two domain experts independently annotate the fine-grained requirements a valid reproduction must satisfy. Disagreements are resolved through joint discussion before the property list is finalized. Each property is then compiled into an executable checker that returns a binary pass/fail verdict. The final suite contains 572 checkers: 238 Type-A and 334 Type-B.
Correctness constraints that any valid and backtestable quantitative strategy must satisfy.
Derived from empirical studies of common backtest errors and shared across the benchmark.
The design choices that distinguish this paper's strategy from others in the same category.
Each requirement records its source evidence or its declared operational basis. 325 of 334 are anchored in the paper; the remaining 9 are explicitly identified as benchmark-specified.
Finance papers often describe strategies across scattered passages. For each paper, we implement the strategy and validate the complete pipeline with quantitative experts to obtain a golden reference. The paper remains the primary methodological source; the reference establishes that the declared task admits a coherent executable interpretation.
Inspecting the executable pipeline helps experts check Type-B coverage and cross-stage consistency, including timing, universe filtering, and missing-value handling. Supporting paper evidence and any operational clarification are recorded before evaluation.
Each property is converted to an executable checker with test inputs tailored to it. For simple properties (fixed parameter values, scalar computations) the checker exercises an individual pipeline stage; for properties that span multiple stages, the checker mutates real input data in ways likely to expose a violation — for example, injecting an artificially large future return to detect look-ahead bias.
We evaluate systems under both settings, following the convention introduced by HumanEval. pass@1 uses one attempt per task; pass@5 uses five independent attempts and counts the task as solved if any one of them faithfully reproduces the strategy. Failed executions, timeouts, missing outputs, function-signature violations, and incorrect checker results are all counted as failed submissions. Missing checker output also makes the corresponding property suite incomplete.
Each attempt is executed in an isolated Python virtual environment with a fixed toolchain, so code and results cannot leak across runs. The agent has no access to the golden reference or the checker suite at generation time.
For every failed property we inspect the agent's full trajectory and attribute the violation to one of four reasoning-pipeline failure types.
The agent misses relevant evidence from the paper or instruction. A property may depend on information spread across multiple paragraphs; the agent recovers the main equation but misses a critical textual constraint or timing condition.
The agent retrieves the evidence but misinterprets it — for example, treating a cumulative residual as a rolling-window sum, or substituting the original covariance matrix for its shrunk counterpart.
The agent initially understands the property but later deviates from it. Common when many factors or algorithmic layers are involved — e.g., the trace correctly names twenty boosted P-Trees, but the implementation plan silently drops to five.
The agent states the requirement correctly throughout reasoning, but the
submitted code does not realize it faithfully. Temporal misalignment is
the most common form: aligning the target with .shift(-1)
but then dating the output one step late.
We assess recall with controlled semantic violations and precision with human judgments of actual agent submissions. Mutation testing on the frozen suite yields 98.7% Type-A mutant recall and 87.1% mean task-level Type-B mutant recall. In a balanced audit of 300 real property evaluations, accepted decisions have 97.3% precision and rejected decisions have 91.3% precision; independent reviewers agree on 90.7% of cases before resolution.