\documentclass[../main.tex]{subfiles}
\begin{document}

\section{Design}
\label{sec:design}
% ================================================================

The design has three dimensions: the prompt (zero-shot or few-shot), the degree of agency (chatbot, self-repair, or constrained agent), and the statistical software (Stata, R, or Python).
Each of the two models, Claude Sonnet 4.6 in the Claude Code harness and GPT-5.4 in the Codex harness, attempts 21 econometric tasks under every combination of prompt, agency condition, and software.
Each combination of task, software, prompt, and agency condition forms one cell of the design.
Because the output of a model varies across otherwise identical runs, each model runs every cell twelve times.
The analysis sample therefore holds \numRunsPerArm{} runs per model and \numRunsTotal{} runs in total.
\Cref{fig:benchmark_design} summarizes the design.

\begin{figure}[t]
\centering
\includegraphics[width=\textwidth]{figures/benchmark_design.pdf}
\caption{Design of the benchmark. Each task specifies a dataset, a target estimand, a required result-file structure, a reference output, and tolerances. The prompt condition changes the task statement, the agency condition changes what the model can observe and do before it submits a result, and every run succeeds when its result file agrees with the reference output within tolerance. Each model runs every cell twelve times.}
\label{fig:benchmark_design}
\end{figure}

\subsection{Degree of agency}
\label{sec:implementation}

The agency conditions differ in what the model can see and do between receiving the task and submitting a result.
In every condition the harness sets the context the model receives, the tools it may call, the way its code is executed, the rule that ends the run, and the record the run leaves behind \citep{meng2026agentharness}.
Under the \emph{chatbot} condition, the harness obtains one complete script in a single model call, executes it once, and returns nothing to the model, as in a chat window where the researcher pastes a request and runs the script that comes back.
Under \emph{self-repair}, the harness executes the script in the same way, but when the script fails or writes no result file, the harness returns a fixed message with the captured run output, and the model may submit up to three revised scripts.
Under the \emph{constrained agent}, the model executes shell commands in a per-run working folder that holds the input data, and within a session time limit it writes, runs, and revises its script as often as it chooses and declares when it is finished.
The condition is constrained in that the model is limited to writing and running scripts for the task.\footnote{\Cref{app:sdk_protocol} records, for each harness, the tools it authorized and the limits it applied.}
In every condition, script executions and model calls also run under time limits, and a run that reaches a limit without a scored result file counts as a failure.

\subsection{Tasks and scoring}
\label{sec:evaluation}

Each task specifies a dataset, a target estimand, a required result-file structure, a reference output computed in advance, method checks, and numerical tolerances.
Every dataset is simulated, and the same dataset and the same reference output serve every software environment, prompt condition, agency condition, and repetition of a task, so the design dimensions change nothing about the data or the target.
The 21 tasks cover linear regression and its diagnostics, classical tests, generalized linear models, difference-in-differences, instrumental variables, regression discontinuity, dynamic panel estimation, survival analysis, and time series.\footnote{\Cref{app:tasks} summarizes the task definitions.}

The task statement is the same in both prompt conditions.
It names the software, the dataset and its variables, the quantity to estimate and the sample, the required estimator and inference convention, and the required structure and location of the result file.
The zero-shot prompt contains only that statement.
The few-shot prompt places one worked example before it: a short statement of a related task from the same method family, complete code in the target software on small inline data, and the result-file conventions that the scored task requires.
The example is never the scored task itself and does not mention its dataset, variables, or reference values.

The primary outcome is task success, which is binary.
A run succeeds when the result file it submitted reports estimates that match the stored reference output within the task's tolerances and when the script computed them from the task's input data, so a hard-coded answer or a read of the stored reference output fails.
The scoring system judges the final result file the run saved, whatever number of permitted revisions produced it.
Before the comparison, it applies a formatting fix to that file that changes no reported digit, leaves a truncated or empty file as a failure, and normalizes the conventions with which the three environments write numbers and empty fields, so output formatting does not drive the software comparison (\Cref{app:method_checks}).

Beyond the binary outcome, the scoring system stores five checks for every run: whether the script executed, whether it wrote a well-formed result file, whether the reported numbers agree with the reference output, whether the script shows evidence of the required method, and whether the submission is valid.
Task success requires the numeric-agreement and submission-validity checks, and the other three are recorded as diagnostics.
We call the verdict that requires all five checks the strict five-check diagnostic, and \Cref{sec:results} and \Cref{app:failure_cost} use it to describe how runs fail.

\subsection{Analysis and inference}
\label{sec:analysis}

We summarize task success as cell rates, the share of successful runs in each cell.
We estimate the effects of the design dimensions with a regression model in which each run is one observation and each task has its own intercept,
\begin{equation}
\label{eq:main}
\text{Success}_{ir} = \tau_i + X_{ir}'\beta + \varepsilon_{ir},
\end{equation}
where $\text{Success}_{ir}$ equals one if run $r$ of task $i$ succeeds and zero otherwise, $\tau_i$ is the task fixed effect, and $X_{ir}$ holds indicators for every combination of software, prompt, and agency condition in the $2\times 3\times 3$ design, with Python, zero-shot, and chatbot as the omitted categories.
Because $X_{ir}$ includes all interactions among the three dimensions \citep{angrist2009mostly} and every cell holds twelve runs for each model, the regression reproduces the observed success rate of each of the 18 combinations of software, prompt, and agency condition.
Every estimate we report is a difference between two such rates or between averages of them, so the regression organizes the cell comparisons and supplies their standard errors rather than imposing a functional form on them.

We estimate equation~\eqref{eq:main} separately for each model, and once on the two models pooled, with a fixed effect for every pair of model and task and with cell effects common to the two models.
Because each model contributes twelve runs to every cell, each pooled estimate is the simple average of the two per-model estimates.
\Cref{tab:estimates_three_models} reports the per-model estimates in columns 1 and 2 and the pooled estimates in column 3.
A second pooled specification with model-specific cell effects tests whether an estimate differs between the two models, with cluster-robust standard errors and a $t$ distribution with 20 degrees of freedom.

The software contrasts in \Cref{tab:estimates_three_models} compare R and Stata with Python among zero-shot chatbot runs, the baseline condition of the design.
The prompt contrast compares few-shot with zero-shot after averaging over the nine combinations of software and agency condition.
The agency contrasts compare self-repair and the constrained agent with the chatbot after averaging over the six combinations of software and prompt.
The interaction contrast measures how the few-shot effect changes under the constrained agent relative to the chatbot, averaged over the three software environments, and a positive interaction indicates complementarity between the example and agency, whereas a negative interaction indicates substitution.
\Cref{tab:estimates_three_models} also prints two descriptive contrasts for Stata against Python, one averaged over the six prompt and agency conditions and one for the zero-shot constrained agent.

Runs of the same task share its dataset and its difficulty, so we cluster standard errors at the task level.
Five datasets serve thirteen of the 21 tasks, and task-level clustering does not absorb the dependence that those shared datasets create across tasks (\Cref{app:tasks}).
With only 21 clusters, conventional cluster-robust inference can over-reject, so the headline $p$-values come from the wild cluster bootstrap with Rademacher weights, the null imposed, and $\numWcbReps$ replications \citep{cameron2008bootstrap,mackinnon2018wildboot}.
\Cref{tab:estimates_three_models} reports the cluster-robust standard errors in parentheses and marks significance levels from the bootstrap $p$-values.

% ================================================================

\end{document}