AI Agents & Econometric Coding

Binary response

Probit interaction effects

Probit effects with interaction and curvature.

The task

Task statement

Fit a Bernoulli probit model by maximum likelihood on all rows, with intercept and regressors x, d, x*d, z, and x squared. Do not weight observations. For the subset target==1, calculate the average marginal effect of continuous x, including both its interaction and squared term, and report ame_x. Also calculate the average discrete effect of changing d from 0 to 1 in this same subset, holding x and z at each row's observed values and updating x*d consistently, and report effect_d. Fit the model once on the full sample. Average individual effects over the stated target subset. Do not evaluate effects at mean covariates. For both rows, n_obs is the full estimation sample size. Evaluate the x derivative at each target row's observed x, d, and z.

Data

input.csv: y = Dependent outcome; x = Continuous regressor; d = Binary regressor; z = Additional exogenous covariate; target = Indicator for the population over which probability effects are averaged

Files
input.csv

Required output

Scored estimates: ame_x, effect_d.

Reference output

TermReference estimateSample size
ame_x0.061368467850
effect_d0.11255682850

A run succeeds when every scored estimate lies within 1 percent or 0.005 of its reference value, whichever is larger. Sample sizes are not scored.

Task success

Agency conditionClaude Sonnet 4.6GPT-5.6 Luna
Chatbot93.3%68.3%
Self-repair93.3%81.7%
Constrained agent100.0%100.0%
All runs
89.4% of 360 runs
By prompt condition
Zero-shot 91.7%, One-shot 87.2%
By software
Python 96.7%, R 99.2%, Stata 72.5%

Sources

Sources

About this benchmark task

Method group
Binary response
Software
Stata, R, and Python
Runs
360 runs, ten in each combination of model, software, prompt, and agency condition

Open every recorded run for this task.