Binary response
Probit interaction effects
Probit effects with interaction and curvature.
The task
Task statement
Fit a Bernoulli probit model by maximum likelihood on all rows, with intercept and regressors x, d, x*d, z, and x squared. Do not weight observations. For the subset target==1, calculate the average marginal effect of continuous x, including both its interaction and squared term, and report ame_x. Also calculate the average discrete effect of changing d from 0 to 1 in this same subset, holding x and z at each row's observed values and updating x*d consistently, and report effect_d. Fit the model once on the full sample. Average individual effects over the stated target subset. Do not evaluate effects at mean covariates. For both rows, n_obs is the full estimation sample size. Evaluate the x derivative at each target row's observed x, d, and z.
Data
input.csv: y = Dependent outcome; x = Continuous regressor; d = Binary regressor; z = Additional exogenous covariate; target = Indicator for the population over which probability effects are averaged
- Files
- input.csv
Required output
Scored estimates: ame_x, effect_d.
Reference output
| Term | Reference estimate | Sample size |
|---|---|---|
ame_x | 0.061368467 | 850 |
effect_d | 0.11255682 | 850 |
A run succeeds when every scored estimate lies within 1 percent or 0.005 of its reference value, whichever is larger. Sample sizes are not scored.
Task success
| Agency condition | Claude Sonnet 4.6 | GPT-5.6 Luna |
|---|---|---|
| Chatbot | 93.3% | 68.3% |
| Self-repair | 93.3% | 81.7% |
| Constrained agent | 100.0% | 100.0% |
- All runs
- 89.4% of 360 runs
- By prompt condition
- Zero-shot 91.7%, One-shot 87.2%
- By software
- Python 96.7%, R 99.2%, Stata 72.5%
Sources
Sources
About this benchmark task
- Method group
- Binary response
- Software
- Stata, R, and Python
- Runs
- 360 runs, ten in each combination of model, software, prompt, and agency condition