Binary response
Heteroskedastic probit
Heteroskedastic probit with normalized scale.
The task
Task statement
Fit a heteroskedastic probit by maximum likelihood on all rows. The success probability is Phi((beta0+beta_x*x+beta_d*d)/exp(gamma_z*z)). The scale equation has z only and no intercept; therefore sigma=1 when z=0. This normalization is fixed. The mean equation includes an intercept. Report gamma_z. Report ame_x, the average derivative of the fitted success probability with respect to continuous x, holding d and z fixed. Report effect_d, the average fitted probability difference when d changes from 0 to 1, holding each row's x and z fixed. Use equal weights and all supplied rows for both averages. Set n_obs to the full row count for every result.
Data
Simulated CSV. Columns: y, x, d, z
- Files
- input.csv
- Seed
- 2026091702
Required output
Scored estimates: gamma_z, ame_x, effect_d.
Reference output
| Term | Reference estimate | Sample size |
|---|---|---|
gamma_z | 0.60611433 | 1,400 |
ame_x | 0.21556723 | 1,400 |
effect_d | -0.18096924 | 1,400 |
A run succeeds when every scored estimate lies within 1 percent or 0.005 of its reference value, whichever is larger. Sample sizes are not scored.
Task success
| Agency condition | Claude Sonnet 4.6 | GPT-5.6 Luna |
|---|---|---|
| Chatbot | 66.7% | 68.3% |
| Self-repair | 83.3% | 83.3% |
| Constrained agent | 100.0% | 96.7% |
- All runs
- 83.1% of 360 runs
- By prompt condition
- Zero-shot 82.8%, One-shot 83.3%
- By software
- Python 92.5%, R 96.7%, Stata 60.0%
About this benchmark task
- Method group
- Binary response
- Software
- Stata, R, and Python
- Runs
- 360 runs, ten in each combination of model, software, prompt, and agency condition