Discrete choice
Multinomial logit
Multinomial choice probabilities.
The task
Task statement
Estimate a multinomial logit model for y using an intercept, x, and z. Treat 0 as the base outcome. Outcomes 1 and 2 are unordered alternatives. Use maximum likelihood and conventional observed-information standard errors. Report x_cat1 and z_cat1 for the slopes in log(P(y=1)/P(y=0)). Report x_cat2 and z_cat2 for log(P(y=2)/P(y=0)). Also report prob_cat2_zero as the fitted probability of outcome 2 at x=0 and z=0, with se=null. Report n_obs as the full sample size for every row.
Data
One shared fixed dataset with 480 rows. Columns: id, y, x, z.
- Files
- input.csv
Required output
Scored estimates: x_cat1, z_cat1, x_cat2, z_cat2, prob_cat2_zero.
Reference output
| Term | Reference estimate | Sample size |
|---|---|---|
x_cat1 | 0.42685659 | 480 |
z_cat1 | -0.61930779 | 480 |
x_cat2 | -0.34464356 | 480 |
z_cat2 | 0.39864409 | 480 |
prob_cat2_zero | 0.22344972 | 480 |
A run succeeds when every scored estimate lies within 1 percent or 0.005 of its reference value, whichever is larger. Sample sizes are not scored.
Task success
| Agency condition | Claude Sonnet 4.6 | GPT-5.6 Luna |
|---|---|---|
| Chatbot | 70.0% | 91.7% |
| Self-repair | 100.0% | 93.3% |
| Constrained agent | 100.0% | 98.3% |
- All runs
- 92.2% of 360 runs
- By prompt condition
- Zero-shot 87.2%, One-shot 97.2%
- By software
- Python 90.8%, R 87.5%, Stata 98.3%
About this benchmark task
- Method group
- Discrete choice
- Software
- Stata, R, and Python
- Runs
- 360 runs, ten in each combination of model, software, prompt, and agency condition