Task catalog
Tasks in the paper
The paper’s 40 econometric tasks, each solved in Stata, R, and Python, with task success for Claude Sonnet 4.6 and GPT-5.6 Luna.
In the benchmark (40)
| Task | Method group | Task success |
|---|---|---|
| OLS with robust standard errors | Linear regression | 91.1% |
| One-way clustered OLS | Linear regression | 99.2% |
| Two-way clustered OLS | Linear regression | 79.7% |
| Cook's distance | Linear regression | 98.9% |
| Variance inflation factors | Linear regression | 97.5% |
| Welch t-test | Classical tests | 95.0% |
| One-way ANOVA | Classical tests | 85.8% |
| Chi-square independence test | Classical tests | 87.5% |
| Logit regression | Binary response | 100.0% |
| Probit regression | Binary response | 99.7% |
| Probit interaction effects | Binary response | 89.4% |
| Heteroskedastic probit | Binary response | 83.1% |
| Poisson regression | Poisson and count models | 98.6% |
| PPML with separated zeros | Poisson and count models | 77.8% |
| Poisson hurdle effects | Poisson and count models | 83.6% |
| Tobit regression | Censored and truncated models | 90.3% |
| Truncated normal regression | Censored and truncated models | 76.7% |
| Multinomial logit | Discrete choice | 92.2% |
| Ordered logit | Discrete choice | 93.3% |
| Conditional logit welfare | Discrete choice | 85.0% |
| Staggered difference-in-differences | Difference-in-differences | 98.1% |
| Cohort-specific difference-in-differences | Difference-in-differences | 90.3% |
| Two-way fixed effects | Difference-in-differences | 88.6% |
| Two-stage least squares | Instrumental variables | 93.6% |
| Shift-share IV | Instrumental variables | 72.8% |
| Multi-vintage exposure IV | Instrumental variables | 73.3% |
| Weighted 2SLS spillover effects | Instrumental variables | 93.6% |
| Anderson–Rubin confidence set | Instrumental variables | 83.1% |
| Panel local-projection IV | Instrumental variables | 90.0% |
| Sharp regression discontinuity | Regression discontinuity | 98.6% |
| Fuzzy regression discontinuity | Regression discontinuity | 85.8% |
| Cox proportional hazards | Survival analysis | 91.9% |
| Discrete-time conditional logit | Survival analysis | 83.1% |
| ARIMA forecast | Time series | 88.1% |
| Granger causality | Time series | 98.9% |
| VAR impulse response | Time series | 84.4% |
| Local-projection impulse response | Time series | 92.5% |
| Cumulative lag effects (HAC) | Time series | 79.2% |
| Rolling AR(1) forecasts | Time series | 90.3% |
| Oaxaca decomposition | Decomposition | 97.8% |