Difference-in-differences
Cohort-specific difference-in-differences
Cohort-specific DiD with treated-unit-count weights.
The task
Task statement
Use Python. Read ./input.csv. Available columns: id, time, g, y.
Construct separate cohort-specific difference-in-differences comparisons for adoption cohorts g=3,5,7. g=0 denotes never treated. For cohort c, retain calendar times c-2 through c+2. Eligible units have g=c, g=0, or g>c+2. Within that comparison, retain only units observed at all five window times. Do this balance check separately for each cohort. Define did=1 for units with g=c at times at least c, and 0 otherwise. Regress y on did, unit fixed effects, and calendar-time fixed effects separately for each cohort. Report the did coefficients as att_g3, att_g5, att_g7. No standard errors are requested. Their n_obs is the number of retained unit-time rows in that cohort comparison, including controls. Then report att_weighted as the weighted average of those three coefficients, using each comparison's number of retained treated units as its weight. For att_weighted, n_obs is the sum of the three comparison row counts, including rows reused across comparisons. Use se=null for every row. Do not fit a single unweighted regression to concatenated comparisons. The columns are id (unit), time (calendar period), g (adoption period), and y (outcome). Missing observations are absent rows. No retained row has a missing outcome. Unit-time keys are unique. Calendar times run from 1 through 10.
Report the requested point estimates. Standard errors are not requested. Use json.dump to write result.json with the following structure. Replace each <number> with the computed value. Report at least eight significant digits.
{
"task_id": "broad_cohort_stack",
"platform": "python",
"estimates": [
{
"term": "att_g3",
"estimate": "<number>",
"se": null,
"n_obs": 375
},
{
"term": "att_g5",
"estimate": "<number>",
"se": null,
"n_obs": 275
},
{
"term": "att_g7",
"estimate": "<number>",
"se": null,
"n_obs": 315
},
{
"term": "att_weighted",
"estimate": "<number>",
"se": null,
"n_obs": 965
}
],
"method_metadata": {},
"diagnostics": {}
}Data
One shared fixed dataset with 1485 rows. Columns: id, time, g, y.
- Files
- input.csv
Required output
Scored estimates: att_g3, att_g5, att_g7, att_weighted.
Reference output
| Term | Reference estimate | Sample size |
|---|---|---|
att_g3 | 0.95422298 | 375 |
att_g5 | 1.2377404 | 275 |
att_g7 | 2.1133992 | 315 |
att_weighted | 1.5988696 | 965 |
A run succeeds when every scored estimate lies within 1 percent or 0.005 of its reference value, whichever is larger. Sample sizes are not scored.
Task success
| Agency condition | Claude Sonnet 4.6 | GPT-5.6 Luna |
|---|---|---|
| Chatbot | 83.3% | 73.3% |
| Self-repair | 90.0% | 98.3% |
| Constrained agent | 98.3% | 98.3% |
- All runs
- 90.3% of 360 runs
- By prompt condition
- Zero-shot 83.3%, One-shot 97.2%
- By software
- Python 87.5%, R 95.0%, Stata 88.3%
Sources
Sources
About this benchmark task
- Method group
- Difference-in-differences
- Software
- Stata, R, and Python
- Runs
- 360 runs, ten in each combination of model, software, prompt, and agency condition