AI Agents & Econometric Coding

Difference-in-differences

Cohort-specific difference-in-differences

Cohort-specific DiD with treated-unit-count weights.

The task

Task statement

Use Python. Read ./input.csv. Available columns: id, time, g, y.

Construct separate cohort-specific difference-in-differences comparisons for adoption cohorts g=3,5,7. g=0 denotes never treated. For cohort c, retain calendar times c-2 through c+2. Eligible units have g=c, g=0, or g>c+2. Within that comparison, retain only units observed at all five window times. Do this balance check separately for each cohort. Define did=1 for units with g=c at times at least c, and 0 otherwise. Regress y on did, unit fixed effects, and calendar-time fixed effects separately for each cohort. Report the did coefficients as att_g3, att_g5, att_g7. No standard errors are requested. Their n_obs is the number of retained unit-time rows in that cohort comparison, including controls. Then report att_weighted as the weighted average of those three coefficients, using each comparison's number of retained treated units as its weight. For att_weighted, n_obs is the sum of the three comparison row counts, including rows reused across comparisons. Use se=null for every row. Do not fit a single unweighted regression to concatenated comparisons. The columns are id (unit), time (calendar period), g (adoption period), and y (outcome). Missing observations are absent rows. No retained row has a missing outcome. Unit-time keys are unique. Calendar times run from 1 through 10.

Report the requested point estimates. Standard errors are not requested. Use json.dump to write result.json with the following structure. Replace each <number> with the computed value. Report at least eight significant digits.
{
  "task_id": "broad_cohort_stack",
  "platform": "python",
  "estimates": [
    {
      "term": "att_g3",
      "estimate": "<number>",
      "se": null,
      "n_obs": 375
    },
    {
      "term": "att_g5",
      "estimate": "<number>",
      "se": null,
      "n_obs": 275
    },
    {
      "term": "att_g7",
      "estimate": "<number>",
      "se": null,
      "n_obs": 315
    },
    {
      "term": "att_weighted",
      "estimate": "<number>",
      "se": null,
      "n_obs": 965
    }
  ],
  "method_metadata": {},
  "diagnostics": {}
}

Data

One shared fixed dataset with 1485 rows. Columns: id, time, g, y.

Files
input.csv

Required output

Scored estimates: att_g3, att_g5, att_g7, att_weighted.

Reference output

TermReference estimateSample size
att_g30.95422298375
att_g51.2377404275
att_g72.1133992315
att_weighted1.5988696965

A run succeeds when every scored estimate lies within 1 percent or 0.005 of its reference value, whichever is larger. Sample sizes are not scored.

Task success

Agency conditionClaude Sonnet 4.6GPT-5.6 Luna
Chatbot83.3%73.3%
Self-repair90.0%98.3%
Constrained agent98.3%98.3%
All runs
90.3% of 360 runs
By prompt condition
Zero-shot 83.3%, One-shot 97.2%
By software
Python 87.5%, R 95.0%, Stata 88.3%

Sources

Sources

About this benchmark task

Method group
Difference-in-differences
Software
Stata, R, and Python
Runs
360 runs, ten in each combination of model, software, prompt, and agency condition

Open every recorded run for this task.