Instrumental variables
Weighted 2SLS spillover effects
Population-weighted spillovers among nonrecipients.
The task
Task statement
Use households.csv and census.csv. Keep all households with recipient=0. This includes ineligible households in both village treatment statuses. Fit weighted 2SLS with two endogenous columns dose*(1-eligible) and dose*eligible. Exogenous columns are 1-eligible, eligible, baseline*(1-eligible), and baseline*eligible. Instruments are these four columns plus instrument*(1-eligible) and instrument*eligible. Add no other intercept. Use weight as the regression weight. For each eligibility group, the spillover effect is its dose slope times the household-weighted mean dose among its retained observations. Report ineligible_effect and eligible_nonrecipient_effect. Average these two effects using nonrecipient_population from census.csv. Report population_effect. The counterfactual sets dose to zero for every retained household. Census totals determine only the final between-group weights. For every result, n_obs is the number of nonrecipient household rows used in estimation. The group effect equals the fitted dose slope times its weighted mean dose. Attribution to a zero-dose counterfactual is a stated synthetic convention.
Data
households.csv: household = Synthetic household identifier; eligible = Eligibility-group indicator, zero or one; treated = Own village treatment indicator; recipient = Indicator for actually receiving the transfer; instrument = Supplied excluded instrument for dose; dose = Supplied treatment exposure; baseline = Supplied baseline covariate; weight = Positive observation weight for estimation and within-group averages; outcome = Synthetic dependent variable in the stated regression census.csv: eligible = Eligibility-group indicator, zero or one; nonrecipient_population = Census count used only for final group aggregation
- Files
- input.csv, households.csv, census.csv
Required output
Scored estimates: ineligible_effect, eligible_nonrecipient_effect, population_effect.
Reference output
| Term | Reference estimate | Sample size |
|---|---|---|
ineligible_effect | 2.0530515 | 300 |
eligible_nonrecipient_effect | 5.7819293 | 300 |
population_effect | 2.649672 | 300 |
A run succeeds when every scored estimate lies within 1 percent or 0.005 of its reference value, whichever is larger. Sample sizes are not scored.
Task success
| Agency condition | Claude Sonnet 4.6 | GPT-5.6 Luna |
|---|---|---|
| Chatbot | 80.0% | 81.7% |
| Self-repair | 100.0% | 100.0% |
| Constrained agent | 100.0% | 100.0% |
- All runs
- 93.6% of 360 runs
- By prompt condition
- Zero-shot 94.4%, One-shot 92.8%
- By software
- Python 88.3%, R 97.5%, Stata 95.0%
Sources
Sources
- Egger, Haushofer, Miguel, Niehaus, and Walker (2022), General Equilibrium Effects of Cash Transfers: Experimental Evidence from Kenya
- Egger et al. (2026), corrected replication materials for General Equilibrium Effects of Cash Transfers
- Cameron and Trivedi (2005), Microeconometrics: Methods and Applications
About this benchmark task
- Method group
- Instrumental variables
- Software
- Stata, R, and Python
- Runs
- 360 runs, ten in each combination of model, software, prompt, and agency condition