Paper sample

AI Agents and Prompt Engineering in Econometric Coding

Sebastian Galiani (Tulane University and NBER), Federico A. López, and Raul A. Sosa (Universidad de San Andrés). Updated version of NBER Working Paper 35588.

A study of how large language models write econometric code in Stata, R, and Python with zero-shot or one-shot prompts under three agency conditions: chatbot, self-repair, and constrained agent.

Open the paper (PDF) Presentation

Runs Tasks Models

Benchmark board

Benchmark results

How a run is scored

Scored estimatesEach scored estimate lies within 1 percent or 0.005 of the reference output, whichever is larger.

Scored standard errorsIn nine tasks, whose main decision is the standard error, the standard errors must also match.

Loud failureThe script does not run or returns no complete output.

Silent failureThe script returns every scored value, but at least one of them is wrong.

Leaderboard view

Task success by software and agency condition

Each cluster is one software environment, and each bar is one agency condition. The value of each bar is printed under it.

Cost and quality, together

Costs price every token at API list prices. Rows compare agency conditions within each model, and the last column gives the change from the row above.

Failure outcomes

A failure is loud when the script does not run or returns no complete output, and silent when it returns every scored value but at least one of them is wrong.

Task explorer

Analyzed econometric tasks

The list groups the tasks by method. Open a row to see its prompts, then browse its runs. Each task has a page of its own, and the task catalog lists the tasks with their sources.

Search Sort

Downloads

Paper and data

The results file holds the rates and costs that this site shows, for the two models.