AI Agents and Prompt Engineering in Econometric Coding
We measure how often two language models write econometric code whose estimates match a reference output.
Sebastian Galiani Tulane and NBER · Federico A. López UdeSA · Raul A. Sosa UdeSA
Updated version of NBER Working Paper 35588 · runs · tasks ·
Models can write and run econometric code
An LLM can write the code for an econometric specification in Stata, R, or Python. A coding agent can also run and revise the script, at a higher model cost.
A script that runs and returns numbers can still be wrong.
- We measure how often the estimates match a reference output.
- We compare that rate across agency conditions, prompts, and software.
- We report what each condition costs.
The design varies three dimensions
Software
The same task, answered with Python code.
The same task, answered with a Stata do-file.
The same task, answered with R code.
Prompt
The task statement.
The task statement and one example on other data.
Agency condition
Writes one script. We run it once and return nothing to the model.
We run the script. If it fails or lacks a value, the model may revise it up to three times.
Runs and revises its own code in a shell as often as it chooses.
An example
Each run combines one task, software, prompt, agency condition, and model. We score the output of the final script against the reference output.
Estimate a regression with robust standard errors and report the coefficients. Python and R write result.json.
Stata in this example. The same task is also asked in Python and R.
Self-repair. We run the script and return a failed run for up to three revisions.
task OLS with robust standard errors
software Stata
agency Self-repair
flow script -> run -> output -> revision -> scoreThe analysis sample has runs built like this one
What counts as a task success
Within 1 percent or 0.005 of the reference output, whichever is larger
Also scored in the nine tasks whose main decision is the standard error
The script does not run or returns no complete output
The script returns every scored value, but at least one of them is wrong
Scored on runs
Result 1: task success by agency condition
· both prompt conditions pooled · bars from 0 to 100 percent
Result 2: task success by software
· · bars from 0 to 100 percent
Task success by software and agency condition
Each cluster is one software environment, and each bar is one agency condition, with its value under the bar. In each software environment, task success is highest under the constrained agent.
· Software × agency condition
Result 3: the agent removes loud failures
Under the constrained agent, no run fails loudly. Every failure that remains is silent, so the script runs and returns every scored value, but at least one of them is wrong.
· Task-level comparison
Cost and task success: Claude Sonnet 4.6
Each row is one agency condition. The last column gives the change in task success and in mean cost per run from the row above.
Mean cost per run at API list prices · bars from 0 to 100 percent
Cost and task success: GPT-5.6 Luna
Each row is one agency condition. The last column gives the change in task success and in mean cost per run from the row above.
Mean cost per run at API list prices · bars from 0 to 100 percent
Results by model
The same design runs on .
Held fixed
The same tasks and datasets.
The same reference outputs and tolerances.
The same software environments, prompts, and agency conditions.
Compared between the two models
Whether the effects of software, prompt, and agency differ between the models.
The model cost of each condition, at API list prices.
The cost of one successful run.
Explore the paper
Sebastian Galiani, Federico A. López, and Raul A. Sosa · runs · tasks ·