AI Agents and Prompt Engineering in Econometric Coding
How reliably do large language models implement econometric specifications?
Sebastian Galiani Tulane / NBER · Federico A. López UdeSA · Raúl A. Sosa UdeSA
runs · tasks ·
Models can produce executable econometric code
An LLM can write code for an econometric specification in Python, Stata, or R. A coding agent can also run and revise the script, with additional model cost.
Execution alone leaves specification and target errors undetected.
- It can return the wrong numbers.
- It can make modeling choices no one asked for.
- It can miss the original question entirely.
The experiment varies three dimensions
Econometric software
Same task contract, answered with Python code.
Same task contract, answered with a Stata do-file.
Same task contract, answered with R code.
Agent interface
Writes the code once. It does not run the script.
Writes the code, sees run reports, and revises before it submits. Each harness caps the revisions.
Controls a shell-execution loop for up to twelve model turns and decides when to submit.
An example
Each run is one cell in the design: a task, a software platform, and an agent interface. The model receives the task instructions and must leave behind a result file that the scorer can check.
Estimate a regression and report the coefficient, standard error, and sample size in result.json.
Stata for this cell; the same task is also asked in Python and R cells.
Self-repair: draft code, observe run reports, then submit the repaired script.
task robust-SE regression
software Stata
interface Self-repair
flow draft -> report -> repair -> scoreThe analysis sample contains runs built from cells like this
What counts as a task success
Output matches the reference within the declared tolerances
The run solves the task from the declared inputs
Diagnostic. The script executes without error
Diagnostic. It set up the correct approach and target
Scored on runs
Result 1: task success by agent interface
· both prompt conditions pooled
Result 2: task success by software platform
·
Task success by software and agent interface
Each cluster is one software platform, and each bar is one agent interface. Task success is highest for Constrained agent within Python, R, and Stata.
· Software × agency condition
Result 3: execution leaves specification errors
The lower-scoring tasks require the exact estimator, estimand, variance rule, or output terms.
· Task-level comparison
Cost and quality, together
Each row is one agency condition. The bar reports task success, mean cost reports price, and the incremental column compares adjacent displayed conditions.
· Mean cost per run · success bars start at 50%
Robustness across models and providers
The same design runs on .
Held fixed
The same task contracts and fixed datasets.
The same reference outputs, tolerances, and integrity checks.
The same software platforms, prompts, and agent interfaces.
Compared across models and providers
Whether conclusions about software, prompting, and interface depend on the model.
The model expenditure associated with each condition.
The expenditure required to obtain a correct result.
Explore the paper
runs · tasks ·