AI Agents & Econometric Coding

Paper sample

AI Agents and Prompt Engineering in Econometric Coding

A controlled study of how large language models write econometric code in Stata, R, and Python with zero-shot or few-shot prompts under three agency conditions: chatbot, self-repair, and constrained agent.

Open manuscript View the talk Explore results Review task set
Runs Tasks Models

Benchmark board

Benchmark results

What the scoring checks

Right numbersReported values match the reference output within the declared tolerances.
Valid submissionThe run solves the task from the declared inputs.
It runsDiagnostic. The generated script executes without manual repair.
Right methodDiagnostic. The estimator and convention match the task contract.

Leaderboard view

Task success by software and agent interface

Each cluster is one software platform, and each bar is one agency condition.

Where runs fail

Pass rate for each of the four success criteria. A run can execute and still fail when it estimates the wrong target or writes the wrong result.

Cost and quality, together

Each model has a separate cost comparison. Claude costs are billed amounts. GPT-5.4 costs use API list prices. Rows compare agency conditions within each model.

Diagnostic failures

Counts classify the first failed check under the strict five-check diagnostic. This diagnostic also checks execution, result-file structure, and method evidence.

Task explorer

Analyzed econometric tasks

The list groups the tasks by method: binary regression, OLS, difference-in-differences, instrumental variables, regression discontinuity, time series, Poisson regression, and a group of single tests and diagnostics. Each row has a page of its own, and the task catalog lists the paper’s tasks with their sources.

Downloads

Paper, data, and task definitions

The results file contains the paper’s two-model sample.