Linear regression
OLS with robust standard errors
The OLS coefficient can remain consistent when error variance changes, while its classical standard error fails.
The task
Task statement
Estimate y_lin on x1 and x2 with an intercept. Use HC1 heteroskedasticity-robust standard errors. Report rows named const, x1, and x2.
Data
Columns: id, x1, x2, y_lin, y_bin_logit, y_bin_probit.
- Files
- input.csv
Required output
Scored estimates: const, x1, x2.
The standard errors of const, x1, x2 are also scored, because the main decision in this task is how to compute them.
Reference output
| Term | Reference estimate | Reference standard error | Sample size |
|---|---|---|---|
const | 1.0115765 | 0.021954265 | 2,000 |
x1 | 0.49266398 | 0.02280658 | 2,000 |
x2 | -0.32191135 | 0.021272206 | 2,000 |
A run succeeds when every scored estimate lies within 1 percent or 0.005 of its reference value, whichever is larger, and every scored standard error lies within 0.5 percent of the reference or of an accepted alternative convention. Sample sizes are not scored.
Task success
| Agency condition | Claude Sonnet 4.6 | GPT-5.6 Luna |
|---|---|---|
| Chatbot | 66.7% | 83.3% |
| Self-repair | 98.3% | 98.3% |
| Constrained agent | 100.0% | 100.0% |
- All runs
- 91.1% of 360 runs
- By prompt condition
- Zero-shot 91.1%, One-shot 91.1%
- By software
- Python 100.0%, R 98.3%, Stata 75.0%
One coefficient, two variance models
Ordinary least squares solves the same normal equations under constant or changing error variance. The covariance estimator must reflect the variance pattern (White, 1980).
The HC0 sandwich replaces the constant-variance middle term with observation-specific squared residuals:
HC1 multiplies this matrix by $n/(n-k)$.
A three-observation meat matrix
For a slope-only illustration, suppose $x=(1,2,3)$ and residuals are $(1,-1,2)$. Each row contributes $x_i^2\hat u_i^2$.
| Observation | $x_i^2$ | $\hat u_i^2$ | Contribution |
|---|---|---|---|
| 1 | 1 | 1 | 1 |
| 2 | 4 | 1 | 4 |
| 3 | 9 | 4 | 36 |
The last observation dominates the covariance estimate because it combines a large regressor with a large residual.
Assumptions and implementation
Robust covariance estimates require independent observations and enough information for asymptotic approximation. They do not repair an inconsistent coefficient caused by omitted variables or simultaneity.
Software labels include HC0, HC1, HC2, and HC3. These choices modify leverage and finite-sample corrections. Widening residual spread changes uncertainty around the same fitted line.
Interpreting robust results
The HC covariance estimator changes the estimated sampling uncertainty. It leaves $\hat\beta$, fitted values, and residuals unchanged. Report the coefficient in outcome units with its robust confidence interval. Also name the HC version because HC0, HC1, HC2, and HC3 use different finite-sample adjustments.
A difference between classical and robust standard errors reveals sensitivity to the constant-variance assumption. It does not measure the amount of heteroskedasticity directly. The coefficient still needs a correctly specified conditional mean or an appropriate linear-projection interpretation.
Calculation and model checks
Fit OLS once, then build the sandwich from the same design matrix and residual vector. HC1 multiplies the HC0 covariance by a degrees-of-freedom factor. HC2 and HC3 increase the influence of residuals at high-leverage observations. Verify software output with a direct matrix calculation for a small subset. Check that the intercept and any transformed regressors appear in both calculations.
Plot residuals against fitted values and key predictors. A fan shape supports robust variance estimation, while curvature points to a mean-function problem. Add transformations or interactions when subject knowledge supports them. Clustered or time-correlated errors require covariance estimators that preserve those dependence patterns. Ordinary HC methods assume independence across observations. In small samples with extreme leverage, HC3 often gives more stable protection, though no correction repairs sparse support (Long and Ervin, 2000).
Further reading
White gives the foundational heteroskedasticity-consistent covariance estimator (White, 1980). Long and Ervin compare common finite-sample variants (Long and Ervin, 2000).
Sources and further reading
- Halbert White. 1980. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica 48(4): 817–838. doi:10.2307/1912934.
- J. Scott Long, Laurie H. Ervin. 2000. “Using Heteroscedasticity Consistent Standard Errors in the Linear Regression Model.” The American Statistician 54(3): 217–224. doi:10.1080/00031305.2000.10474549.
About this benchmark task
- Method group
- Linear regression
- Software
- Stata, R, and Python
- Runs
- 360 runs, ten in each combination of model, software, prompt, and agency condition