Classical tests
Welch t-test
Welch's test compares two means without imposing equal population variances.
The task
Task statement
Compare y between group 1 and group 0 using Welch's unequal-variance two-sample t-test. Define the t statistic as mean(y | group=1) minus mean(y | group=0), divided by the Welch standard error. Use the Welch-Satterthwaite approximation for the degrees of freedom. Report rows t, df, and p_value.
Data
Columns: group, y.
- Files
- input.csv
Required output
Scored estimates: t, df, p_value.
Reference output
| Term | Reference estimate | Sample size |
|---|---|---|
t | 2.9850733 | 200 |
df | 163.0833 | 200 |
p_value | 0.0032718561 | 200 |
A run succeeds when every scored estimate lies within 1 percent or 0.005 of its reference value, whichever is larger. Sample sizes are not scored.
Task success
| Agency condition | Claude Sonnet 4.6 | GPT-5.6 Luna |
|---|---|---|
| Chatbot | 98.3% | 75.0% |
| Self-repair | 96.7% | 100.0% |
| Constrained agent | 100.0% | 100.0% |
- All runs
- 95.0% of 360 runs
- By prompt condition
- Zero-shot 95.0%, One-shot 95.0%
- By software
- Python 100.0%, R 100.0%, Stata 85.0%
A mean difference with separate variance estimates
For independent groups, Welch's statistic is (Welch, 1947):
Its degrees of freedom use the Welch-Satterthwaite approximation.
A calculation
| Group | Mean | Variance | Size |
|---|---|---|---|
| 1 | 10 | 4 | 20 |
| 2 | 8 | 9 | 15 |
The standard error is $\sqrt{4/20+9/15}=0.894$. The statistic is $2/0.894=2.236$. The approximate degrees of freedom are 23.0.
Assumptions and implementation
The groups must be independent. Each group mean needs enough observations for the approximation. Strong skewness or heavy tails can motivate randomization or robust alternatives.
Software can default to the pooled equal-variance test. Side-by-side distributions reveal each mean, spread, and standard-error contribution.
Interpretation and reporting
The observed mean difference is $10-8=2$. Welch's denominator combines the two variance contributions without pooling them. The resulting statistic is $2.236$, with about $23.0$ reference degrees of freedom.
State the subtraction order so the sign has a clear meaning. Report both group means, standard deviations, sample sizes, the mean difference, and its confidence interval. Keep the fractional degrees of freedom because rounding it to a group size changes the approximation.
Welch's method handles unequal variances. It does not protect a mean comparison from severe outliers or dependence between observations. Paired data require a paired analysis, while clustered samples require covariance that reflects their grouping.
Define the sampling unit before applying the test. Repeated measurements from one person or firm are dependent. Analyze within-unit differences or use a model that represents that pairing. Welch's variance adjustment cannot recover covariance that the calculation omitted.
Reproducible implementation
Define the two independent groups and remove missing outcomes within each group. Compute means and unbiased sample variances. Insert these values into the standard-error and degrees-of-freedom formulas.
Compare the manual statistic with a software call that explicitly disables equal-variance pooling. Confirm the group order used by the command. Reversing it should change the statistic's sign and preserve its magnitude.
Plot each group distribution and inspect extreme values. For small, highly skewed samples, add a justified randomization or robust-location analysis. Save the group coding and complete-case counts with the result (Welch, 1947).
Further reading
Welch derives the unequal-variance comparison (Welch, 1947). Satterthwaite supplies the degrees-of-freedom approximation (Satterthwaite, 1946).
Sources and further reading
- Bernard L. Welch. 1947. “The Generalization of Student's Problem When Several Different Population Variances Are Involved.” Biometrika 34(1/2): 28–35. doi:10.2307/2332510.
- Franklin E. Satterthwaite. 1946. “An Approximate Distribution of Estimates of Variance Components.” Biometrics Bulletin 2(6): 110–114. doi:10.2307/3002019.
About this benchmark task
- Method group
- Classical tests
- Software
- Stata, R, and Python
- Runs
- 360 runs, ten in each combination of model, software, prompt, and agency condition