Survival analysis
Discrete-time conditional logit
Exact conditional odds for monthly employment exits.
The task
Task statement
Workers belong to cohorts. Each cohort-month forms its own risk set. All workers enter before month one. Cohorts have independent baseline exit odds. training is a fixed binary covariate. end_month records either employment exit or right censoring. event=1 indicates employment exit. At each month, include workers whose end_month is at least that month. Evaluate all exits jointly before removing same-month censored workers. Remove both groups before the next month. No worker reenters.
Estimate Cox's discrete-time conditional odds model. Its coefficient beta multiplies training. For a risk set R with d observed exits and observed training sum S, the conditional log likelihood is beta*S-log(sum_A exp(beta*sum_{j in A} training_j)). The sum covers all distinct size-d subsets of R. Add contributions across cohorts and months with at least one exit. Use exact enumeration or an elementary symmetric polynomial recursion. Maximize over beta in [-4,4].
Return beta, exp(beta) as the conditional odds ratio, and the exact log likelihood without additional constants. At the fitted coefficient, return the conditional probability, given its exit count, that exactly the observed set of workers exits in cohort 0, month 1. This is the probability of those named workers, without multiplying by the number of sets with the same training sum. Also return its natural logarithm. The odds ratio is a conditional discrete-time quantity. For every result, n_obs is the number of input worker rows.Data
input.csv: cohort = Independent cohort-specific risk set identifier; worker = Worker identifier within the cohort; training = Fixed binary training covariate; end_month = Month of employment exit or censoring; event = One for an exit; zero for right censoring
- Files
- input.csv
Required output
Scored estimates: training_coefficient, conditional_odds_ratio, log_likelihood, first_event_set_probability, first_event_set_log_probability.
Reference output
| Term | Reference estimate | Sample size |
|---|---|---|
training_coefficient | 1.3451113 | 144 |
conditional_odds_ratio | 3.8386137 | 144 |
log_likelihood | -144.52236 | 144 |
first_event_set_probability | 0.00038003953 | 144 |
first_event_set_log_probability | -7.8752353 | 144 |
A run succeeds when every scored estimate lies within 1 percent or 0.005 of its reference value, whichever is larger. Sample sizes are not scored.
Task success
| Agency condition | Claude Sonnet 4.6 | GPT-5.6 Luna |
|---|---|---|
| Chatbot | 85.0% | 51.7% |
| Self-repair | 100.0% | 73.3% |
| Constrained agent | 100.0% | 88.3% |
- All runs
- 83.1% of 360 runs
- By prompt condition
- Zero-shot 78.3%, One-shot 87.8%
- By software
- Python 93.3%, R 93.3%, Stata 62.5%
Sources
Sources
About this benchmark task
- Method group
- Survival analysis
- Software
- Stata, R, and Python
- Runs
- 360 runs, ten in each combination of model, software, prompt, and agency condition