Worth knowing
Updating the Epoch Capabilities Index methodology to use a Bayesian model
This methodological upgrade improves the statistical rigor of a prominent index used to aggregate benchmark scores and track the progress of frontier AI models.
Reported by Epoch AI ·
What happened
Epoch AI has updated the methodology behind its Epoch Capabilities Index (ECI), transitioning from a frequentist approach to a Bayesian Item Response Theory model. This update replaces bootstrapping with Markov Chain Monte Carlo sampling and introduces a beta likelihood with benchmark-specific noise to provide more reliable uncertainty estimates and direct model comparisons.
Who it’s about
- Epoch AI
- Claude 3.5 Sonnet
- GPT-5