METR says Anthropic’s benchmark scores are rising nearly twice as fast. Public indexes show 1.3 to 1.5 times as fast.
Anthropic is speeding up on public benchmarks, and OpenAI isn’t.
Kevin Hu Founder
Subscribe
On November 5, METR, an independent AI evaluator, reported that benchmark scores for Anthropic’s models have risen 1.9 times as fast since October 2025 as before. It estimated, with substantial uncertainty, that AI explains most of the increase. The 1.9 figure uses AECI, Anthropic’s internal version of Epoch AI’s Capabilities Index. AECI draws mostly on benchmarks that aren’t public and includes models Anthropic hasn’t released.
Village’s LLM Atlas scores models from public benchmarks alone, on the same scale as AECI, where Claude 3.5 Sonnet is 130 and GPT-5 is 150. It combines 773 benchmarks, more than 13 times as many as Epoch’s index, accounts for how hard each benchmark is and how much reasoning a model was given, gives every score a range of uncertainty, and links each result to where it was published. We applied METR’s method to two public indexes: Atlas and Epoch’s index. Like METR, we define a lab’s speed-up as how fast its best scores have risen since October 7, 2025, divided by how fast they rose before.
Public indexes can’t see unreleased models, so they measure something narrower than METR does, and METR’s own range is wide: it “would not be surprised if the current rate of growth was closer to 1.3x or 3x the historical rate.” Atlas puts Anthropic’s speed-up at 1.3 times; Epoch’s index gives 1.5. OpenAI’s scores have climbed no faster since October 2025 on either index, so the speed-up that public benchmarks do show belongs to Anthropic.
Unreleased models explain only about a third of the gap between METR’s estimate and Atlas’s
- Even on released models, AECI shows 1.71 times. METR’s own comparison gives that figure after leaving out Anthropic’s unreleased models and dating the rest by public release rather than by when Anthropic staff could use them.
- Public benchmarks came close through the same method. Epoch’s index, whose public code AECI uses, gave 1.59 times on the same released models on October 30.
- The statistical method matters as much as the benchmarks. Epoch moved its index to a Bayesian model on November 4 and added newer data, and its speed-up fell to 1.45 times, or 1.40 on METR’s list of models.
- Atlas is lower still, at 1.28 times. It scores released models on far more benchmarks, with a different statistical model.
AECI’s speed-up on released models rose from 1.4 to 1.7 times when Anthropic changed its benchmarks in September
- In early September, AECI looked much like the public indexes. On the same 11 released models, scores in Anthropic’s Fable 5.1 card give 1.39 times by our calculation, between Atlas’s 1.24 and Epoch’s 1.41. That card called Claude Mythos Preview’s jump “a one-time event that shifted the entire trend line upward, rather than a permanent accelerant of the pace of future progress.” METR now says recent models make that view “look less likely than a permanent trend change.”
- Anthropic’s September update took AECI to 1.66 times. Anthropic added benchmarks, including “several new hard internal evaluations,” and recalculated the index, raising its newest models by 4.1 to 6.1 points. Anthropic says it made the changes “to improve coverage at the top and middle of the scale, and to correct known errors in the inputs,” and we can’t say which version is closer to the truth. Even on the new version, its card found a one-time jump a better fit than a lasting change of pace in 99 of 100 resampled fits, and said “under either reading the slope has not doubled.”
AECI is also thinly linked to other labs’ models. Its scores for Anthropic’s models rest mostly on Anthropic’s own benchmarks, and Anthropic calls the link between those and other labs’ scores “sparse,” “relying on a small number of overlapping evaluations and models,” which is why it says AECI isn’t directly comparable to Epoch’s index. Which models set the scale can move the speed-up either way: in METR’s comparison, setting it on Anthropic’s models alone takes AECI from 1.71 to 1.41 times and Epoch’s index from 1.59 to 1.84.
Anthropic recalculated AECI again in October. Claude Opus 5.5 scored 169.12 in its Sonnet 5.5 card on September 28 and 174.56 in its Haiku 5.5 card on October 7, and Anthropic says scores from different AECI versions aren’t comparable. That recalculation changed the scores more than the speed-up, which went only from 1.66 to 1.72 times on the same 11 models. AECI’s benchmarks are hidden, so we can’t tell which ones produce its steeper rise. On Atlas, dropping the hardest benchmarks barely changes the speed-up.
Anthropic’s scores used to rise more slowly than OpenAI’s, and now they rise faster
- The switch shows on both public indexes. On Atlas, Anthropic went from about 16 points a year before October 2025 to 21 since, and OpenAI from about 18 to 16. On Epoch’s index, Anthropic went from about 14 to 21, and OpenAI from about 18 to 16.
- The faster rise closed Anthropic’s gap but hasn’t opened a lead. On Atlas, Anthropic averaged 4.3 points behind OpenAI in the year to September 2025, drew level with Claude Opus 4.5 in November 2025, and has gained 15.6 points since, to OpenAI’s 17.0.
Counted from METR’s March 2024 start, public benchmarks back part of METR’s finding. They can’t test its unreleased models, but they show a speed-up at Anthropic that OpenAI doesn’t share.
How we built this
We used Atlas’s version of November 6, 2026, and Epoch’s index as published on November 5. For each lab we took the running record of its best-scoring models since March 2024 by public release date, then fit one line whose slope changes on October 7, 2025, the break METR estimated, weighting each record equally. The figures of 1.90, 1.74, 1.71, 1.59, 1.41 and 1.84 come from METR’s comparison chart, which dates its 1.90 and 1.74 by when models reached Anthropic staff. We reproduced its 1.59, 1.65 and 1.73 from published scores on METR’s list of 12 Anthropic models. To compare AECI’s versions, we use that list without Opus 5.5, which the September 1 version doesn’t include. We use each index’s own running record instead of that list so that one rule covers both labs. On METR’s list, Atlas gives 1.23 and Epoch’s index 1.40. AECI scores for individual models come from Anthropic’s system cards, read from their charts where they aren’t printed.
Atlas is Village’s own index. None of this tests METR’s estimate that AI explains most of the speed-up. Atlas’s range reflects uncertainty in each model’s score and in which models set records, while METR’s reflects which benchmarks are sampled, so the two aren’t strictly alike. We count Mythos Preview as an Anthropic record although it wasn’t generally released; leaving it out gives 1.27. Atlas scores Fable 5 and 5.1 where AECI scores Mythos 5 and 5.1; Anthropic says Fable 5 has the same model weights as Mythos 5, with added safeguards. GPT-6 Astra Pro, OpenAI’s best as of November 6, rests on six benchmarks. OpenAI’s speed-up rises if the count starts later: starting in March 2025, after its o1 models, gives 1.42 on Atlas and 1.35 on Epoch’s index, against Anthropic’s 1.28 and 1.34. Atlas’s estimate for Anthropic held at 1.28 across its three latest versions, and between 1.13 and 1.46 under single changes to its method or data.
