How we score models
Every score is built from published evidence, each fact with its source and the date we read it. This page says what we collect, how it becomes a score, and where scores stop.
Principles
Published evidence only
We read what providers, hubs and benchmark authors publish. We never present a result we produced as theirs.
Every fact has a source
Each fact on a model page links to where it comes from and when we read it.
Machines collect, people review
Crawlers collect and models on Cloudflare Workers AI help structure and categorize; a person reviews every score before it carries the badge.
Scores move with the evidence
When a model, its documentation or a benchmark changes, the score is recomputed and dated.
Sources
- Model hubs: model cards, files, checksums, and the hubs’ own malware and pickle scans
- Providers’ documentation, technical reports, licences and copyright policies
- Training-content summaries published under Article 53(1)(d)
- The AI Office: designations of systemic-risk models, templates and Code of Practice signatories
- Benchmark results published by their authors, providers and public leaderboards
The three scores
Each score is a set of checks. A check is rated from its evidence, and the score is the share of applicable checks met, out of 100, where partly met counts half.
How a check is rated
Met
The evidence is published and complete.
Partly met
Published, but incomplete or out of date.
Not met
The evidence shows the check fails, as with a file the hub rates unsafe. It counts as not evidenced does.
Not evidenced
Nothing published that we could find.
Not applicable
The obligation does not bind this model, as with Article 55 for a model without systemic risk; it counts neither way.
Safety and vulnerabilities
What could harm the systems that load the model, or the people who use it.
- Safe weight formats for every file
- Clean results from the hubs’ malware and pickle scans
- Verifiable provenance: publisher, checksums, signatures
- A licence that permits the use it is offered for
- Published safety evaluations and red-teaming
EU AI Act
One check per obligation in Articles 50 to 56 and Annexes XI and XII that binds the model’s provider, read against published evidence.
- Systemic-risk classification decides whether Article 55’s checks apply
- The open-source exemption of Article 53(2) decides whether points (a) and (b) apply
- Deployer duties (Articles 4, 26, 50) are listed beside the score, not counted in it
Capability
Per category, each published benchmark result relative to the best published result among the models we compare, out of 100.
- Categories: coding, maths, reasoning, knowledge, conversation, languages
- Only results the benchmark’s or model’s authors published, each with its version
- Compute needs shown beside the score, not counted in it
The benchmarks the capability score counts
A result counts toward its category when the model’s authors publish it on the model card, for a benchmark below, and at least two models in the catalog report the same benchmark, configuration and metric. It scores its share of the best result among them; a category scores the mean of its results, and the capability score the mean of its categories.
Coding
- HumanEval (Defined in arxiv.org/abs/2107.03374)
- MBPP (Defined in arxiv.org/abs/2108.07732)
- LiveCodeBench (Defined in arxiv.org/abs/2403.07974)
- SWE-bench (Defined in arxiv.org/abs/2310.06770)
Maths
- GSM8K (Defined in arxiv.org/abs/2110.14168)
- MATH (Defined in arxiv.org/abs/2103.03874)
Reasoning
- GPQA (Defined in arxiv.org/abs/2311.12022)
- ARC (Defined in arxiv.org/abs/1803.05457)
- HellaSwag (Defined in arxiv.org/abs/1905.07830)
- WinoGrande (Defined in arxiv.org/abs/1907.10641)
- BIG-Bench Hard (Defined in arxiv.org/abs/2210.09261)
- DROP (Defined in arxiv.org/abs/1903.00161)
Knowledge
- MMLU-Pro (Defined in arxiv.org/abs/2406.01574)
- MMLU (Defined in arxiv.org/abs/2009.03300)
- TriviaQA (Defined in arxiv.org/abs/1705.03551)
Conversation
- IFEval (Defined in arxiv.org/abs/2311.07911)
- MT-Bench (Defined in arxiv.org/abs/2306.05685)
- AlpacaEval (Defined in arxiv.org/abs/2404.04475)
- Arena-Hard (Defined in arxiv.org/abs/2406.11939)
Languages
- MGSM (Defined in arxiv.org/abs/2210.03057)
- Global-MMLU (Defined in arxiv.org/abs/2412.03304)
- Results a card attributes to the Open LLM Leaderboard are the leaderboard’s runs, not the authors’, and do not count.
- Epoch AI’s evaluations are shown apart, on the benchmark pages, and do not count: they are Epoch’s runs.
- Error metrics, where lower is better, do not count.
- Where one card writes 0.71 and another 71 for the same comparison, 0.71 is read as 71.
The Facthory Verified badge
A model carries the badge when all three scores clear the thresholds we publish with them, no critical finding is open, and a person has reviewed the evidence. It is withdrawn as soon as one of those stops being true.
Where scores stop
- A score describes published evidence, not the model’s real behaviour in your systems.
- A document we could not find is “not evidenced”, not a breach of the EU AI Act.
- Scores are not legal advice. Whether a use is high-risk depends on the use, not on the model.
- Benchmarks measure what they measure: test a model on your own tasks before you rely on it.
Corrections
A provider or author who sees a fact wrong or out of date can tell us, with the source. We check it and update the score with its date.
