We value your privacy

We use cookies to enhance your browsing experience, serve personalized content and analyze our traffic. By clicking “Accept all” you consent to our use of cookies. Cookie Policy

More options
Skip to content
FacthoryFacthory
FacthoryFacthory
    • Workspace
    • Agent
    • Knowledge
    • Analytics
    • Office apps
    • Desktop
    All features
    What’s new
    Download
  • By work

    • Organizational Knowledge Risk & Gap Intelligence
    • Scientific Research & Experimentation
    • Autonomous Analytics & Insight Discovery
    • Workforce Readiness & Adaptive Learning
    • SOP Governance & Operational Standardization
    • Process Discovery & Continuous Optimization
    • All solutions

    By industry

    • Manufacturing
    • Automotive
    • Construction & Infrastructure
    • Real Estate
    • Public Sector
    • Life Sciences
    • All industries
  • Facthory for

    • Regulated industries
    • On-premises
    • Developers

    Discover

    • What’s new
    • Blog
    • Case studies
    • Webinars
    • Models

    Learn

    • Documentation
    • Help centre
  • Pricing
  • Download
Sign inGet started
Back to modelsMethodology

How we score models

Every score is built from published evidence, each fact with its source and the date we read it. This page says what we collect, how it becomes a score, and where scores stop.

Principles

Published evidence only

We read what providers, hubs and benchmark authors publish. We never present a result we produced as theirs.

Every fact has a source

Each fact on a model page links to where it comes from and when we read it.

Machines collect, people review

Crawlers collect and models on Cloudflare Workers AI help structure and categorize; a person reviews every score before it carries the badge.

Scores move with the evidence

When a model, its documentation or a benchmark changes, the score is recomputed and dated.

Sources

  • Model hubs: model cards, files, checksums, and the hubs’ own malware and pickle scans
  • Providers’ documentation, technical reports, licences and copyright policies
  • Training-content summaries published under Article 53(1)(d)
  • The AI Office: designations of systemic-risk models, templates and Code of Practice signatories
  • Benchmark results published by their authors, providers and public leaderboards

The three scores

Each score is a set of checks. A check is rated from its evidence, and the score is the share of applicable checks met, out of 100, where partly met counts half.

How a check is rated

Met

The evidence is published and complete.


Partly met

Published, but incomplete or out of date.


Not met

The evidence shows the check fails, as with a file the hub rates unsafe. It counts as not evidenced does.


Not evidenced

Nothing published that we could find.


Not applicable

The obligation does not bind this model, as with Article 55 for a model without systemic risk; it counts neither way.

Safety and vulnerabilities

What could harm the systems that load the model, or the people who use it.

  • Safe weight formats for every file
  • Clean results from the hubs’ malware and pickle scans
  • Verifiable provenance: publisher, checksums, signatures
  • A licence that permits the use it is offered for
  • Published safety evaluations and red-teaming

EU AI Act

One check per obligation in Articles 50 to 56 and Annexes XI and XII that binds the model’s provider, read against published evidence.

  • Systemic-risk classification decides whether Article 55’s checks apply
  • The open-source exemption of Article 53(2) decides whether points (a) and (b) apply
  • Deployer duties (Articles 4, 26, 50) are listed beside the score, not counted in it

Capability

Per category, each published benchmark result relative to the best published result among the models we compare, out of 100.

  • Categories: coding, maths, reasoning, knowledge, conversation, languages
  • Only results the benchmark’s or model’s authors published, each with its version
  • Compute needs shown beside the score, not counted in it

The benchmarks the capability score counts

A result counts toward its category when the model’s authors publish it on the model card, for a benchmark below, and at least two models in the catalog report the same benchmark, configuration and metric. It scores its share of the best result among them; a category scores the mean of its results, and the capability score the mean of its categories.

Coding

  • HumanEval (Defined in arxiv.org/abs/2107.03374)
  • MBPP (Defined in arxiv.org/abs/2108.07732)
  • LiveCodeBench (Defined in arxiv.org/abs/2403.07974)
  • SWE-bench (Defined in arxiv.org/abs/2310.06770)

Maths

  • GSM8K (Defined in arxiv.org/abs/2110.14168)
  • MATH (Defined in arxiv.org/abs/2103.03874)

Reasoning

  • GPQA (Defined in arxiv.org/abs/2311.12022)
  • ARC (Defined in arxiv.org/abs/1803.05457)
  • HellaSwag (Defined in arxiv.org/abs/1905.07830)
  • WinoGrande (Defined in arxiv.org/abs/1907.10641)
  • BIG-Bench Hard (Defined in arxiv.org/abs/2210.09261)
  • DROP (Defined in arxiv.org/abs/1903.00161)

Knowledge

  • MMLU-Pro (Defined in arxiv.org/abs/2406.01574)
  • MMLU (Defined in arxiv.org/abs/2009.03300)
  • TriviaQA (Defined in arxiv.org/abs/1705.03551)

Conversation

  • IFEval (Defined in arxiv.org/abs/2311.07911)
  • MT-Bench (Defined in arxiv.org/abs/2306.05685)
  • AlpacaEval (Defined in arxiv.org/abs/2404.04475)
  • Arena-Hard (Defined in arxiv.org/abs/2406.11939)

Languages

  • MGSM (Defined in arxiv.org/abs/2210.03057)
  • Global-MMLU (Defined in arxiv.org/abs/2412.03304)
  • Results a card attributes to the Open LLM Leaderboard are the leaderboard’s runs, not the authors’, and do not count.
  • Epoch AI’s evaluations are shown apart, on the benchmark pages, and do not count: they are Epoch’s runs.
  • Error metrics, where lower is better, do not count.
  • Where one card writes 0.71 and another 71 for the same comparison, 0.71 is read as 71.

The Facthory Verified badge

A model carries the badge when all three scores clear the thresholds we publish with them, no critical finding is open, and a person has reviewed the evidence. It is withdrawn as soon as one of those stops being true.

Where scores stop

  • A score describes published evidence, not the model’s real behaviour in your systems.
  • A document we could not find is “not evidenced”, not a breach of the EU AI Act.
  • Scores are not legal advice. Whether a use is high-risk depends on the use, not on the model.
  • Benchmarks measure what they measure: test a model on your own tasks before you rely on it.

Corrections

A provider or author who sees a fact wrong or out of date can tell us, with the source. We check it and update the score with its date.

Send a correction

A workspace for what your organization knows, with an agent that works on it.

Product

  • All features
  • Workspace
  • Agent
  • Knowledge
  • Analytics
  • Office apps
  • Desktop
  • Download
  • Pricing

Solutions

  • Organizational Knowledge Risk & Gap Intelligence
  • Scientific Research & Experimentation
  • Autonomous Analytics & Insight Discovery
  • Workforce Readiness & Adaptive Learning
  • SOP Governance & Operational Standardization
  • Process Discovery & Continuous Optimization
  • All solutions
  • All industries

Resources

  • Blog
  • Case studies
  • Webinars
  • Documentation
  • Help centre
  • Models
  • ROI calculator

Company

  • Who we are
  • Careers
  • Press hub
  • Brand kit
  • Philosophy
  • Contact us

Join our newsletter

Product news from Facthory, by email.

​

© 2026 Facthory. All rights reserved.
  • Privacy policy
  • Cookie policy
  • Terms and conditions
  • Imprint
  • Cookie settings