Every fact in the catalogue carries one of three evidence levels: documented, vendor-stated or not documented. The level tells you what kind of page backs the value, or that we found no page at all. This guide explains each level and how to apply the same check to any AI testing tool.
The three levels
- Documented: the value appears on a documentation-class page. The fact links to that exact page.
- Vendor-stated: the vendor says it only on a marketing-class page. The fact links to that page.
- Not documented: we looked on the public pages and did not find the value. The fact has no link and no confirmed value.
Not documented does not mean "no". It means we could not confirm the value either way. A feature is recorded as absent only when the docs say so. In that case the fact is documented, with the docs page as its source.
We never upgrade a level. If a docs page cannot be opened, the fact stays vendor-stated or not documented.
What counts as documentation
These page types can support a documented fact:
- product docs, help centres, knowledge bases and API references
- changelogs and release notes
- pricing pages, for prices, plans and free tiers
- terms of service, data processing terms and licence files
- public source repositories: README, LICENSE and code
- official marketplace listings, for integrations
These page types support vendor-stated only:
- homepages, feature pages and landing pages
- vendor-written comparison pages
- case studies, ROI calculators and webinars
- vendor blog posts about capabilities or outcomes
Coverage percentages, time savings and speed figures always stay in the vendor claims block on the tool page. That rule holds even when the docs repeat the figure.
Examples from the data
TestRail records a founding year of 2004 as vendor-stated. The source is the TestRail About page, which is a company page, not documentation.
Checkly records prompt-to-test creation as vendor-stated. The source is the Checkly homepage.
Checkly's output value, standard Playwright, is documented. The source is the Playwright Check Suite docs page.
CloudQA records its AI capability list as vendor-stated. The source is the CloudQA homepage.
CloudQA's self-healing fact is documented. The CloudQA release notes record an auto-heal feature enabled by default in October 2025.
mabl records its free tier as not documented. The mabl pricing page prints a 14-day free trial, and a trial is not a free tier.
How the levels are distributed
Counts as of 2026-09-24, across 330 tools:
| Fact | Documented | Vendor-stated | Not documented |
|---|---|---|---|
| Pricing model | 277 | 26 | 27 |
| Hosting | 275 | 37 | 18 |
| Output | 156 | 16 | 158 |
Pricing model is documented for 277 of 330 tools. Output is not documented for 158 tools, close to half the catalogue. For those tools, we found no public page stating whether tests can run outside the vendor's product.
The "verify before relying" flag
Every fact shows the date we opened its source, labelled verified on. When that date is more than 90 days before the build date, the tool page marks the fact "verify before relying".
Pricing pages change often. The monthly update re-verifies pricing and status for every tool. Other facts are re-checked when their source changes or on the next full pass.
A flagged fact is still sourced. The flag means the linked page may have changed since we opened it.
Applying the check yourself
- Find the statement on the vendor's site and note which page type it sits on.
- Search the docs, help centre, changelog or pricing page for the same statement.
- If a documentation-class page states it, note the URL and the date you opened it.
- If only a marketing-class page states it, treat it as vendor-stated.
- If no public page states it, treat it as not documented rather than guessing.
The full rules, including cadence and inclusion, are on the methodology page.