# How we score AI runtime security tools

URL: https://airuntimesecuritytools.com/method/
Updated: 2026-11-16

One research plan covered all 16 vendors on 16 Nov 2026 and produced 1,469 facts across 32 vendor files, each checked against a quote. One scorer scored blind, a second blind reviewer checked, and weights were committed before any evidence or score existed.

## The weights

The weights were fixed on 2026-11-16, before any score existed, and have not changed since.

**Criteria and weights (out of 100)**

| Criterion | What it measures | Weight |
|---|---|---|
| Action types stopped before they complete | Which harmful actions it stops before they complete, named: shell command, file read or write, data paste or upload, package install, network call, credential access, unsafe tool call, prompt injection or harmful model output | 30 |
| Layers covered | Which layers it enforces at, as documented: model (prompts and outputs), application (an AI app's requests), agent (tool calls and plans), endpoint (actions on the machine) | 20 |
| Latency and failure mode | Published latency of an inline decision and the published behavior when the control fails (fail open or fail closed, timeouts) | 15 |
| Deployment | How it deploys (API, SDK, proxy or gateway, agent, agentless), what it needs, time to deploy as published | 15 |
| Detection and response | Alerts, an audit trail of blocked and allowed actions, investigation, and export to SIEM or SOAR | 5 |
| Transparency | Public documentation readable without sign-in, a published price, published research | 10 |
| Proof | Named customers, funding, independent analyst coverage | 5 |

## Weights by situation

**Weights used for each situation view**

| Situation | Action types stopped before they complete | Layers covered | Latency and failure mode | Deployment | Detection and response | Transparency | Proof |
|---|---|---|---|---|---|---|---|
| [Developers run coding agents with shell, file and package access](/best-for/coding-agents/) | 35 | 20 | 10 | 15 | 10 | 5 | 5 |
| [A company shipping a customer-facing LLM application](/best-for/customer-llm-app/) | 25 | 20 | 25 | 15 | 5 | 5 | 5 |
| [Staff use agents inside SaaS and cloud workspaces that act on company data](/best-for/saas-agents/) | 30 | 30 | 5 | 15 | 10 | 5 | 5 |
| [A regulated company that must prove what was blocked and respond to incidents](/best-for/regulated-response/) | 25 | 15 | 10 | 10 | 30 | 5 | 5 |
| [A lean team that must have runtime control in place this quarter](/best-for/fast-rollout/) | 25 | 15 | 10 | 35 | 5 | 5 | 5 |

## Score anchors

- **0**: none found on the read date
- **1-3**: mentioned, partial, or limited to one surface, one layer or one action type
- **4-6**: documented and working, with gaps the documentation shows
- **7**: the most a vendor claim or vendor documentation alone can earn: detailed, specific, covering the whole criterion
- **8-10**: detailed and confirmed by a second source (third-party write-up, or observed directly in public documentation, code, a marketplace listing or a release record)

## Rules

- Every vendor is scored 0 to 10 on every criterion, whole numbers, from the evidence file only, by a scorer that works from coded packets with every vendor name removed.
- A vendor claim scores at most 7 until a second source confirms it (kind third-party or observed).
- None found scores 0 and is written "none found on the read date".
- Breadth facts (module counts, integration counts, headcount, funding) inform Proof only, which weighs 5, so they carry at most 10 percent of any ranking.
- The total is the weighted sum of the scores divided by 10 (0 to 100 scale, published to one decimal).
- Equal totals print as a joint rank (for example joint 2nd); inside a joint rank vendors are listed alphabetically for display only.
- Once the first score exists, no weight and no criterion changes. A score changes only when a sourced fact changes, logged with the fact.

## What research was done?

A single research plan was applied to every vendor on 16 Nov 2026. The same questions were asked of each: which actions are stopped, at which layers, with what published latency and failure mode, how it deploys, what it records, what is public and who backs it.

The result is 1,469 facts across 32 vendor files, because the files also serve other comparisons; the 16 files for this site hold 896 of them. Each fact holds a claim, a quote, a source URL, a read date and a kind: vendor, third-party, observed or none-found. Every quote was checked against the page it came from.

## How does blind scoring work?

One scorer scored every vendor from the evidence file only, working from coded packets with every vendor name removed, so the scorer did not know which vendor was which. One blind reviewer then checked each score against its cited facts. Disagreements were settled by returning to the quote.

Scores are whole numbers from 0 to 10. A score changes only when a sourced fact changes, and the change is logged with the fact.

## Why are weights fixed first?

The weights, the five situation weightings and the scoring anchors were committed on 16 Nov 2026 before any vendor evidence or score existed. After the first score, no weight and no criterion changes.

Breadth counts such as module numbers, integrations, headcount and funding inform only the proof criterion, which weighs 5, so they carry at most 10 percent of any ranking.

## How are latency and failure mode scored?

Only as published. A millisecond figure on a vendor page, a documented timeout or a stated fail-open or fail-closed default earns credit. A figure seen only in a demo panel is treated as a mention. Silence scores 0 and is written as none found on 16 Nov 2026.

This favors vendors that document their control, which is the point: a buyer cannot plan around a failure mode nobody has written down.

## How are errors corrected?

Send a correction with a source to editor@airuntimesecuritytools.com. A confirmed fact change updates the fact, the score and the changelog on the same day, and the old and new values stay visible.

Vendors may supply public documentation at any time. A claim without a public source stays at the claim ceiling of 7.

## Changelog

- 2026-11-16: weights and criteria fixed.
- 2026-11-16: evidence read for every vendor by one research plan.
- 2026-11-16: scores published (blind: one scorer, then one blind reviewer).
- 2026-11-16: this site built from the committed scores.

## Related pages

- [The ranking these weights produce](/ranking/)
- [Vendor profiles with evidence per score](/vendors/)
- [Editorial policy](/editorial-policy/)
- [Corrections](/corrections/)
- [Glossary](/glossary/)
- [Runtime Action Type Matrix](/runtime-action-type-matrix/)
- [Enforcement Point Map](/enforcement-point-map/)
