# Runtime security metrics: false positives and blocked rate

URL: https://airuntimesecuritytools.com/guides/runtime-security-metrics-false-positives-and-blocked-rate/
Updated: 2026-11-16

Measure a runtime control with five numbers: blocked rate, false positive rate, miss rate from testing, decision latency and coverage by action type. No standard benchmark exists, and vendors publish few figures. Sources read on 16 Nov 2026 include Google and Straiker documentation.

## How do you measure a runtime control?

Measure outcomes by action type, not one overall score. Track how many actions of each type were allowed, blocked and asked, how long each decision took, and how often a person overrode a block.

Coverage is a metric too: the share of agents, apps and machines that have the control on. A strong policy on half your estate is half a control.

## What is a good false positive rate?

There is no published standard figure, so set a budget from the cost of a wrong block. A customer chat that blocks one real question in a hundred may be acceptable, and a build pipeline that blocks one command in a hundred may not.

Google's Model Armor documentation says false positives can degrade the user experience by blocking legitimate prompts and responses, and that stricter confidence settings raise that risk. Straiker says Defend AI achieves 98.1 percent detection accuracy at under 300 ms, a vendor claim with no stated method.

Sources: [Model Armor overview, Google Cloud](https://cloud.google.com/security-command-center/docs/model-armor-overview); [Straiker Defend AI](https://www.straiker.ai/products/defend-ai).

## How do you find misses?

Misses show up only through testing and incident review. Run a fixed set of safe test cases for each action type and record what got through, and review allowed actions near the policy edge.

MITRE ATLAS and OWASP's list give categories to build test sets from. Keep test cases harmless and out of production data.

Sources: [MITRE ATLAS](https://atlas.mitre.org/); [OWASP Top 10 for LLM Applications 2025](https://genai.owasp.org/llm-top-10/).

## How do audit and inspect-only modes help?

Run new policies in a mode that logs without blocking, then compare what would have been blocked with what staff were doing. Model Armor's inspect-only mode exists for this: it logs violations without stopping the request.

Move to blocking one action type at a time, and keep the log to show the change in blocked rate.

Sources: [Model Armor overview, Google Cloud](https://cloud.google.com/security-command-center/docs/model-armor-overview).

## How do you report to leadership?

Report a short table: actions stopped by type, estimated false positives, decision latency at p95 and coverage. Add the open gaps in plain words.

Avoid a single detection accuracy number unless the vendor publishes the test set and method, since without them it cannot be checked.

## What does blocked rate tell you?

Blocked rate alone is ambiguous. A rising rate can mean more attacks, a policy that got stricter or a new app with odd traffic. Pair it with the reason for each block and a sample review.

A falling rate after a policy change should prompt a check for misses, not celebration.

## How often should you review the numbers?

Review weekly during rollout and monthly after. Look at the top blocked rules, the rules with the most overrides and the actions with no decisions at all, which can mean a gap in coverage.

Change one rule at a time and keep the change log with the metric before and after.

## What should you ask vendors to publish?

Ask for the test set, the method, the false positive rate on benign traffic and the date of the test. A detection figure with no benign-traffic rate is only half a measurement.

This site scores latency and failure mode only as published, and the same standard applies to accuracy claims: no method, no credit.

## Sources

- [Model Armor overview, Google Cloud](https://cloud.google.com/security-command-center/docs/model-armor-overview) (cloud.google.com), read 2026-11-16
- [Straiker Defend AI](https://www.straiker.ai/products/defend-ai) (straiker.ai), read 2026-11-16
- [MITRE ATLAS](https://atlas.mitre.org/) (atlas.mitre.org), read 2026-11-16
- [OWASP Top 10 for LLM Applications 2025](https://genai.owasp.org/llm-top-10/) (genai.owasp.org), read 2026-11-16

## Related pages

- [What to Log From Runtime Security Decisions](/guides/what-to-log-from-runtime-security-decisions/)
- [False Positive Budget (glossary)](/glossary/false-positive-budget/)
- [Block vs Detect (glossary)](/glossary/block-vs-detect/)
- [All guides](/guides/)
- [Runtime Security Latency Budgets for LLM Apps](/guides/runtime-security-latency-budgets-for-llm-apps/)
- [The ranking](/ranking/)
- [Glossary](/glossary/)
