Foresite Catalyst
MXDR for Google Cloud
Why agentic SOC metrics are measuring the wrong thing
The agentic SOC market has settled on a scoreboard. Alerts handled autonomously. Investigations closed without a human. Tokens spent proving something was safe. These numbers look impressive on a slide, but they say very little about whether your organization is more secure.
Volume is not an outcome
If your organization isn’t seeing twice the malicious activity, you shouldn’t see twice the alerts.
When a new AI platform lights up your environment like a Christmas tree, the right response is a question, not a celebration: is there actually malicious activity happening here, or did we just turn up the noise? Treating alert volume as proof of value hides the state of your environment from the people responsible for protecting it.
Every CISO knows the needle-in-a-haystack problem. Many agentic SOC vendors haven’t solved it. They’ve delivered ten more haystacks and an AI that sorts hay faster.
The fundamentals haven’t changed
Observability, detection, and response still work best as a continuous cycle of refinement. Each pass should answer three questions: what did we miss, what fired that shouldn’t have, and what can we fix at the source so it never fires again?
Logical tuning, estate visibility, and automated remediation are platform problems. Your vendor’s AI will not solve them for you. Somewhere along the way, the industry stopped valuing platform mastery and deep subject expertise, and started saying “an agent can do that” instead.
Context is king in security. But an agent built on a poorly managed platform is working with incomplete logs, misparsed fields, stale asset data, and untuned detections. That’s garbage in, garbage out, now at machine speed and delivered with confidence.
Paying AI prices for decade-old automation
A surprising amount of what’s marketed as agentic value is work that playbooks and basic integrations have handled reliably for more than a decade. If a vendor needs an AI agent to look up the reputation of an artifact in an alert, they are burning tokens at your expense.
Sold as AI, solved by automation
| Task being sold as “AI” | What it actually is |
|---|---|
| IP, domain, URL, and file hash reputation | A threat intel API call |
| GeoIP and ASN lookup | A deterministic enrichment |
| WHOIS and domain age | A single query |
| User context: department, manager, account status | A directory lookup |
| Asset owner, criticality, and business unit | A CMDB or asset inventory join |
| Is this host vulnerable to the exploit in the alert? | A vulnerability scanner query |
| Known scanner, pentest, or allowlisted source | A reference list match |
| Duplicate and related alert grouping | Correlation logic |
| Email header parsing and attachment detonation | A sandbox integration |
| Blocking a confirmed-bad indicator | A playbook action |
| Isolating a host or disabling an account | An EDR or IdP API call |
| Ticket creation and routing | Workflow automation |
None of these require reasoning. They require well-built integrations and well-managed platforms. They should be fast, cheap, consistent, and already complete before an AI ever sees the alert.
How Foresite measures AI
AI can be excellent or woefully unequipped depending on the circumstances around an alert. That truth rarely comes up in agentic SOC conversations, and it’s the one that matters most.
Our principal AI metric: disagreement
How often our AI and our analysts reach different conclusions.
Agreement shows where AI is earning trust. Disagreement shows where to look.
Every disagreement is a signal pointing at a root cause we can fix.
When AI and analysts disagree
| Likely cause | What we fix |
|---|---|
| Missing context | Telemetry coverage or enrichment gaps |
| A noisy or poorly scoped detection | Detection logic and tuning |
| Bad or inconsistent data | Parsing, normalization, and data integrity |
| Agent reasoning | Agent instructions and guardrails |
We track disagreement by detection, by alert type, and by client environment. That gives us a living map of where AI accelerates our analysts and where it still needs a human in the lead.
Agentic triage as an accelerant, not a replacement
Most vendors validate their AI by spot checking it. We don’t. Agentic triage sits inside our detection pipeline as an accelerant, not as a separate layer bolted on top. The order matters:
- Platform excellence first. Clean data, healthy telemetry, and high-fidelity detections across Google SecOps, Wiz, Tenable, Tanium, EDR, and firewall management.
- Deterministic automation second. Every enrichment and response action that doesn’t need reasoning is handled by proven automation.
- Agentic triage third. AI reasons over complete, trusted context and recommends a verdict.
- Analyst validation and feedback. Every disagreement feeds back into tuning, detections, data quality, or the agent itself.
Agentic triage sits third, after platform excellence and deterministic automation, and every disagreement with an analyst feeds back into the pipeline.
That loop is the lifecycle of refinement security has always needed, with AI making each turn faster. Catalyst brings it together, giving clients a single view across their estate and full transparency into how every alert was handled.
The better question
The next time a vendor tells you how many alerts their AI handled, ask two questions instead. Why were there that many in the first place? And how often does your AI disagree with your analysts?
If they can’t answer the second one, they aren’t measuring their AI. They’re marketing it.
See it for yourself. Catalyst gives you a single view across your estate and full transparency into how every alert was handled. Book a demo.
Frequently asked questions
Why is alert volume a poor measure of an agentic SOC?
Alert volume shows activity, not outcomes. If your organization isn’t seeing more malicious activity, it shouldn’t see more alerts. A spike after deploying an AI platform should prompt a question: is there real malicious activity, or did the platform just turn up the noise?
Which SOC tasks don’t need an AI agent?
Any enrichment or response step that needs no reasoning: reputation and GeoIP lookups, WHOIS, directory and asset lookups, vulnerability checks, alert grouping, sandboxing, blocking indicators, isolating hosts and routing tickets. Integrations and playbooks handle these faster and cheaper, before AI ever sees the alert.
What is an AI disagreement rate?
It is how often an AI and human analysts reach different conclusions on the same alert. Agreement shows where AI is earning trust. Each disagreement points to a fixable root cause: missing context, a noisy detection, bad data or the agent’s reasoning.
How does Foresite use agentic triage?
As an accelerant inside the detection pipeline, not a replacement for analysts. Platform excellence comes first, then deterministic automation, then agentic triage that reasons over trusted context and recommends a verdict. Analysts validate, and every disagreement feeds back into tuning, detections, data quality or the agent.
What should I ask an agentic SOC vendor?
Ask why there were so many alerts in the first place, and how often their AI disagrees with their analysts. A vendor that can’t answer the second question isn’t measuring its AI.
