Evaluating AI Agent Vendors for Enterprise Security

Vendor evaluation for AI agents needs a different rulebook than the one procurement teams have used for twenty years. Traditional software buying runs on features, uptime guarantees, and reference calls, all of which assume the product does what you told it, when you told it, and nothing else. Agents don't hold up their end of that bargain: they reason through problems, call tools you didn't know existed, delegate to other agents, and remember things across sessions, often with no human anywhere near the decision. Security teams still grading agent vendors on the old checklist are, frankly, grading the wrong exam.

Take a UK insurer I heard about through a colleague who audited the aftermath. A platform scores highest on capability and price, the contract gets signed, and months later an internal audit goes looking for the reasoning trail behind a customer-facing policy recommendation and found the agent's logs couldn't show which version of the policy wording it had pulled before giving advice. The model probably performed fine. The logs just never captured what it did, and by the time anyone asked, there was nothing to reconstruct. That's not a model failure. That's a procurement failure, because nobody put "can you prove what happened" on the shortlist. This piece walks through the criteria that belong there, in the order they should show up in a real evaluation: scale, tooling gaps, access control, runtime monitoring, audit logs, and what a vendor's plugin ecosystem can do to your environment.

How fast agents are spreading inside enterprises, and why governance hasn't kept up

Start with scale, because most security teams are lowballing it. Gravitee's State of AI Agent Security 2026 survey polled more than 900 executives and practitioners and found nearly 38% of organizations already run more than 100 agents. By the end of 2026, 40% of enterprise applications are expected to ship with task-specific agents built in, up from under 5% in 2025. Over 81% plan to add more agents in the next year. This isn't a plateau, it's early innings, and everyone's still figuring out the rules mid-game.

Deployment is outrunning approval, and the gap is wider than most people assume. In that same survey, 80.9% of technical teams are past planning and into active testing or production, but only 14.4% say every agent they've deployed went live with full security or IT sign-off. Shadow AI compounds it: over 59% of security leaders confirm or suspect employees are using AI tools nobody in IT ever approved. Ask five people at a given company how many agents are running and you'll get five different numbers. None of them will be right, and I'd bet money on that.

None of this is one rogue team going around policy. It's structural. 81% of respondents feel pressure to ship agents fast even when governance isn't ready; 25.8% call that pressure significant. And the coverage gap shows it: mean monitoring coverage sits at 52%, meaning roughly half of all agents running in production today have no security eyes on them. Only 9.5% of organizations monitor more than 81% of their deployed agents. So walk into a vendor evaluation with that picture in mind. You're not evaluating for a clean slate. You're evaluating for an environment that already has agents running without an inventory, without a policy, and without anyone watching.

Why existing security tools weren't built for agents and can't be stretched to cover them

A web application firewall has no concept of an agent's reasoning chain. Data loss prevention tools don't look inside an LLM's context window. A cloud access security broker can't tie a tool call back to a human user when the call came from an agent acting on its own initiative. None of this is a bug in these tools. They were built to watch different traffic, full stop.

Most enterprise threat models still treat the model endpoint as the perimeter worth defending. In agentic systems, the real exposure sits around that endpoint: content the agent retrieves, responses it gets back from tools, memory it carries across sessions, sub-agents it spins up on the fly, MCP servers it talks to. Prompt injection shows exactly where current defenses run out of road. The Agent Security Bench, presented at ICLR 2025, tested attacks and defenses across 10 agent scenarios, more than 400 tools, 13 LLM backbones, and nearly 90,000 test cases. The strongest attacks succeeded 84.3% of the time on average, and not one tested defense held up well. Separately, the 2025 WASP benchmark found simple, human-written prompt injections partially succeeding in up to 86% of realistic web-agent tasks.

This isn't lab talk, either. In June 2025, researchers at Aim Security documented EchoLeak, a prompt injection attack against Microsoft 365 Copilot running in production. One inbound email triggered it. Nobody clicked anything. Confidential files went to an attacker-controlled server while Copilot kept behaving normally from the user's side of the screen the entire time.

Non-human identity is where the access problem really piles up. Machine identities now outnumber human ones somewhere between 45 to 1 and 100 to 1 inside enterprise environments, and the hygiene around them is bad. Researchers found 24 million leaked non-human-identity credentials sitting on GitHub in 2025; 70% of credentials leaked back in 2022 were still valid when someone checked. Nearly half of teams, 45.6%, still lean on shared API keys so agents can authenticate to each other, and 27.2% went back to custom, hardcoded authorization logic because nothing off the shelf fit their setup. Only 21.9% treat AI agents as their own identity-bearing entities with dedicated access controls, a gap that access-control platforms like Oso have begun addressing specifically for agent workloads. Everyone else is improvising, and improvising with credentials tends to end badly.

Supply chain risk now runs through the skills and plugins an agent calls at runtime, a category that barely existed two years ago. Formal vulnerability tracking for agentic AI systems is maturing rapidly, with malicious skill packages targeting enterprise developer environments already documented, underscoring how quickly the threat model for plugin ecosystems is developing. Industry risk frameworks have since begun folding runtime skills and plugins into their formal supply chain categories. Emerging security frameworks for agentic applications consistently highlight prompt injection and jailbreaks, memory poisoning, and tool or plugin misuse among the most critical risk categories. None of them have solid coverage in the security stack most enterprises already own, which means evaluating a vendor against this list is mostly additive work: the controls don't exist yet.

The access control test: what "least privilege" actually means for an agent

Least privilege for an agent is something a vendor has built or hasn't. It is not a checkbox in a settings panel, no matter how the sales deck frames it. In typical enterprise deployments, agents get roughly 10 times more access than the workflow actually needs, and that excess creates toxic combinations of privilege: a single compromised agent turns into a much bigger problem than it should ever have been able to become. SailPoint's research found 80% of organizations say their AI agents have already done something outside their intended scope, whether that's reaching a system they shouldn't have touched, sharing sensitive data, or exposing credentials somewhere they didn't belong.

The payoff for fixing this is one of the cleanest numbers in the whole space, and it's worth sitting with. Teleport's 2026 research found organizations enforcing least-privilege access on their AI agents report a 17% incident rate, a substantially lower figure than those that don't. Of the controls Teleport tracked, this one showed a measurable drop in risk.

So what do you actually check across the table from a vendor? Start with whether they can automatically discover every agent and its credentials, including shadow agents nobody provisioned through IT and whatever someone built in "maker mode" over a weekend. Ask about effective-authority mapping: not what permissions got assigned on paper, but what the agent can actually reach given the full chain of credentials it holds. Find out whether authorization gets enforced at runtime or only checked once at deployment, because a static snapshot from six months ago misses whatever privilege drift has happened since then. Ask how fast access gets revoked when an agent is paused, decommissioned, or found compromised, and press on whether agent-to-agent calls use short-lived, scoped credentials or shared API keys under the hood. That last answer tells you almost everything.

The OWASP Agentic Top 10 maps cleanly onto non-human identity failures: over-privileged identities, exposed secrets, vulnerable third-party integrations, and credentials that never expire show up as root causes across multiple top risks. A vendor's NHI posture is a decent stand-in for how seriously they've taken that list overall. One phrase to distrust on sight: "we support your existing IAM." That's a non-answer unless they can explain how agent identities stay separate from human ones and how policy gets enforced on agents specifically, at runtime, not just at login.

Runtime monitoring: the difference between knowing an agent behaved badly and stopping it before it does

Sit with this gap for a second: 88% of organizations confirmed or suspected an AI agent security incident in the past year, and in that same pool, 82% of executives said they felt confident their existing policies would catch unauthorized agent actions. Same organizations, both numbers. That confidence doesn't square with the outcomes, and the disconnect traces straight back to monitoring that tells you what happened yesterday, not what's happening right now.

Real runtime monitoring means watching tool calls continuously, not pulling logs from an API gateway after the fact. It means catching an agent whose reasoning has started to drift, one that begins reaching for data outside its normal lane. It means an alert fires the moment a policy gets violated, not the next morning in a SIEM digest. And it has to follow the whole chain: when Agent A hands off to Agent B, monitoring can't lose the thread at the handoff.

The real test is whether a vendor's platform monitors or enforces, because detection and prevention get sold using the same marketing copy despite being different products entirely. Monitoring tells you what happened. Enforcement stops it before it finishes happening. Ask directly what occurs between the moment a violation is detected and the moment action gets taken. Is there a blocking layer, or just an alert that lands in someone's inbox at 9am? A vendor whose architecture only detects, with enforcement bolted on as an optional add-on, fails this test no matter how good the dashboards look.

Given that WASP found prompt injections partially succeeding in up to 86% of realistic web-agent tasks, monitoring has to cover retrieved content and tool responses coming back into the agent, not just what the agent sends out. Ask what layer the behavioral monitoring actually operates at: the API, the model's output, the tool call, or all three together. Ask what happens when a sub-agent spins up dynamically mid-task and was never in the static registry to begin with. And ask about latency between detecting a violation and enforcing against it. Milliseconds, or minutes? The answer usually tells you which one it actually is.

Audit logs that can actually reconstruct what an agent did — and why most can't

Back to that procurement failure. The platform passed every capability and pricing test on the sheet, and it still couldn't answer the one question that mattered when a regulator asked it: what exactly did the agent retrieve before producing that output? A log may exist and still be unable to answer that question — a distinction worth sitting with, because it's the whole ballgame.

An adequate agent audit log captures the full reasoning chain: what got retrieved, which tools got called and in what order, what parameters were passed, what came back to the user or whatever system sat downstream. An inadequate one captures API calls and model outputs but drops the context window content, the retrieved documents, the intermediate responses in between. That gap is exactly where the insurer got burned. Immutability matters too; a log an admin or the vendor can edit after the fact won't survive a SOC 2 audit or a financial services exam.

Retention timelines aren't optional either. Financial services typically requires seven years, healthcare six. Confirm the vendor's default retention window and export format before you sign anything, not after the fact when it's too late to negotiate. Most enterprise agent platform evaluations fail on governance, not on how good the model is. Security, single sign-on, and audit log export are table stakes now, and data residency is fast becoming part of that same baseline conversation.

On compliance: does SOC 2 Type II coverage include the agent runtime itself, or only the infrastructure sitting underneath it? Can logs get pulled into a SIEM or data lake in a structured format, or do they only live inside the vendor's own console? Is there a chain-of-custody record for multi-agent workflows, showing who delegated to whom, under what authority, and when? Emerging agentic AI certification frameworks are beginning to define structured requirements across safety and security domains, and it's worth asking vendors where they stand on whichever frameworks apply to their platform. Third-party accredited auditors are starting to appear for some of these frameworks, giving them practical teeth instead of just a document sitting on a website.

Supply chain and plugin risk: evaluating what the vendor's ecosystem can do to your environment

Forrester's 2026 supply chain taxonomy now formally includes the skills and plugins an agent calls at runtime. That matters because it means a compromised third-party tool becomes your compromise, without a single vulnerability ever touching your own stack.

The OpenClaw incidents from January and February 2026 make the point better than any hypothetical could. CVE-2026-25253, the first CVE ever assigned to an agentic AI system, came from remote code execution through a crafted skill package. Then ClawHavoc hit: hundreds of malicious skills published to the marketplace, each one deploying a credential stealer onto enterprise developer machines. Both attacks got through for the same reason: the platform's plugin ingestion process checked what a skill claimed to do on the way in, never what it actually did once it was running.

Ask the vendor whether their plugin or skill registry is curated and vetted, or an open marketplace anyone can publish to. Ask whether skill behavior gets checked at runtime or only once, at ingestion. Ask how third-party integrations get permissioned: do they inherit the agent's full credential scope, or is each one scoped down on its own? And ask what happens when a plugin that was already approved turns out to be compromised. How fast can the vendor push a revocation across every customer running it, and have they actually had to do it?

One more question if the vendor's agents talk to MCP servers: are MCP server responses inspected before they get fed back into the agent's context window? Prompt injection doesn't only arrive through what a user types. It rides in just as easily on a tool's response, quiet and unannounced. Given how fast agents act and how many of them chain together, a single bad plugin doesn't stay contained. It moves at whatever speed the agents themselves are running, which these days is a lot faster than the people watching them.