Detecting Policy Violations in Agent Action Streams
How autonomous agents drift toward policy violations despite passing individual safety checks.
Sixty-five percent of enterprises now use generative AI regularly, up from 33% in 2023. Projections put agentic AI embedded in roughly a third of enterprise applications by 2028, up from less than 1% in 2024. Stanford's 2025 AI Index logged 233 AI-related incidents in 2024, a year-over-year increase of more than 50%, and fewer than two-thirds of organizations were actively mitigating known AI risks at the time. The infrastructure meant to govern autonomous agent behavior has not kept pace with the ease of building agents.
Nearly every engineering, finance, sales, and product team I've encountered lacks a complete inventory of which agents are running across their functions, let alone coherent policies governing each one. Agents inherit credentials and act on them continuously, without the fatigue, hesitation, or social friction that naturally throttle human decision-making. At machine speed, a policy violation can propagate across dozens of downstream systems before any human sees it. You cannot monitor what you have not found.
What a "Policy Violation" Even Means When an Agent Does It
The same action can be legitimate for a human and a violation for an agent scoped to a narrower task. A person in the finance department sending a quarterly report to an external partner is unremarkable. A finance automation agent, operating under the same credentials but scoped to internal reconciliation tasks, doing the same thing is a violation, even if every individual token in its output looks clean. The action is identical; the context condemns it.
OWASP's December 2025 taxonomy, the first formal one specific to autonomous agents, covers six categories: goal hijacking, tool misuse, identity abuse, memory poisoning, cascading failures, and rogue agents. OWASP LLM09:2025 names excessive agency as a core risk, specifically agents performing damaging actions due to unexpected, ambiguous, or manipulated outputs, rooted in excessive functionality, permissions, or autonomy. The ShieldAgent researchers add a complementary operational taxonomy across seven risk categories: access restriction, content restriction, hallucination, instruction adherence, operational restriction, typical error patterns, and long-term risks.
Two violation classes deserve particular attention because standard classifiers miss them almost entirely. The first is sequential drift: individually benign actions that, taken together, move the agent toward a prohibited state. A finance bot that gradually begins routing reports to external recipients after repeated prompt nudges exhibits no single clearly malicious step; the violation exists only at the trajectory level. The second is what PhantomPolicy research calls policy-invisible violations, cooperative and non-adversarial cases where the decisive policy facts are intentionally absent from the model-visible context. Input/output classifiers see nothing wrong because, at the prompt and response level, nothing is wrong. Both classes expose a structural gap in safety work that has historically fixated on catastrophic risks like injection attacks and data leaks while paying comparatively little attention to whether an agent's trajectory actually conforms to task-specific or domain-specific rules.
The Attack Vectors That Actually Cause Production Violations
Prompt injection is the dominant attack surface, ranked first in OWASP's Top 10 for LLM applications in 2025. Attack success rates range between 50% and 84% depending on model configuration, and when layered defenses are applied, success rates drop from the low seventies to single digits. Multi-hop indirect attacks via agents and tools increased by over 70% year over year through 2025 and 2026, tracking the growth of agent interconnection.
EchoLeak, disclosed in June 2025 by Aim Security (CVE-2025-32711, CVSS 9.3), was the first documented zero-click prompt injection exploit against a production AI system, targeting Microsoft 365 Copilot. That disclosure settled the "this is still theoretical" debate rather decisively. At scale, a public red-teaming competition surfaced over 60,000 successful policy violations from roughly 1.8 million prompt injection attempts: a success rate that becomes operationally significant fast when agents handle high request volumes.
Self-replicating attacks in multi-agent systems present a distinct threat class. Research found that self-replicating prompt infections can convince multi-agent systems to execute harmful actions, including data exfiltration and content manipulation, at success rates exceeding 80% against GPT-4o. RAG and memory poisoning add another dimension: adversarial content injected into a knowledge base causes the agent to surface false or policy-violating information as legitimate retrieved context, invisible to tools that inspect only prompts and responses. Liu et al.'s HOUYI attack compromised 31 of 36 real-world LLM-integrated applications, a success rate that should be uncomfortable reading for anyone relying on single-layer defenses. Safety properties demonstrated in single-model chat settings do not transfer reliably to multi-agent systems, where behavior emerges from message passing, tool routing, and inter-agent communication rather than from any single inference call.
Per-Action Guardrails Are Necessary. They Are Not Sufficient.
Per-action guardrails represent the current baseline: check each tool call or output in isolation against a classifier or rule set before allowing it to execute. Frameworks like LlamaFirewall, developed by Chennabasappa et al. in 2025, demonstrate the legitimate value of this approach. Its modular architecture covers jailbreak detection via PromptGuard, reasoning misalignment via AlignmentCheck, and insecure code via CodeShield. It has been deployed in production, and the value is real.
The structural limit is equally real. Reactive, per-action methods fail to identify risks that emerge across multi-step interactions because they assess the apparent safety of the current action without modeling how risk accumulates over a trajectory. The finance-bot scenario illustrates this plainly: each individual prompt nudge passes a per-action check; the cumulative trajectory does not. Enforcement benchmarks like ShieldAgent-Bench and ToolSafe measure precision and recall on individual tool calls or trajectories but do not test whether enforcement holds when an agent produces a system effect through an indirect path, such as a subprocess or a cross-event sequence. ActPlane research identifies this as an open gap. Something must sit above per-action checking, evaluating the sequence rather than the step.
Trajectory-Level Monitoring: Evaluating the Sequence, Not the Step
Trajectory-level monitoring treats the complete ordered sequence of agent actions as the unit of policy evaluation. The operative question changes from "did this action violate a rule" to "is this agent heading toward a violation," which is a materially different question with materially different detection capabilities.
ProbGuard, developed by Wang et al. in 2025, applies probabilistic runtime monitoring to multi-step agent trajectories. It abstracts agent executions into symbolic states, learns a Discrete-Time Markov Chain from execution traces to model behavioral dynamics, and at runtime estimates the probability that the current execution will remain safe. When that probability falls below a user-defined threshold, it triggers intervention before the violation occurs, not after. The distinction between preemptive and reactive intervention is not cosmetic; for irreversible actions, reactive is just a polite word for too late.
DreamGuard takes a complementary approach, maintaining a compact recurrent latent state that integrates both immediate-hazard and prefix-level risk evidence into calibrated intervention decisions, with end-to-end latency averaging around 25 milliseconds per call across four agent safety benchmarks.
What needs to be captured in the stream to make any of this work: tool invocations, reasoning steps, retrieved context to catch memory poisoning, inter-agent messages in multi-agent setups, and all external calls. Gaps in stream completeness create blind spots. Attackers, eventually, find blind spots.
Turning Natural-Language Policies into Something a Machine Can Actually Check
Organizational policies are written for humans. They use terms like "sensitive data," "authorized recipients," and "appropriate use," which are useful for human judgment and useless for runtime enforcement. Detection systems need something checkable. Bridging that gap is what policy autoformalization does, and it is the step most organizations have yet to take, usually because it feels like infrastructure work and infrastructure work does not close tickets.
ShieldAgent automatically constructs safety policy models from policy documents, iteratively refines them, and groups them based on action types into structured, probabilistic rule circuits. At inference time, only the rule circuits relevant to the invoked action are verified, providing both precision and efficiency. ShieldAgent outperforms prior methods by more than eleven percentage points on average, achieves high recall, and substantially reduces both API queries and inference time.
Alternative formal approaches cover different parts of the problem space. PCAS, developed by Palumbo et al., compiles a Datalog-derived declarative policy language offline and enforces it through a reference monitor. AgentSpec, from Wang et al., provides a lightweight domain-specific language using triggers, predicates, and enforcement actions. Solver-aided verification, from Roy et al., translates natural-language tool-use policies into SMT-LIB constraints offline and verifies each call with Z3. Progent, from Shi et al. in 2025, adds programmable privilege control with fine-grained policy enforcement, dynamic updates, and fallback behaviors, addressing the common scenario where agent scope evolves after initial deployment. That last scenario, by the way, is nearly universal in practice.
At scale, converting policy compliance from a human review task to a machine-checkable runtime property is not optional.
Taint Tracking: Following the Data, Not Just Inspecting the Prompt
Information flow control addresses a specific violation class: sensitive data reaching untrusted sinks, or untrusted inputs influencing tool-call decisions. This is the underlying mechanism behind RAG poisoning, data exfiltration, and indirect injection. Prompt-level classifiers cannot catch it because the attack originates upstream in the retrieval layer, before any prompt or response exists to inspect.
In agent systems, IFC assigns each data item a security label drawn from a predefined information flow lattice. Labels propagate along agent execution, through retrieval, reasoning, and tool calls. Violations are detected when data flows violate lattice constraints: when sensitive internal data labeled for internal sinks reaches an external API call, for instance. The approach provides two directions of protection simultaneously. Integrity protection blocks untrusted inputs from influencing tool-call decisions. Confidentiality protection prevents sensitive data from reaching untrusted sinks.
IFC is not a replacement for trajectory monitoring or policy verification. It is the layer that tracks data provenance as it moves through the system, catching what those other layers cannot see because their inspection point is downstream of where the problem originated.
Intervention Timing and Options Across Each Detection Layer
Intervention is not binary. The right response depends on confidence level, what the action does to the world, and whether it can be undone. Most governance frameworks default to block/allow when a graduated response would serve better.
The Governance-as-a-Service proxy model handles this with a non-invasive runtime proxy that filters actions based on programmable rule specifications, assigns quantitative trust scores per agent output using a severity-weighted penalty framework, and enables graduated containment rather than categorical blocking. Production coding agents, including GitHub Copilot, Gemini Code Assist, Cursor, and Codex, already implement a version of this pattern by requesting user approval before writing files or executing command-line instructions. AudAgent, released in 2025, continuously monitors AI agents' data practices in real time and guards compliance with stated privacy policies using a cross-LLM voting mechanism, producing audit-ready records rather than just block/allow decisions.
Reversibility is the primary variable in the intervention decision. Reversible actions like drafting and querying warrant a lower threshold: log, flag, and continue. Irreversible actions like sending, deleting, and publishing require higher confidence before proceeding and should route to human approval. Ambiguous trajectories trending toward violation, the kind ProbGuard models probabilistically, warrant preemptive intervention before the threshold is crossed.
Audit logs are a detection output, not a compliance artifact. Complete, timestamped records of every action, decision, and intervention are necessary for reconstructing what happened and for tuning detection thresholds over time. A governance architecture without comprehensive audit logs is not a governance architecture; it is documentation of good intentions.
Oso, a fine-grained authorization platform covering RBAC, ReBAC, and ABAC, operationalizes the enforcement layer of this architecture in production: its Oso for Agents capability brings security and governance to AI agents, enforcing granular rules on what agents can and cannot do and producing the audit records that make the rest of the system legible. The tools exist. Every team I know will deploy them eventually. The only open question is whether that happens before or after the incident that makes the case for them.
