Anomaly Detection in Agentic Workflows

Classical monitoring is passive by design: it evaluates discrete events against fixed thresholds on predefined data streams, written for known patterns, calibrated to known failure modes. The entire paradigm assumes you can define what bad looks like before it arrives. Agents do not cooperate with that assumption.

They read files, send messages, execute code, query databases, and call APIs, often with no human review at any intervening step. The action lands before anyone has considered the reasoning behind it. Classical logging captures inputs and outputs at defined checkpoints; it misses the agentic loop entirely: the reasoning process, the tool selection logic, the intermediate states, the chaining decisions connecting one action to the next. You get the beginning and the end. Everything interesting happened in between.

Static thresholds work reasonably well for infrastructure metrics. They have no representation whatsoever for "agent chose a tool outside its normal scope." That is not an infrastructure event. It is a behavioral event, and behavior has no threshold. This is a categorical mismatch, not a calibration problem you can tune your way out of.

Where this gets alarming is multi-agent compounding. When agents hand off context and instructions to other agents, a message that appears benign at one layer can carry embedded instructions that activate at a higher-privilege layer. Classical logging at the edge captures neither the handoff semantics nor the downstream activation. Documented attacks against enterprise AI assistants have exploited exactly this gap: malicious instructions embedded in data fields that appeared benign to the processing agent but triggered consequential action when passed upward. No per-event rule catches this, because the rule would need to understand the relationship between the first event and the fourth.

That is not a rule. That is a model.

The old risk question was what a model would say. The new risk question is what a system will do. Those require different tools, and pretending otherwise is how teams end up surprised.

The Deviations That Matter Are Sequences, Not Events

Two meaningfully distinct anomaly types exist in agentic workflows, and neither one produces a single event that crosses a rule-based threshold.

The first is a plan that is structurally valid but misaligned with the assigned task: the agent executes a coherent sequence of actions, just not the ones it was supposed to execute. The second is a structurally malformed plan, where steps do not follow from each other, loops appear, or actions directly contradict the agent's own earlier reasoning. Detecting either requires reading the sequence as a whole. There is no individual event to catch.

Prompt injection is the primary driver of these deviations in practice. OWASP ranks it first in its Top 10 for LLM Applications 2025, because large language models cannot reliably distinguish trusted instructions from untrusted data. This is a predictable outcome whenever an agent encounters adversarial input in the wild, not an edge case.

CVE-2025-32711, dubbed EchoLeak, is instructive. It caused Microsoft Copilot to exfiltrate sensitive data from engineered email prompts with no user interaction, as documented in the CVE record and Microsoft's May 2025 security advisory. At the level of any individual event, the agent's behavior was indistinguishable from normal operation. Only the sequence of actions and the destination of the output revealed the anomaly. A threshold on any single event would have missed it entirely, which is uncomfortable to sit with if your current posture relies on thresholds.

In multi-agent systems, the deviation patterns extend further: manipulation of inter-agent communication channels, rogue agents operating outside monitoring boundaries, exploitation of trust and delegation relationships that were never designed to be adversarial surfaces. OpenAI's December 2025 guidance on agentic system safety acknowledged that prompt injection remains unsolved at the architectural level. That is not a reason to despair but a reason to invest heavily in detection. If prevention cannot be guaranteed architecturally, detection carries the full weight of the response posture.

Why the Obvious Fixes Do Not Work

The instinct when facing a new detection problem is to reach for existing unsupervised methods: variational autoencoders, Isolation Forests, embedding-based similarity measures. Apply them to agent logs, tune a threshold, and move on. This approach fails, and the reason is architectural.

The mean-pooling failure is the core problem. When you average embeddings across a trajectory's steps to produce a single vector representation, the anomalous steps, which might be one or two in a sequence of dozens, dissolve into the aggregate. The trajectory looks normal because the normal steps vastly outnumber the compromised ones. The attack is statistically laundered by the very method you deployed to catch it. I have watched teams test their detection on synthetic single-step anomalies, declare victory, and then deploy into production multi-step workflows. The problem remains unsolved.

Contrastive-only approaches have a different failure mode: they can flag a trajectory as different from a normal one, but they do not evaluate whether the sequence itself is internally coherent. A trajectory can be unusual and entirely valid. It can also be familiar-looking and internally incoherent. Contrastive methods conflate those two dimensions, generating false positives on novel but legitimate behavior and false negatives on attacks that mimic the structural surface of normal workflows. Both failure modes are expensive; the second one is dangerous.

Measured against trajectory anomaly benchmarks, standard unsupervised methods on pre-trained embeddings plateau at F1-scores around 0.69. LLM-based evaluators do substantially better, reaching the low-to-mid 0.90s in some assessments, but at latencies between 550 and 730 milliseconds per evaluation. When agents are acting across multiple systems simultaneously at machine speed, that latency is disqualifying for real-time use. You are auditing, not detecting.

Work on Trajectory Guard, which applies a Siamese Recurrent Autoencoder with a hybrid loss function, addresses the architectural gap more directly. The hybrid loss simultaneously learns semantic alignment between the assigned task and the agent's plan via contrastive learning, and sequential validity of the plan's structure via reconstruction. The architecture maintains memory across steps rather than evaluating each step in isolation. Benchmarked results put F1-scores between 0.88 and 0.94 on synthetic data, with recall between 0.86 and 0.92 on real-world logs. That is not a closed problem, but the dual-objective framing is the right one, and the mean-pooling instinct is the wrong one.

Multi-Agent Systems Need Graph-Aware Detection

Individual agent trajectory detection is necessary but insufficient once you have more than one agent in a workflow — which describes most production deployments. The threat surface expands to the communication layer between agents, and that layer is essentially unmonitored in the majority of enterprise deployments currently running.

Graph-based detection maps agent communication flows as a network structure and identifies anomalies in how agents relate to each other: covert data leakage paths, unauthorized tool invocations that propagate across agent boundaries, structural changes in the communication topology that do not correspond to any legitimate workflow. Research on SentinelAgent applies this approach specifically to inter-agent anomaly detection and illustrates why the graph structure carries information that per-agent logging cannot. A single agent's logs may look entirely clean while the communication pattern between three agents reveals an obvious extraction pattern. The per-agent view is not just incomplete; it is actively misleading.

Behavioral baseline monitoring operates as a complementary layer. Establish normal operating patterns per agent before deployment: which data it accesses, which tools it invokes, where it routes outputs. Then alert on deviations from that agent-specific baseline, not from a universal rule. An inventory management agent that begins issuing SQL DROP TABLE commands, or accessing privileged directories it has never touched, should trigger detection immediately, not because a universal rule prohibits those commands in the abstract, but because they fall outside that specific agent's established behavioral envelope. A universal rule will either miss context-specific anomalies or generate alert fatigue across the entire fleet. Agent-specific baselines avoid both failure modes.

Circuit breakers complete the response layer. When behavioral deviation crosses defined thresholds, the agent is suspended before failures propagate downstream. The key property shared across graph-based detection, behavioral baselines, and circuit breakers is that they are agent-aware. They model what a specific agent should do in context. That specificity is what makes detection tractable in practice rather than aspirationally correct in theory.

You Cannot Detect What You Cannot See

Traditional logging captures edges: inputs and outputs. Agentic observability must capture the interior: reasoning steps, tool selections, intermediate states, retry patterns, latencies, and costs within each workflow execution. The fundamental data structure is not a point-in-time log entry; it is a trace across a multi-step, potentially multi-model, potentially multi-modal workflow. This sounds obvious until you look at what most teams are actually collecting.

Enterprise-grade agentic observability requires: full trace visibility across every step of every agent workflow; real-time dashboards that update continuously rather than on polling intervals; anomaly detection operating on behavioral patterns rather than metric thresholds; high-availability monitoring infrastructure that stays operational when the systems it monitors are degraded; and complete, unsampled audit logs of every agent action. That last point is non-negotiable. You cannot reconstruct what happened from a statistical summary of what happened. Sampling made sense when compute was expensive and events were high-volume but low-variance. Agent actions are the inverse: lower volume, high variance, individually consequential.

The compliance dimension is not abstract. The EU AI Act's Article 14 mandates that high-risk AI systems include human-machine interface tools enabling qualified persons to interpret outputs and intervene. Structurally, that is an observability requirement. You cannot interpret what was never recorded, and intervention requires the ability to reconstruct events fully. Cisco's State of AI Security 2026 report found that only 29% of organizations consider themselves prepared to secure agentic AI deployments. The observability infrastructure gap is a leading contributor to that number, not a trailing one.

"Human-in-the-Loop" Is Mostly a Comforting Story

Traditional human-in-the-loop oversight means a human reviews and approves before action occurs. At the cadence agents operate, this is structurally incompatible with the deployment model. Requiring human approval before each agent action either breaks the performance model entirely or creates approval theater, where humans rubber-stamp outputs they cannot meaningfully evaluate in the time available. Both outcomes are bad. The second is worse because it creates the illusion of oversight without the substance.

The operational model that actually works is human-on-the-loop: the agent runs autonomously, humans monitor dashboards, intervene on exceptions, and retain the capacity to override or halt. This is not a degraded form of oversight; it is the appropriate form for the speed regime. What it requires is that the detection layer beneath it is fast enough and precise enough to surface exceptions before the window for meaningful intervention closes. Anomaly detection and circuit breakers must trigger within a sub-second window when behavior crosses defined thresholds. Otherwise the "human-on-the-loop" posture quietly becomes human-notified-after-the-fact, which is forensics.

Automation complacency is the failure mode that compounds quietly over time. The more reliable a system appears, the less vigilant its overseers become. Humans rationalize anomalies, over-trust consistent outputs, and gradually reduce the cognitive effort they apply to reviewing behavior they have never seen go wrong. This is well-documented across aviation, nuclear operations, and software operations, and grows more acute as agents accumulate a track record of benign behavior. The monitoring posture must account for this explicitly — revisiting baselines and alert thresholds on a regular cadence rather than treating them as set-and-forget infrastructure.

Article 14 of the EU AI Act does not require human approval of every agent action. It requires that qualified persons can interpret what the system did and intervene when necessary, which is at its core a monitoring and audit log requirement. The distinction gives teams room to deploy agents at speed while remaining compliant. Squandering that room by under-investing in the monitoring layer is the avoidable mistake.

The Detection-First Posture, Applied

The foundational shift is from perimeter and rule-based controls to continuous behavioral monitoring of the agent itself as the primary detection surface. The perimeter is no longer coherent when the agent can act across multiple systems simultaneously, with inherited human-scale permissions and none of the friction that slows human actors down. Trying to enforce a perimeter around something that is, by design, permissionless and fast is a losing proposition.

Four capabilities define a detection-first posture in practice. Discover every agent running in the environment; shadow agent deployments are already a real phenomenon in enterprises running pilot programs, and you cannot monitor what you do not know exists. Establish per-agent behavioral baselines before deployment, not after an incident, because the post-incident baseline is forensics. Monitor the full agentic loop in real time: tool selection, intermediate reasoning states, output routing, not merely API call logs at the edge. And enforce policy at runtime. Detection without the capacity to halt or constrain the agent is observation. It is not control.

MITRE ATLAS, updated in late 2025 to cover 16 tactics and over 80 techniques, and OWASP's Agentic Top 10 2026 provide useful threat taxonomies for mapping which agent behaviors correspond to which attack patterns. They are practical starting points for deciding which baselines and detection rules to prioritize first. The caveat worth holding: MITRE ATLAS is reactive by nature. It catalogs observed attacks, and the technique count in the agentic AI category remains sparse relative to how fast deployment is accelerating. Treat it as a floor for your detection coverage, not a ceiling.

IBM's Cost of a Data Breach Report 2025 found that AI-related incidents, including prompt injection, contributed to total global breach costs exceeding $4.4 billion in 2025, with enterprises experiencing roughly double the incident response costs when prompt injection was involved. That number clarifies prioritization conversations in a way that technical arguments sometimes cannot.

By the time an agent has acted on a compromised instruction at machine speed across multiple systems, the question is no longer prevention. It is containment, and containment depends entirely on how complete the prior monitoring was. The detection layer is not a future concern to address once agents are mature and well understood. It is a prerequisite for deploying them responsibly at all.