Baselining Normal Agent Behavior for Alerting

A baseline is a behavioral contract for one agent in one operational context. Not a cross-agent average. Not a company-wide policy. Two agents with identical permission sets can have entirely different expected behavior if their tasks diverge, and conflating them is where most programs go wrong before they've shipped a single alert.

Defining normal requires specificity: which tools and APIs the agent calls and in what sequence; what request volumes look like across hourly, daily, and weekly cycles; which records or storage buckets fall within its typical data access scope; what token usage and per-step latency look like under normal load; which external endpoints it is expected to reach. Each dimension narrows the model. Together, they make anomalies legible instead of just loud.

Timing is structural, not incidental. An agent running nightly batch jobs carries a different behavioral fingerprint than one responding to real-time user prompts. Both fingerprints are correct. Applying one agent's temporal pattern to another and treating the deviation as suspicious is a reliable way to teach your security team to stop reading alerts. The analogy that actually fits: a fire alarm that triggers every time someone makes toast. Eventually someone yanks the batteries, and then you have no alarm at all.

Function constrains the envelope in ways that raw telemetry alone cannot tell you. A data integration agent making high volumes of external API calls is doing its job. A summarization agent doing the same thing is anomalous, even if the call count is identical. The baseline is the boundary between "agent operating as designed" and "agent operating outside that design." Without per-agent context, you cannot draw that line.

The Telemetry Signals That a Baseline Is Built From

Three signals belong on every agent, regardless of function: tool and API call authorization logs, step count with loop detection, and per-step latency. Step count is the cheapest anomaly signal available, requires almost no infrastructure to collect, and catches runaway agents before they become expensive problems. Latency is simultaneously a user experience indicator and an early warning that something upstream has quietly changed.

Richer baselines require more: token usage per trace aggregated by model and prompt, tool success and retrieval quality rates, request volume and timeout rates, cache hit ratios, outbound activity patterns including new or unfamiliar external destinations. Gaps in signal coverage are precisely where real anomalies learn to hide — not because attackers are clever enough to find them, but because gaps mean you simply never look there.

The OpenTelemetry project's GenAI Special Interest Group, formed in April 2024, is standardizing how these signals are structured across vendors, covering LLM client spans, agent spans for multi-step workflows, prompt and completion content events, and aggregated metrics. OpenTelemetry graduated from the Cloud Native Computing Foundation on May 21, 2025, marking a milestone in instrumentation standardization. Using its conventions gives baselines a vendor-neutral schema, which matters considerably when a single agent spans multiple tools, models, and providers.

Instrument first, baseline second. This sequencing is not optional. Collecting inconsistent or incomplete telemetry and then deriving a behavioral model from it produces a model of your logging gaps, leaving you knowing a great deal about what you failed to capture, and very little about the agent itself.

Why Static Baselines Break, and What Rolling Baselines Miss

AI agents are designed to change. Model updates shift latency distributions. Prompt revisions alter tool-calling sequences. New integrations add API destinations that were outside scope during the last security review. All of this is legitimate operational change, and all of it registers as anomalous behavior against a fixed snapshot. A static baseline catches everything, including every intentional improvement you ship.

The result is a steady stream of false positives on deliberate updates. Security teams that cannot manage that overhead stop running the checks. The program dies quietly under the weight of its own rigor, which is an ignominious end.

Rolling baselines solve that problem by using the recent production window, typically the last seven to fourteen days, as the reference. The model adapts automatically to intentional change. Gradual drift, though, is precisely what rolling baselines were not built to catch. If an agent degrades incrementally across each window, the baseline shifts with it, no threshold is ever breached, and the agent can move a substantial distance from its original intended behavior without producing a single alert. You are, in effect, noticing someone age by only ever looking at them today.

Neither approach alone is sufficient, and the choice between them has to be calibrated to the agent's update cadence and risk profile. A high-risk agent with infrequent, deliberate updates might tolerate a static anchor paired with a formal change-management process to suppress expected drift. A high-velocity agent updated weekly probably requires a rolling baseline paired with a longer secondary anchor that catches cumulative movement. These are design decisions, not defaults. A baseline is not a one-time artifact; it needs its own maintenance process, one that can distinguish expected change from unexpected drift rather than absorbing everything equally and calling it oversight.

Three Types of Drift and How to Tell Them Apart

Treating all drift as equivalent is how you produce alerts that security teams have learned to dismiss by calendar. There are three distinct types, they route to different people, and conflating them is its own form of operational failure.

Data drift occurs when the distribution of inputs to the agent changes while the agent's underlying relationship between inputs and outputs remains stable. The agent is receiving different prompts or different data than it saw during the baseline period; it is not behaving differently given equivalent inputs. Data drift is usually a signal for model retraining or prompt review. Routing it to your SOC is a waste of everyone's afternoon.

Behavioral drift is the type that warrants concern. The agent's outputs and actions shift even when inputs are comparable to the baseline period. This can emerge from gradual conditioning across sessions that corrupts persistent state: a slow-motion compromise with no dramatic threshold breach, no single anomalous interaction, just incremental degradation unfolding over days or weeks. Behavioral drift is the hardest type to catch with simple thresholds and the type most likely to warrant a security investigation.

Access pattern drift is the most directly observable. The agent queries records outside its typical scope, calls endpoints it has never called before, or operates at times that fall outside its expected windows. You can see this in telemetry without sophisticated modeling. It often warrants immediate policy enforcement rather than a monitoring ticket, because by the time the ticket is triaged, the action is already done.

Each type maps to a distinct response playbook. Data drift routes to data science. Behavioral drift routes to security investigation. Access pattern drift may trigger automated enforcement. Alerting systems that conflate all three produce undifferentiated notifications, and undifferentiated notifications are what get batch-dismissed on Friday afternoons.

Building an Alerting Architecture That Doesn't Collapse Under Its Own Volume

Alert fatigue is well-documented in security operations. Agent workloads make it structurally worse in ways that standard suppression playbooks cannot address, because the typical fix is to tune down false positives. The problem is that with agents, high-volume alerts can be genuine true positives. One documented case involved an agent that downloaded over sixteen million files while all other users and applications in the same environment combined accounted for roughly one million; this incident has been reported in security industry literature, and readers should verify the source before citing it further. Suppression rules calibrated to human-scale false alarm rates will bury that incident in the queue where it belongs, incorrectly, until the damage is done.

Three structural mismatches break standard alerting architectures on agent workloads. Alert volume may reflect true-positive events rather than false alarms, so suppression logic must be designed with that possibility explicitly in mind. Correlation must run on action chains rather than individual events: a single API call is unremarkable, but twenty consecutive calls to a new endpoint at three in the morning is a story. And by the time a traditional alert fires and routes through human triage, a data exfiltration event conducted at agent speed may already be complete, which means remediation logic needs to live upstream of alert emission, not downstream.

Effective threshold design uses multiple severity tiers: monitoring for minor drift, investigation for moderate drift, immediate action for severe drift. Thresholds should adapt to historical patterns like known high-volume batch windows rather than fire on every peak. Sustained deviation across multiple consecutive measurement windows, not a single spike, should be required before escalation. These calibration anchors are illustrative starting points rather than universal standards: P50 latency under 500 milliseconds, P99 under five seconds, error rates in the one to two percent range, a cost alert when token usage hits double the baseline. Appropriate values will vary by deployment context.

One heuristic worth keeping: use dynamic detection for alerting and static policy for blocking. Dynamic baselines catch behavioral drift; static rules enforce hard limits the system will not cross regardless of context. All agent telemetry should flow into an enterprise SIEM for cross-layer correlation. Mean time to detect and mean time to respond are the tuning signals that tell you whether the architecture is actually working or producing the appearance of oversight. Those two things are the same in name only.

What the Regulatory Environment Now Expects from Agent Monitoring

The NIST AI Risk Management Framework treats continuous monitoring as foundational to managing AI risk. For agents specifically, that means baselining behavior and alerting on drift as a runtime operational practice, not a periodic audit exercise conducted quarterly and then filed somewhere. The MEASURE function of the RMF was designed primarily as a pre-deployment and periodic review mechanism; it does not map cleanly onto systems whose behavioral properties shift continuously in production. The frameworks were written before anyone had to manage a fleet of agents running autonomously around the clock, which is a polite way of saying they are already partially obsolete.

NIST AI 600-1, published July 26, 2024, identifies risk categories specific to generative AI. Organizations running LLM-based agents should apply it alongside the base RMF rather than treat it as an optional supplement. The Cloud Security Alliance published the AAGATE reference architecture in December 2025, providing a concrete technical instantiation of agentic governance principles that operationalizes many of the RMF's more abstract requirements.

The regulatory direction is consistent: adaptive governance, not periodic reviews. Baselining and alerting are the operational mechanism that makes continuous monitoring real rather than aspirational language in a compliance document. Organizations that cannot produce audit logs of agent actions and evidence of anomaly detection face a gap that is likely to attract regulatory scrutiny as agentic deployments scale. Organizations that treat this as a deferred problem may find themselves building retroactively under pressure.

How to Stand Up a Baseline Program in Practice

Start with discovery. You cannot baseline an agent you do not know exists, and in most organizations, a meaningful share of agents are running outside IT-approved channels. Complete the inventory first, including shadow deployments. This step is almost always more uncomfortable than anticipated; most teams find agents running in places no one officially approved, built by people who meant well and never thought to mention it.

Prioritize by risk. Agents with broad data access or financial transaction permissions get baselined first. Agents operating across multiple SaaS systems follow. Low-privilege, single-function agents can come later. Alphabetical order is not a risk framework, but it is approximately what you get if you do not make an explicit prioritization decision.

Deploy OpenTelemetry-compatible tracing before building any behavioral model. Structured, consistent telemetry is the prerequisite. Without it, you are baselining your logging configuration, not your agent's behavior, and the two are very different things.

Run a learning period long enough to capture natural variation: business-cycle peaks, batch windows, normal edge cases. Setting thresholds on the first week of telemetry produces a baseline that represents one week. The agent's actual operational envelope is larger, and the mismatch compounds over time. This is one of those mistakes that feels minor when you make it and significant three months later.

Codify the baseline as enforceable policy, not just a monitoring dashboard threshold. Anomalies should trigger action, not just observation. An agent operating outside its behavioral contract should encounter friction at the infrastructure level. A ticket in a queue is not friction; it is a record of something that already happened.

Version baselines alongside agent versions. When a model update or prompt revision ships, snapshot the old baseline and begin a new learning period. Drift detection cannot function if the reference model is unsynchronized with the agent's current intended behavior. That synchronization is a process problem as much as a technical one, and it is the part most programs skip because it does not show up in any vendor's feature list.

Several platforms address different layers of this problem. Exabeam embeds agent behavior analytics within a SIEM context, making it a practical fit for organizations that want cross-layer correlation without building a separate pipeline. Some fine-grained authorization platforms extend into AI-agent governance, making them relevant where the goal is translating behavioral baselines into enforceable access-control policy rather than leaving them as descriptive thresholds. Organizations running broad agent portfolios typically need coverage at more than one layer, because no single platform addresses discovery, instrumentation, behavioral modeling, and enforcement with equal depth.

A perfect baseline on day one is not the goal. The goal is a feedback loop that runs consistently and adjusts based on alert-to-incident ratios. The program that ships something, measures it, and tunes it every month is more valuable than the program that spent six months trying to get the initial thresholds exactly right before anyone outside the team ever saw it. Done and iterating beats perfect and theoretical, every time.