AI Threat Modeling for Agentic Systems: A Practical Guide

Your team is about to ship an agent that reads internal documents, opens tickets, and calls service APIs. The prompt has been reviewed, the demo works, and someone has asked security to approve it. What's still unclear is which identity executes those calls, whether retrieved text can redirect them, and what a hijacked run could change before anyone notices.

AI threat modeling should answer those questions before it becomes a catalog of hostile prompts. Start with the assets the system can reach and the actions its credentials permit. Then require every proposed defense to demonstrate both that it blocks an attack and that people can still get legitimate work done.

Classify Your AI System Before You Model Threats

Listing attacks before agreeing what the system is wastes the modeling session. A fraud scorer and a support agent may share model infrastructure, but their security boundaries and failure consequences aren't interchangeable.

NIST's adversarial machine learning taxonomy organizes assessment around system type, lifecycle stage, attacker objective, capabilities, and knowledge. Use those dimensions to settle scope before debating mitigations.

Answer the classification questions

1. Is it predictive or generative AI? A fraud model predicts a score from transaction features. A support chatbot generates responses, retrieves help-center pages, and may act as an agent when it opens tickets. Record the application's behavior, not just the model's name.

2. Which lifecycle stage is in scope? For fraud scoring, include transaction collection, labeling, retraining, and serving. For the chatbot, inference and retrieval may be your immediate scope, but explicitly identify who owns upstream model and data risks.

3. What does the attacker want? A fraudster may want an integrity violation that makes a fraudulent transaction appear legitimate. A chatbot attacker might want another customer's information, unauthorized ticket changes, service disruption, or misuse of generated responses.

4. What access does the attacker have? Distinguish a public scoring client from an insider who can alter labels. For the chatbot, separate an authenticated customer from someone who can edit a retrieved help-center page. Those actors enter through different trust boundaries.

5. What does the attacker know? Record whether they know feature definitions, training examples, model behavior, prompts, or tool schemas. Do not assume an attacker lacks knowledge that ordinary documentation or repeated queries could reveal.

Complete the system card

Keep a one-page system card beside the architecture diagram. Include the system name, accountable owner, lifecycle scope, data sources, outputs, callable tools, execution identities, and excluded components with their owners.

For the chatbot, “generates support answers” isn't sufficient. Write down which ticket API it calls, whose permissions it uses, and whether opening a ticket can trigger another workflow. The prompt is an input to that system, not a complete description of it.

Map Attack Surfaces Across the ML Lifecycle

The deployed model is only one surface. Training inputs, artifact promotion, output consumers, and tool credentials all need a place on the diagram.

This isn't a risk category invented for chatbots. NIST's historical account of adversarial machine learning traces early evasion research to 1988 and documents poisoning attacks against worm-signature generation in 2006. The engineering implication is straightforward: review how the system learns and how its decisions become actions, not just what users type.

Walk the architecture in execution order

Begin at collection. Identify scraped sources, labeling vendors, and user feedback that can eventually become training examples. Ask who can contribute records, who can change labels, and whether the team can trace a suspicious batch back to its origin.

Follow those records through notebooks, feature stores, and CI training jobs. Then inspect the registry, weight files, and container images used for deployment. A trustworthy training run doesn't protect a model artifact that someone can replace afterward.

At inference, inspect both directions. Inputs include scoring requests, chat messages, and batch uploads; outputs include scores, explanations, and logs. Finally, identify the consumer that converts an output into a fraud hold, moderation action, or merged pull request.

Attack surfaces by ML lifecycle stage

For agents, draw retrieval and tool connections explicitly rather than hiding them inside an application box. Label the identity on each connection and distinguish read access from write access. That makes the attack-surface analysis useful to the engineer who must implement a boundary.

Practical rule: Remove a surface from scope only after writing down why it isn't exposed and who verified that assumption.

Mark each applicable row against a concrete component or repository path. If your team doesn't train the model, record that boundary instead of deleting training risk without notice. You still need an owner for evaluating the artifact you receive.

Threat Templates for Models, Data and Agents

A useful threat entry is an engineering work item, not a label such as “poisoning possible.” Write down the asset, lifecycle location, attacker prerequisites, violated security property, attack path, detection signal, mitigation, residual risk, and validation test.

Use those fields to separate what an attacker must already control from what your system accidentally permits.

Write an entry someone can implement

Say a fraud system retrains from customer dispute outcomes. The asset is the integrity of fraud decisions; the lifecycle location is feedback ingestion and retraining. The attacker prerequisite is an ability to submit accepted dispute outcomes, not direct access to the training job.

The attack path is to introduce mislabeled transactions that influence the next model. The violated property is integrity. Monitor performance on a protected canary set, segmented by transaction characteristics, rather than treating a global accuracy score as sufficient.

The mitigation is to retain source provenance, quarantine suspicious feedback, and require promotion checks before retraining data reaches production. The residual risk is that coordinated, plausible submissions may pass those checks. The validation test retrains isolated candidates with and without a planted batch, then compares targeted behavior and legitimate transaction handling.

Treat what you feed the model as a governed dependency. Document who can approve a source, alter its transformation rules, and reverse a bad promotion.

Prioritize the threat library

Rank entries by attacker feasibility and blast radius, and record the reasoning alongside the system card. Validate high-impact paths first; don't let an uncertain likelihood estimate excuse a potentially unacceptable outcome.

Common AI threat families with signals and tests

These signals are investigation leads, not proof of an attack. A confidence gap needs a controlled privacy evaluation; a query burst may be a legitimate integration.

Before accepting an entry, require a runnable test and written residual-risk acceptance from its owner. Blank residual risk means the decision is unfinished, not that the control is perfect.

Agent Threats Live in the Whole System

A stronger system prompt isn't a substitute for constraining an agent's authority. Agent risk comes from the interaction of the model, callable tools, orchestration harness, and execution environment.

NIST's treatment of agent security describes agent-specific research as early and identifies tool-enabled hijacking that can lead to arbitrary code execution or exfiltration of environmental data. Model behavior matters, but the surrounding system determines what that behavior can accomplish.

Trace a combined attack path

Consider a coding agent that reads repository issues, runs a shell, and operates in an environment containing a CI token. Each capability has a reasonable purpose: understand the requested fix, execute tests, and interact with build infrastructure.

An attacker places instructions in an issue that look like troubleshooting guidance. The harness passes that issue into the agent's context. The agent follows the instructions, uses the shell to read an environment secret, and attempts to send it to an external destination.

The decisive questions aren't whether the issue contained suspicious wording. They're whether the runtime exposed the secret, whether the shell could reach an unapproved destination, and whether the token carried authority beyond the task. An output filter that checks the final answer wouldn't necessarily observe the attempted transfer.

Don't give an agent permanent authority merely because some legitimate tasks occasionally need it.

Keep untrusted task content distinguishable from approved instructions, but enforce consequential boundaries outside the model. Use short-lived task-scoped credentials, isolated workspaces, constrained outbound access, and approval for sensitive actions. Approval must bind to the actual operation and parameters, not a vague request to continue.

This is why securing the agentic development lifecycle requires reviewing orchestration and runtime changes alongside model changes.

Complete the blast-radius worksheet

For each agent, record:

• Identities: User delegation, service accounts, token lifetime, and impersonation rights.
• Tools: Callable operations, parameter constraints, and destructive capabilities.
• Reachable data: Repositories, secrets, customer records, memory, and mounted files.
• Approval boundaries: Which action needs authorization and where enforcement occurs.
• Worst realistic outcome: What a hijacked run could disclose, modify, delete, or trigger.

Reduce any outcome the business can't accept. Removing shell access may prevent necessary development work; limiting filesystem mounts and network destinations may preserve it. Choose the narrower control that enforces the required boundary, then test the actual workflow.

Test Guardrails Without Breaking the Agent

An agent that refuses every request is difficult to exploit and useless to deploy. A control can lower attack success by preventing all meaningful action, so security and utility need separate measurements.

Use the adversarial attack framing in NIST's adversarial machine learning guidance to define attacker objectives and test conditions. For agent release decisions, report attack outcomes alongside legitimate task completion rather than compressing both into a single score.

Metrics to report together for an agent

Define the denominator for every metric. Keep unsuccessful attack attempts in the record, and distinguish an observed tool request from a completed unauthorized action.

Build a reproducible evaluation

1. Establish benign tasks and success checks. For ticket creation, verify the correct customer, category, content, and absence of unintended changes.
2. Introduce an isolated threat mechanism. Test malicious user input, poisoned retrieved content, and compromised tool responses separately.
3. Measure attack and task outcomes independently. An agent can complete the requested task while also leaking data.
4. Combine threat mechanisms. Test whether retrieved instructions and a misleading tool response defeat controls that handle either independently.
5. Repeat across sessions and permissions. Exercise multi-turn interactions, persisted state, and both restricted and privileged identities.
6. Apply security and business gates. Define unacceptable actions and minimum useful task completion before reviewing results.

Use tool traces, authorization decisions, and resulting state as evidence. A clean final response doesn't prove that the agent avoided an unauthorized side effect.

Report the failures that averages hide

Don't count only overtly harmful output, ignore stealthy behavior, or average read-only search together with privileged deployment tools. Single-turn testing also leaves persistent context and delayed actions unexamined.

A defense earns deployment only when it constrains the attacker without disabling the work you intended to authorize.

Stratify results by attack strategy, model version, tool scope, and session length. Keep the failing cases as regression tests so a prompt adjustment doesn't reopen a previously closed path.

Keep Threat Models Current in Your Delivery Workflow

Refresh the threat model when engineers introduce:

• Changed training data: New sources, labels, filters, or transformations.
• A new model version: Including a replacement behind an existing endpoint.
• New retrieval sources: Especially sources with different editors or access rules.
• New or altered tools: Including expanded parameters and changed behavior.
• Changed permissions: New identities, roles, scopes, or delegation.
• A moved deployment boundary: Different hosting, network access, or runtime isolation.

After such a change, an untouched threat model may describe an architecture you no longer operate. Route the affected threat entries to their owners rather than reopening every unrelated design decision.

Put review evidence beside the change

Store the system card, threat entries, and approved controls beside design documents and engineering tickets. A pull request that expands a tool's permissions should identify the affected boundary, run its validation test, and flag a broken control before merge.

SANS's AI security report reports that 81% of security professionals are concerned about AI-powered threats. Concern isn't a release process. Give reviewers a concrete decision to make and evidence they can inspect without reconstructing the system from chat messages.

At DevArmor, we bring real-time security design feedback into design docs, Jira, and PRs. We analyze specs and tickets to generate structured, explainable feedback and policy checks that run instantly in PRs. Judge automation against the same standard as a manual review: did it identify the changed boundary and produce an actionable, traceable decision?

Measure the review loop

Track model freshness as the relationship between the current implementation and its last relevant review, not just the document's modification date. Measure review latency from a reviewable change to a decision, separating time spent waiting from time spent fixing a finding.

Keep a register of policy exceptions, their owners, and expiry conditions. Count blocked unsafe changes only when someone confirms the control caught a real violation. Record which attack paths were validated and connect post-deployment incidents to missing assumptions, ineffective controls, or uncovered paths.

These measures help distinguish useful enforcement from noisy friction. If engineers repeatedly override a check, investigate whether the architecture needs changing or the rule misrepresents approved behavior.

Define release blockers before the release meeting. Unauthorized cross-customer access, unbounded secret exposure, and bypassable approval for irreversible actions deserve explicit decisions, not informal reassurance. Name the person authorized to accept residual risk and store their acceptance with the finding, supporting evidence, and conditions for reconsideration.

Run Your First AI Threat Model This Sprint

Choose a system your team can change next week. Don't begin with a portfolio inventory that postpones testing until every application has perfect documentation.

1. Pick the consequential system. Start with the application making sensitive decisions or the agent holding the broadest permissions.
2. Complete its system card. Answer the classification questions with the engineering owner and record unresolved assumptions.
3. Mark exposed surfaces. Walk the lifecycle map against the deployed architecture, including inherited services and credentials.
4. Write five threat entries. Choose concrete attack paths, assign owners, and attach runnable validation tests.
5. Evaluate the agent's behavior. Record attack success, benign task success, task success under attack, and stealth outcomes using observable side effects.
6. Set the operating rules. Assign refresh triggers, release blockers, and a named residual-risk approver.

Finished means a signed-off model, an owned list of open risks, and tests that run on the next relevant change. A completed workshop without those artifacts isn't a release control.

You can run this process by hand or automate the repetitive work, depending on how often the system changes. At DevArmor, every review is traceable to an approved design decision or threat model, and we document security architecture and decisions as audit-ready artifacts supporting compliance work such as SOC 2 and ISO 27001. Keep engineering validation separate from evidence generation: a well-documented assumption still needs a test.

---

DevArmor creates a threat model and generates requirements in less than 10 minutes, giving your team a starting point for reviewing agent permissions, trust boundaries, and controls. Visit DevArmor to bring an application design into the review workflow and connect its security decisions to implementation checks.