Develop AI threat scenarios

Build end-to-end scenarios

A threat name such as “prompt injection” is not yet a threat model. A scenario should explain the actor, preconditions, path, affected boundary, control failure, asset impact, consequence, and observable evidence.

Scenario structure

Scenario ID and title:
Actor and starting access:
Preconditions and assumptions:
Entry surface:
Attack or failure sequence:
Trust boundaries crossed:
Assets and required properties affected:
Technical and organizational consequence:
Existing controls and known weaknesses:
Detection or evidence expected:
Recovery requirement:

Use numbered diagram flows and asset IDs so the scenario can be reviewed against the architecture.

Explore the full lifecycle

Acquisition and supply chain

Consider malicious or incorrectly licensed models, packages, containers, adapters, prompts, datasets, and drivers; compromised repositories or maintainers; missing signatures; and approval records that do not match deployed artifacts.

Build and deployment

Consider dependency substitution, secret exposure, policy bypass, unreviewed configuration, mutable tags, unauthorized promotion, environment mismatch, and rollback to a vulnerable release.

Ingestion and knowledge

Consider poisoned or stale documents, parser exploitation, permission loss, cross-tenant indexing, hidden instructions, lineage loss, deletion failure, and retrieval manipulation.

Inference and interaction

Consider instruction manipulation, sensitive-data extraction, ungrounded authoritative output, unsafe content, model denial of service, routing bypass, context leakage, and output used outside its intended purpose.

Tools and agents

Consider excessive authority, confused delegation, tool-description manipulation, parameter tampering, approval bypass, replay, unsafe memory, and actions that lack a verifiable receipt.

Operations and evidence

Consider privileged misuse, monitoring gaps, alert exhaustion, capacity starvation, evidence tampering, clock or identity ambiguity, backup compromise, and recovery processes that restore an unapproved state.

Use threat prompts without becoming checklist-bound

Prompt the review with questions about spoofing or identity misuse, tampering, repudiation or missing evidence, information disclosure, denial of service, privilege expansion, model and data manipulation, unsafe autonomy, and dependency loss.

These prompts improve coverage, but the system’s purpose and flows determine which scenarios are credible.

Connect scenarios to verification

For high-priority scenarios, define one test demonstrating the prohibited path is prevented or contained and another showing the approved path still works. Record the model, policy, data, artifact, and environment versions used in the test.

Next, use Prioritize threats and residual risk to turn the scenario set into decisions.