What an AI security assessment covers
An AI security assessment evaluates how an AI system could be attacked or misused and whether its controls are adequate. It differs from a broader AI risk assessment (which also covers bias, legality and business impact) by focusing on confidentiality, integrity and availability of the AI system, its data and the systems it can reach.
The scope is wider than the model. A typical LLM application includes the model (often a third-party API), system prompts, retrieval data and vector stores, tools and plugins the model can call, user interfaces, logs, and the identity and permissions under which it runs.
For AI risk assessment and governance under NIST AI RMF and ISO/IEC 42001, see AI Risk Assessment and AI GRC: Frameworks, Laws and Controls for 2026.
Why demand is rising
Recent breach data shows why AI systems now need their own security assessment:
- IBM’s 2026 Cost of a Data Breach Report found AI-related breaches made up 21% of breaches, up from 13% a year earlier, and 92% of organizations with an AI-related incident lacked proper AI access controls, according to Baker Donelson’s summary.
- The same summary reports shadow AI incidents involving employees rose to 43% from 20%, averaging $5.39 million in cost.
- Verizon’s 2026 DBIR found employee use of unapproved AI tools rose from 15% to 45% in a year.
Frameworks to use
| Framework | What it gives you |
|---|---|
| OWASP Top 10 for LLM Applications 2025 | The most cited list of LLM application risks (below) |
| NIST AI 100-2 E2025 (March 24, 2025) | A taxonomy and terminology of adversarial machine learning attacks and mitigations, including poisoning, evasion, privacy and abuse, with attention to large language models |
| NIST IR 8596 preliminary draft, Cybersecurity Framework Profile for AI (December 16, 2025) | Maps AI to CSF 2.0 across three focus areas: securing AI systems, AI-enabled cyber defense, and thwarting AI-enabled attacks. NIST planned an initial public draft in 2026. |
| NIST AI 600-1 Generative AI Profile (July 26, 2024) | Generative AI risks and suggested actions under the AI RMF |
OWASP Top 10 for LLM Applications 2025
| ID | Risk | What to check |
|---|---|---|
| LLM01 | Prompt Injection | Direct and indirect injection via user input, documents and web content |
| LLM02 | Sensitive Information Disclosure | Data leaking through outputs, logs or training |
| LLM03 | Supply Chain | Third-party models, datasets, plugins and libraries |
| LLM04 | Data and Model Poisoning | Integrity of training, fine-tuning and retrieval data |
| LLM05 | Improper Output Handling | Model output passed unvalidated to code, queries or browsers |
| LLM06 | Excessive Agency | Tools and permissions beyond what the task needs |
| LLM07 | System Prompt Leakage | Secrets or logic exposed through system prompts |
| LLM08 | Vector and Embedding Weaknesses | Access control and poisoning in retrieval stores |
| LLM09 | Misinformation | Confident but wrong outputs relied on by users |
| LLM10 | Unbounded Consumption | Cost and denial-of-service through excessive use |
Running the assessment
- Inventory AI systems, including embedded AI in vendor products and employee tools.
- Map the architecture: data flows, trust boundaries, identities, tools the model can call.
- Threat model each system against OWASP LLM risks and the NIST adversarial ML taxonomy.
- Review controls: access control, input and output filtering, least-privilege tool access, data classification for retrieval sources, logging and monitoring, rate limits, human approval for high-impact actions.
- Test: prompt injection (including indirect, through documents), data leakage, jailbreaks, tool misuse and output handling. Red-team high-risk systems.
- Review third parties: model provider terms, data retention, training use, security reports.
- Rate and remediate findings; retest.
- Monitor for drift, new attack techniques and model or vendor changes.
Controls 2026 data says are missing
| Gap | Control |
|---|---|
| No AI access controls (92% of affected organizations, per IBM 2026 summary) | Identity-based access to AI tools and data; least privilege for agents and tools |
| Shadow AI | Approved tool list, monitoring, and a fast path to approve useful tools |
| No AI governance policy (68%, per the same summary) | Policy covering acceptable use, data handling and review |
| Indirect prompt injection through documents | Treat retrieved content as untrusted; isolate tool calls; require approval for actions |
Common weaknesses
- Assessing the model but not the tools, data and permissions around it
- No testing of indirect prompt injection through documents or web pages
- Agents given broad permissions “for convenience”
- Vendor AI features never assessed
- One-off testing before launch, nothing after
AI security assessment checklist
- AI inventory includes vendor and employee tools
- Architecture, data flows and trust boundaries mapped
- Threat model covers OWASP LLM Top 10 2025
- Access control and least privilege for AI and agents
- Input, retrieval and output handling controls in place
- Prompt injection, leakage and tool misuse tested
- Model and AI vendor terms reviewed
- Logging, monitoring and retesting scheduled
Review AI security evidence with AskDona
AI security assessments generate a lot of documentation: architecture records, policies, vendor terms, test reports and remediation evidence. AskDona reads them against the controls you define, flags gaps such as missing access control evidence, untested OWASP risk categories or vendor terms that allow training on your data, and links each finding to its source page. Run a sample AI security evidence review on one AI application. AskDona shows the evidence; your security team decides.
FAQ
What is an AI security assessment? An evaluation of how an AI system and its surrounding data, tools and permissions could be attacked or misused, and whether controls are adequate.
How is it different from an AI risk assessment? A risk assessment also covers bias, legality and business impact. A security assessment focuses on attacks, misuse and security controls.
What is the OWASP Top 10 for LLM Applications? A list of the most critical LLM application security risks. The 2025 edition starts with prompt injection and sensitive information disclosure.
What is prompt injection? Input that manipulates a model into ignoring instructions or taking unintended actions, either directly from a user or indirectly through content the model reads.
How often should AI systems be security assessed? Before launch, after significant changes to models, data or tools, and periodically as attack techniques evolve.
References
All pages opened and checked by October 14, 2026.
| Source | Type | Used for |
|---|---|---|
| OWASP, Top 10 for LLM Applications 2025 | Industry project | Risk list |
| NIST AI 100-2 E2025 | US standards publication | Adversarial ML taxonomy |
| NIST, Draft Cyber AI Profile (Dec 2025) | US standards agency | NIST IR 8596 draft |
| NIST AI Risk Management Framework | US standards agency | AI 600-1 |
| Baker Donelson, IBM 2026 Cost of a Data Breach takeaways | Law firm summary | AI breach statistics |
| Verizon, 2026 DBIR release (May 19, 2026) | Industry report | Shadow AI |