--- title: "AI security risks: what your governance has to cover" canonical: https://www.grcmana.io/blog/ai-security-risks published: 2023-10-11 updated: 2026-10-22 authors: ["Harry West"] --- # AI security risks: what your governance has to cover ## Key takeaways AI security risks are the ways an AI system can be attacked or misused to leak data, take unauthorised actions or produce manipulated output. For a UK B2B tech firm the main seven are prompt injection, sensitive data in prompts, poisoning, over-permissioned agents, unchecked output, third-party model risk and model theft. Each needs an owner, a control and evidence, anchored in the UK AI Cyber Security Code of Practice. - **UK AI Cyber Security Code of Practice** — A voluntary code published by DSIT in January 2025 that sets baseline cyber security principles for AI systems across the lifecycle, from secure design to secure end of life. - **Agentic AI as delegated access** — NCSC guidance says every AI agent should have its own unique identity, only the permissions its task needs, short-lived credentials, and activity treated as user activity in security monitoring. - **Data poisoning** — An attack that introduces malicious data into training datasets to compromise an AI system's behaviour; the UK Code warns it may be found only after training, if at all. - **EU AI Act Article 15** — Requires high-risk AI systems to achieve appropriate accuracy, robustness and cybersecurity, with measures against AI-specific attacks such as data poisoning, model poisoning, adversarial examples and confidentiality attacks. AI security risks are the ways an AI system can be attacked or misused to leak data, take unauthorised actions or produce manipulated output. The main ones are prompt injection, sensitive-data exposure, over-permissioned agents and third-party model risk. Each needs a named owner, a control and evidence, like any other security risk. An enterprise prospect's security questionnaire now has an AI section. It asks how you stop prompt injection, where customer data goes when it reaches a model, and what your AI agents are allowed to do. Your product shipped an AI feature last quarter, and nobody owns those answers yet. This list sets out seven AI security risks for a UK B2B tech firm that uses AI or ships AI features. Each item passed one test: the UK government's AI Cyber Security Code of Practice or NCSC guidance names it, the OWASP Top 10 for LLM Applications lists it, and a firm of 25 to 250 people can answer it with a governance control. Societal risks and model-engineering fixes are left out. ## What are the main AI security risks? The main AI security risks for a UK B2B tech firm are prompt injection, sensitive data in prompts and outputs, data and model poisoning, over-permissioned AI agents, unchecked model output, third-party model risk, and model or IP theft. The table maps each one to the governance control that answers it and the framework that asks for it. | Risk | How it happens | Governance control | Framework reference | | --- | --- | --- | --- | | 1. Prompt injection | Instructions hidden in user input or fetched content redirect the model | Treat model input as untrusted; limit what the model can reach | UK Code principles 2 and 3; OWASP LLM01 | | 2. Sensitive data in prompts and outputs | Staff or customers paste personal or confidential data; the model repeats it | Data classification rules for AI; supplier terms on retention and training | UK Code 5.4 and 10.1; OWASP LLM02 | | 3. Data and model poisoning | Tampered training, fine-tuning or retrieval data changes model behaviour | Documented data sources and change control | UK Code 8.2; EU AI Act Article 15(5); OWASP LLM04 | | 4. Over-permissioned AI agents | An agent acts with more access than its task needs | Agents as identities with least privilege, logging and a kill switch | NCSC agentic AI guidance; UK Code 2.6; OWASP LLM06 | | 5. Unchecked model output | Generated text, code or commands flow into other systems unvalidated | Defined output paths and human approval for high-impact actions | UK Code principle 4; OWASP LLM05 | | 6. Third-party model and supply-chain risk | A model, plugin or provider you rely on is weak or changes without notice | AI-specific supplier due diligence and re-evaluation on model change | UK Code 2.4 and 7.3; OWASP LLM03 | | 7. Model and IP theft | Repeated queries extract model behaviour, weights or training data | Access and rate controls on AI interfaces; protection of weights and data | UK Code 6.2 and 5.4.2 | The [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/) is the practitioner reference behind this list. Its 2025 edition names each risk in more technical depth; we cite its numbering rather than repeat it. The [AI Cyber Security Code of Practice](https://www.gov.uk/government/publications/ai-cyber-security-code-of-practice), published by DSIT on 31 January 2025, is the UK anchor: a voluntary baseline of 13 principles across the AI lifecycle. ## Which AI security risks come in through inputs and data? Three AI security risks arrive through what goes into a model: prompt injection, sensitive data in prompts and outputs, and poisoned training or retrieval data. They share one governance lesson. A model treats instructions and data as the same stream of text, so every input source needs an owner and a trust decision before the feature ships. ### 1. Prompt injection **What it is.** Prompt injection is input crafted to override the model's instructions. It can be typed by a user or hidden in a document, email or web page the model reads, which the Code calls indirect prompt injection and lists among the ways AI differs from ordinary software. **When it matters.** Whenever your AI feature reads content you do not control, such as customer uploads, inbound email or web search results, and can then take an action or reveal data. **What to do.** Record each AI feature's input sources in your risk register, with an owner and a risk rating. Decide what the model may reach if an injection succeeds, and keep that small. The NCSC's advice on agents applies here too: "do not rely on prompting alone." ### 2. Sensitive data in prompts and outputs **What it is.** Personal or confidential data that staff or customers put into prompts, and that the model or its provider then stores, logs, trains on or repeats. OWASP lists sensitive information disclosure as its second risk. **When it matters.** Whenever personal data reaches a model. The Code reminds firms that personal data, "including pseudonymized data", may bring data protection obligations under ICO guidance. **What to do.** Extend your data classification policy to say which classes of data may enter which AI tools. Check each provider's terms on retention and training. The Code also requires System Operators to tell users "where and how their data will be used, accessed and stored", so update your privacy notice and product documentation to match. Unsanctioned staff use of AI tools is its own problem, known as shadow AI, and needs a separate policy. ### 3. Data and model poisoning **What it is.** The Code defines data poisoning as "a type of adversarial attack where malicious data is introduced into training datasets to compromise the AI system's performance or behaviour." The same risk applies to fine-tuning data and to the documents a retrieval feature searches. **When it matters.** When you fine-tune a model, train your own, or let a feature answer from a knowledge base that others can edit. The Code warns that publicly sourced training data may already be poisoned, and that discovery "is likely to occur after training (if at all)". **What to do.** Code provision 8.2 requires developers to document how they obtained public training data. Keep the same record for fine-tuning and retrieval data: where it came from, and when. Put the knowledge base under the same change control as production code. You cannot audit a model's behaviour if you cannot say what it learned from. ## Why do AI agents need governing as delegated access? AI agents need governing as delegated access because they act with real credentials on real systems. An agent that can send email, change records or run commands holds a user's power without a user's judgement. The NCSC's agentic AI guidance treats agents like accounts: their own identity, least privilege, logging and a way to stop them at once. ### 4. Over-permissioned AI agents **What it is.** An AI agent or assistant given more access than its task needs, so that a mistake or a successful injection reaches further than it should. OWASP calls this excessive agency. **When it matters.** As soon as an agent can act rather than only suggest. The NCSC's [guidance on managing the cyber risk of agentic AI](https://www.ncsc.gov.uk/blogs/managing-the-cyber-risk-of-agentic-ai) says the greater an agent's autonomy, the greater the potential impact if it "takes actions outside its intended scope". **What to do.** Give each agent "its own unique identity in a class which differentiates them from human or individual systems", as the NCSC puts it, with only the permissions its task needs and short-lived credentials. Log its activity into your security monitoring, because the NCSC says agentic activity "should be treated as a form of user activity". Name an owner, and keep the ability to stop the agent immediately. An agent with an admin token is an admin, whatever the job description says. ### 5. Unchecked model output **What it is.** Model output that flows into another system, such as a database query, a customer email or a deployment step, without being checked first. OWASP lists this as improper output handling. **When it matters.** When output is acted on automatically, or when a person is likely to accept it without review because it looks authoritative. **What to do.** Map where each feature's output goes, and treat output headed for another system as untrusted input to that system. Set the actions that need human approval before they run. The Code's principle 4 asks firms to build in human oversight, and asks developers to make outputs easy for people to assess. ## What AI security risks sit in your supply chain? AI security risks in your supply chain sit with the models, plugins and data sources you build on but did not create. When your product runs on a provider's model, they are often the largest. The Code makes them yours to assess: for any external component, you run an AI security risk assessment and keep the evidence. ### 6. Third-party model and supply-chain risk **What it is.** Weakness or unannounced change in a model, model provider, plugin or library your product depends on. OWASP ranks supply chain third in its list. **When it matters.** Every time you adopt a new model or provider, and every time a provider updates the model behind an API you already use. **What to do.** Add AI-specific questions to your supplier due diligence, building on the supplier process you already run under [ISO 27001 Annex A 5.19](https://www.grcmana.io/vault/iso-27001/annex-a-5-19-information-security-in-supplier-relationships). The Code requires an "AI security risk assessment and due diligence process" for external components, and asks firms to "re-run evaluations on released models that they intend on using". Record which model version each feature runs on. ### 7. Model and IP theft **What it is.** Attempts to copy a model's behaviour, steal its weights, or infer details of its training data by querying it. The Code names model inversion and membership inference among the AI-specific attacks threat modelling should cover. **When it matters.** When you have fine-tuned or trained a model that carries commercial value, or trained it on customer or personal data, and expose it through an API. **What to do.** Treat model weights and training data as confidential assets in your asset inventory. Where you offer an API, the Code expects controls such as "placing limits on model access rate to limit an attacker's ability to reverse engineer" the model. ## How do frameworks answer these AI security risks? Four frameworks answer these AI security risks for a UK firm. The UK Code sets the security baseline. ISO/IEC 42001 provides a certifiable AI management system. NCSC guidance covers secure development and agents. EU AI Act Article 15 adds legal duties, but only for high-risk systems on the EU market. We recommend starting with the Code and ISO 42001. The [NCSC's guidelines for secure AI system development](https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development) cover four stages: design, development, deployment, and operation and maintenance. [ISO 42001](https://www.grcmana.io/vault/iso-42001) turns the same risks into a management system that an accredited certification body can certify. It shares its management-system structure with ISO 27001, so the risk process will feel familiar. The [NIST AI RMF](https://www.grcmana.io/vault/nist-ai-rmf) is a useful secondary reference when a US customer asks for it. The [EU AI Act](https://www.grcmana.io/vault/eu-ai-act) matters when your system falls into a high-risk category and you place it on the EU market. The Code accepts that "a higher level of risk will remain in AI systems despite the application of controls". That is why each risk needs an owner, not only a control. Our article on [AI accountability](https://www.grcmana.io/blog/ai-accountability) makes the case for naming one. ## How to use this list Start with an inventory of AI in use and in your product: every model, provider, agent and AI feature, with its owner, its data and its permissions. The Code asks for "a comprehensive inventory of their assets (including their interdependencies/connectivity)". Without that list, none of the seven risks can be scoped. Then work through the table row by row against the inventory. For each AI feature, mark which risks apply, record them in your existing risk register, and give each one an owner, a control and the evidence that shows the control works. In most firms items 1, 4 and 6 apply, because most firms read untrusted content, connect agents to tools and rely on a provider's model. Last, decide your role under the Code. A software vendor that offers AI services to customers is likely to be a Developer; a firm that only uses third-party AI is likely to be a System Operator. The role tells you which provisions are yours to evidence when the next questionnaire arrives. A risk with no owner is a risk nobody is managing. ## Frequently asked questions ### Is the UK AI Cyber Security Code of Practice mandatory? No. DSIT published it as a voluntary code, using "shall" for its requirements and "should" for recommendations. It is still the UK government's baseline for securing AI. DSIT said the Code would form the basis of an ETSI standard, and the NCSC's agentic AI guidance now points to ETSI EN 304 223, the baseline cyber security requirements for AI models and systems. ### Does EU AI Act Article 15 apply to a UK SaaS firm? Only if you provide a high-risk AI system on the EU market. Article 15 requires high-risk systems to achieve appropriate accuracy, robustness and cybersecurity, with measures against attacks such as data poisoning, model poisoning and adversarial examples. Most AI features in B2B SaaS are not high-risk, so check the classification first. ### Which AI security risk should a small firm tackle first? Start with the risks your AI inventory shows apply, then rank them by impact. Over-permissioned agents and sensitive data in prompts usually rank highest, because both turn a single mistake into a data breach or an unauthorised action. Both are answered by controls you already run: access control, data classification and supplier due diligence, extended to cover AI. ### How do AI security risks fit with ISO 27001? They sit inside it. AI features are information assets, AI providers are suppliers and AI agents are identities, so most controls extend existing ISO 27001 practice. ISO 42001 adds what ISO 27001 does not cover: AI-specific risk and impact assessment, and governance of how AI systems are designed and used.