AI monitoring: how to govern AI systems once they are live

An enterprise customer's AI questionnaire has one question your team can't answer cleanly: how do you know your AI still works as intended after go-live? The model passed testing months ago. Since then, the data has moved, the vendor has shipped two updates, and nobody owns the answer. AI monitoring is how you answer that question with evidence.

This guide is for heads of security, compliance leads and CTOs at UK tech firms of 25 to 250 people who already use or embed AI. It treats AI monitoring as a governance control, not a telemetry stack. You will leave with a monitoring plan you can run per AI system, the signals worth watching, and where the evidence should land.

What is AI monitoring?

AI monitoring is the ongoing check that an AI system in use still performs, behaves and is used as approved, and that problems reach an owner who acts on them. Technical observability is one input, not the whole control. The governance record of what you watched, found and did is what customers and auditors ask for.

Most pages that rank for the term describe dashboards and telemetry. Those matter, but they answer a narrower question. The table shows the difference.

A dashboard that nobody is accountable for is not a control.

If you already run continuous control monitoring for security controls, AI monitoring fits the same pattern. The difference is in what you watch.

What should AI monitoring cover?

AI monitoring should cover four groups of signal: accuracy and model drift, use outside the approved purpose, incidents and the human signals around them, and changes the vendor makes to AI you buy or embed. Each AI system needs all four considered, even if the plan for some is a light quarterly check.

Accuracy and model drift

Watch whether the outputs are still as good as when you approved the system. For a classifier, that might be error rates against a sample checked by a person. For a generative feature, it might be a periodic review of a fixed set of test prompts. The point is a baseline, and a check against it.

Use outside the approved purpose

Watch for the system being used for things nobody assessed. A summarisation tool approved for internal notes that starts drafting customer advice is a different risk. Scope creep rarely announces itself, so look for it on purpose. Our guide to shadow AI governance covers the related problem of tools nobody approved at all.

Incidents, complaints and human overrides

Watch what people tell you. Customer complaints about AI outputs, support tickets and staff overriding the AI's suggestion are all early warnings. A rising override rate often says more than any accuracy metric, because it records people choosing not to trust the system.

Vendor changes to bought and embedded AI

Watch what your suppliers change. Many firms buy AI rather than build it, so the model can change under you without a release on your side. Track model version notices, changes to the vendor's terms on data use and training, and any new sub-processors.

Bought AI still changes. You just don't get told unless you ask.

What is model drift, in plain terms?

Model drift is the gradual decline in an AI system's accuracy because the world it works in has changed since it was trained or tested. The system has not broken. The customers, the data or the behaviour it sees have moved, so answers that were right last year are quietly less right now.

The ICO's guidance on AI and accuracy explains that a system accurate on an existing population "may not continue to perform well" if that population changes. It says you "should regularly assess drift and retrain the model on new data where necessary".

For a governance lead, the practical question is the one NIST's playbook asks: "How much distributional shift or model drift from baseline performance is acceptable?" That answer is your threshold. Without it, drift is something you notice, not something you manage.

How do you build an AI monitoring plan?

Build one AI monitoring plan per AI system, starting from your AI register and tiered by risk. For each system, set what you watch, how often, who reviews it, the threshold that triggers action, the action itself, and where the evidence is kept. Higher-risk systems get tighter cadences and named senior reviewers.

The ICO puts the principle simply: after deployment, monitoring frequency "should be proportional to the impact an incorrect output may have on individuals". We recommend three tiers.

Then work through each system in five steps.

1. Start from the register. Every AI system in your inventory gets a plan, including bought tools and AI features inside other software. A system missing from the register can't be monitored.
2. Set the tier. Use the system's risk assessment, not its technical complexity. A simple model that scores customers can sit above a complex internal assistant.
3. Fill in each signal. For each of the four signal groups, write what you will check, how often, and the threshold that triggers action. "Overrides above the level agreed at approval, two months running" is a threshold. "Monitor overrides" is not.
4. Name the owner and the action. Each threshold needs a person and a decision: investigate, restrict use, suspend, or escalate to the vendor.
5. Decide where the evidence lands. Pick one place for review records and keep it there. The record is the control.

An alert without a named owner is a log entry, not a control.

What does the EU AI Act require for AI monitoring?

The EU AI Act sets specific monitoring duties only for high-risk AI systems. Providers must run post-market monitoring under Article 72. Deployers must monitor in line with the instructions for use and keep logs under Article 26, which applies to Annex III high-risk systems from 2 December 2027. Most firms' AI isn't high-risk.

Start with the scale. The European Commission's AI Act overview says the Act introduces no rules for minimal-risk AI, and that "the vast majority of AI systems currently used in the EU fall into this category". Monitoring is still good governance for those systems; it just isn't an AI Act duty.

For high-risk systems, the duties split by role. A provider develops the system and places it on the market under its own name. A deployer uses it under its own authority. A SaaS firm can be both: provider of an AI feature it sells, deployer of AI it buys.

Regulation (EU) 2026/1744, the Digital Omnibus on AI, moved the date for the high-risk rules in Chapter III. Those rules include Article 26 and the rules that decide whether a system is high-risk. They now apply from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems. Articles 72 and 73 sit in a different chapter, which keeps the Act's general date of 2 August 2026, but they only cover systems classified as high-risk. The Omnibus also gave providers more flexibility: the Commission will publish guidance and a template for the post-market monitoring plan by 2 September 2027, rather than imposing one by implementing act.

One detail matters for financial services. Under Article 26(5), a deployer that is a financial institution meets the monitoring duty by complying with the internal governance rules of EU financial services law. See our EU AI Act guide for how the Act classifies systems.

How do ISO/IEC 42001, NIST AI RMF and the UK code treat AI monitoring?

ISO/IEC 42001:2023, the NIST AI RMF and the UK AI Cyber Security Code of Practice all expect AI systems to be monitored once live. ISO/IEC 42001 makes it auditable through clause 9.1 and an Annex A control on operation and monitoring. NIST frames it as part of MEASURE and MANAGE. The UK code makes it a security principle.

ISO/IEC 42001:2023. Clause 9.1 expects you to plan your monitoring and measurement, evaluate what it finds, and keep evidence of the results. Annex A adds a control on AI system operation and monitoring (A.6.2.6). One plan per system, with its review records, is a direct answer to both. Our ISO/IEC 42001 guide covers the wider management system.

NIST AI RMF. The playbook's MEASURE 2.4 says the functionality and behaviour of the AI system "are monitored when in production". MANAGE 4.1 expects "post-deployment AI system monitoring plans" covering user input, appeal and override, decommissioning, incident response, recovery and change management. MANAGE 3.1 adds that risks from third-party resources "are regularly monitored". See our NIST AI RMF overview.

UK AI Cyber Security Code of Practice. Principle 12 of the voluntary Code of Practice for the Cyber Security of AI is "Monitor your system's behaviour". It says system operators should analyse logs so that models "continue to produce desired outputs", and watch for changes over time such as data drift.

The frameworks agree on the substance. One plan per system, reviewed and recorded, serves all three.

Where should AI monitoring results go?

AI monitoring results should feed three places you already run: the risk register, management review and your customer evidence. A threshold breach updates the system's risk entry. A summary of each period's findings goes to management review. The review records become the answer to customer questionnaires about AI.

• Risk register. Each AI system's risk entry should cite its latest monitoring result. A breach that changes the likelihood or impact of a risk changes the entry.
• Management review. ISO/IEC 42001 expects leadership to see how the AI management system performs. A one-page summary per tier, with breaches and actions, is enough.
• Customer questionnaires. When a customer asks how you know your AI still works as intended, you can describe the plan, the cadence and the last review date. That answer carries more weight than a list of tools.

Monitoring that doesn't change a decision somewhere is just collecting data.

What are the most common AI monitoring mistakes?

The most common AI monitoring mistakes are treating observability dashboards as the whole control, monitoring only the AI you build while ignoring the AI you buy, and setting no thresholds, so nothing ever triggers action. Each one leaves you with data but no evidence that anyone governed the system.

• Dashboards with no owner. Metrics exist, but nobody reviews them on a schedule or records a decision.
• Built AI only. Bought tools and embedded features change too, often without notice.
• No thresholds. Without a defined level of acceptable drift or overrides, every review ends with "looks fine".
• Monitoring as an engineering task only. The engineers see the signal; the risk owner needs the decision.
• Assuming the EU AI Act applies to everything. It sets monitoring duties for high-risk systems. Check your systems' classification before you build a compliance programme around Article 72.

Frequently asked questions