AI incidents rarely come with a warning. An incorrect output reaches a customer. An automated system makes a decision that no one reviewed. An AI tool processes data it should never have received.
When it happens, the question is always the same: what do we do now?
The difference between a manageable incident and a full-scale crisis is rarely the severity of the initial problem. It is whether the organisation had a response protocol before the incident occurred.
Without a protocol, the response is improvised.
With a protocol, the damage can be contained.
What Is an AI Incident?
An AI incident is not necessarily a technical failure. It is any situation in which the use of an AI system produces an unintended outcome with real-world consequences.
It may be a recruitment algorithm that unintentionally filters candidates using discriminatory criteria.
It may be a generative AI tool producing a report with incorrect information that is sent to a client without verification.
It may be an employee using ChatGPT with confidential contractual information without organisational approval.
It may be a remote proctoring system incorrectly flagging a student for suspicious behaviour.
Or it may be a chatbot providing inaccurate information about a product with legal consequences.
In all these cases, the AI system may be functioning exactly as designed.
The incident is not technical. It is a failure of judgement, oversight or governance.
Phase 1: Immediate Containment
During the first hours, the priority is simple: stop the impact from spreading.
This may involve pausing the affected AI system, removing the incorrect output, temporarily suspending access to the tool while the situation is assessed, or contacting the provider if the issue originates from their service.
The most important principle at this stage is to communicate externally only what is strictly necessary.
Premature communication based on incomplete information is often more damaging than a short period of informed silence.
Phase 2: Assess the Scope
Once the immediate risk has been contained, the organisation must understand exactly what happened.
How many people or business processes have been affected?
Has personal data been processed unlawfully?
Is there a legal obligation to notify a supervisory authority?
Is this an isolated event or evidence of a systemic issue?
The assessment should identify which AI system caused the incident, how long it had been occurring, who was affected, whether personal data was involved, and whether existing oversight mechanisms should have detected the problem earlier.
Phase 3: Response and Communication
Once the scope is understood, the organisation can communicate responsibly.
The principles are the same as in any crisis: communicate quickly, clearly and honestly about what is known, while being transparent about what is still under investigation.
Avoid speculation until the facts have been confirmed.
If customers, employees, candidates or students have been affected, informing them becomes a priority.
Where legal notification requirements apply, they must be respected. Under the GDPR, certain personal data breaches must be reported within 72 hours. Additional obligations may also arise under the AI Act, depending on the circumstances.
Phase 4: Review and Prevention
Once the incident has been resolved, the organisation must answer a difficult question:
Why was it not detected sooner?
An honest answer to that question is the foundation for preventing similar incidents in the future.
Every post-incident review should produce at least three outcomes:
- An update to the organisation’s AI policy to address the weakness that allowed the incident.
- Stronger human oversight mechanisms for the affected AI system.
- Internal communication explaining what happened, what has changed and why.
What Distinguishes Organisations That Manage AI Incidents Well?
- They maintain an inventory of the AI systems they use and clearly identify who is responsible for each one.
- They have a defined escalation process, so responsibilities are clear before an incident occurs.
- They train their people, ensuring that employees know how to recognise an AI incident and understand how to report it.
- And they maintain an up-to-date AI policy that provides a clear framework for determining whether an incident represents a breach of internal rules or external obligations.
Organisations without these foundations usually respond more slowly, with greater uncertainty and significantly higher consequences.
Frequently Asked Questions
What is an AI incident in an organisation?
An AI incident is any situation in which the use of an AI system produces an unintended outcome with real-world consequences. This may include errors in automated decision-making, personal data exposure, incorrect outputs sent to customers, or breaches of the organisation’s AI policy.
Should an AI incident be reported to a supervisory authority?
It depends on the nature of the incident. If personal data is involved, the organisation may be required to notify the relevant data protection authority within 72 hours under the GDPR.
If the incident concerns a high-risk AI system under the AI Act, additional notification obligations may apply.
When in doubt, organisations should seek specialist legal advice.
What is the difference between an AI incident and a cybersecurity incident?
A cybersecurity incident involves the compromise or failure of an information system.
An AI incident can occur even when the technology functions exactly as intended. The system works correctly, but produces an unacceptable outcome because of inadequate judgement, human oversight or organisational governance.
