AI Incident

What an AI incident is

An AI incident is a failure of an AI tool in use. The site’s definition gives two examples: an assistant sending wrong information to a client, or surfacing something it should not. The point that matters most is not the failure itself, because every tool will fail sometimes. It is that who is told, and in what order, should be agreed before it happens rather than worked out under pressure.

This page uses that meaning. It does not give legal advice, and it does not describe how to investigate a failure technically. It explains what an incident is, why many businesses have no route for one, what a route needs, and where the law enters. Where it mentions a legal duty, it reports what the site’s own articles say.

What counts as an AI incident

The ownership article gives the two examples already mentioned: wrong information sent to a client, and something surfaced that should not be. The AI use policy article lists more, under the incidents section of a policy: pasting the wrong file, acting on a wrong answer, and noticing a tool behaving strangely.

Those examples differ in size, and the page does not try to rank them. What they share is that a tool in use has produced, or exposed, something that should not have gone out. A tool behaving strangely is on the list because strange behaviour is often the first sign of something larger.

An incident is not the same as a hallucination. A wrong answer that a person catches before it goes out is a catch, and the checking is working. It becomes an incident when the wrong answer reaches a client, a decision or a record.

Why incidents have no route

The ownership article says unowned AI rarely fails loudly. It drifts: output quality slips, access widens, rules go stale and incidents have no reporting route. The drift page covers the wider slide.

The article describes what the missing route looks like. An assistant sends wrong information to a client, or surfaces something it should not. Who is told first? Who decides whether it is reportable? In an unowned system, it says, that conversation starts from nothing, under pressure, while the clock runs.

The running cost article makes the same point from the cost side. It counts incident response as part of governance, alongside audit logs, access controls and compliance, and says controls like this are cheaper to design in than to bolt on after an incident. It also says that when maintenance and governance have no named owner, they become nobody’s job, and the cost returns later as an incident.

Who is told, and in what order

The route has to be agreed in advance, and it is short. The ownership article lists four things a tool’s owner does, and the fourth is knowing who to tell, and in what order, if something goes wrong. Its thirty day plan ends on the same step: in week four, agree who is told, and in what order, before anything goes wrong.

The article is firm that the owner is one person, not a team: someone who feels the consequences when the tool goes wrong, which is often the head of the function that uses it and not automatically IT. The AI ownership page covers why.

The policy article says what the written rule should contain. The incidents section names what to do when something goes wrong, says who to tell and how fast, and says clearly that reporting quickly is never punished. That last clause is not decoration, and the page returns to it below.

Who may pause the Tool

An owner who cannot stop the tool cannot contain an incident. The ownership article says the owner needs the authority to pause it, because responsibility without that power is only a title.

It adds a useful test. If nobody can be found who is willing to be accountable for a tool, that is a finding in itself, and a reason to reconsider whether the tool should be in use. An incident route that ends at a person with no power to act is a list of names, not a control.

The output-checking article describes the three safeguards around the moment of failure: review before the response goes out, correction after it, and escalation to a person when the AI is unsure. It says the first and third are the ones worth building for a small business, because correction after the fact is what you do when the first two have failed. An incident route is the part of the design that covers the failure of those two.

When an incident involves Personal Data

Some incidents involve personal data, and then the data protection rules apply. The customer data article has a direct answer for the case where someone has already pasted customer data into a tool.

Do not panic and do not hide it. Record what was pasted, into which tool and on which account type. Check whether the history can be deleted and whether the provider keeps or trains on inputs. Then decide, with whoever owns data protection in your business, whether it is a personal data breach. If it is, the article says the GDPR expects the supervisory authority to be told within 72 hours of becoming aware of it, unless the breach is unlikely to put people at risk.

The practical lesson is the same as for any incident. The clock starts when you become aware, so the route has to exist before that moment, and the person who decides has to be known.

When the EU AI Act applies to an incident

The ownership article reports that under the EU AI Act, an organisation that deploys a high-risk system, as opposed to building one, has duties of its own. It must monitor the system for anomalies, dysfunctions and unexpected performance, keep its logs for at least six months, assign competent people to oversee it, and inform the provider and the authorities if there is a serious incident. The deployer page covers the role, and the high-risk AI system page covers what counts.

The EU AI Act article lists the same duty in its own words: monitor the system and tell the provider and the authority if you believe it presents a risk or has caused a serious incident, and stop using it if needed. It also says most small business use sits at minimal risk, where drafting and summarising are typical, so this duty will not reach every business or every tool.

What each duty presumes is the useful part. The deployer page says a serious incident needs someone who knows who to tell, and that the owner has to exist before the duty applies. Without an owner, the first sign that nobody is monitoring is usually the incident.

Why a ban makes incidents worse

A route only works if people use it, and people use it only if reporting is safe. The banning article says that when use is prohibited, nobody says when it goes wrong. A person who pasted the wrong document into a personal account has every reason to stay quiet, and the business hears about it later, usually from a client.

It states the principle plainly: incident reporting only works when reporting is safe. A rule that treats all use as misconduct makes the honest report the riskiest thing an employee can do, and that is how a small mistake becomes a serious one. The shadow AI page covers the use that results.

The policy article says the same in its advice on being fair about mistakes: if people fear punishment, they hide incidents, so reward quick reporting.

What to put in the AI use Policy

The policy article gives the incident rule a place. One of the sections of a policy is incidents: what to do when something goes wrong, such as pasting the wrong file, acting on a wrong answer or noticing a tool behaving strangely. It should name who to tell and how fast, and say clearly that reporting quickly is never punished.

It also ties incidents to the review cycle. A policy should be reviewed every six months, and again whenever you approve a new tool, change what a tool is connected to or have an incident. The review date page adds one more check for the owner: if an incident happened, did the route work? The incident is a test of the route as well as of the tool.

What a route does not change

A route does not prevent incidents. It decides how quickly they are found and who acts. The checking before output goes out, covered under human in the loop, is what prevents most of them from reaching anyone.

It does not replace the record. The audit trail page defines the record of each run, which is what lets a single run be reconstructed after the fact. When an incident happens, that record is how you find out what the tool did, what was approved and what changed downstream.

And it does not last without an owner. A route that nobody maintains goes stale like any other written rule, which is why the ownership article treats the route as part of the owner’s regular job.

A worked example

An illustration, not a case study. A small business uses an assistant to draft customer replies, and a person releases each one. One afternoon, a customer writes back to say a delivery date in an earlier reply was wrong.

The person who notices tells the tool’s named owner, as the route says. The owner has the authority to pause the assistant and does so while the cause is found. They check the record of each run to see which other replies used the same wrong information. They correct the customers affected. They decide whether any personal data was involved, and, because none was, there is nothing to notify.

The owner then works out why it happened. The approved answers contained an old delivery window, so the fix is to the source material, not to the person who released the reply. The person who reported it is thanked, because the policy says reporting quickly is never punished. The owner notes the incident and brings the next review of the policy forward, as the policy says to do after any incident.

Mix-ups worth avoiding

Four confusions come up often enough to name. The first is treating an incident as a failure of the tool alone, when the missing route and the missing owner are usually part of it. The second is working out who to tell after it happens, which the ownership article says starts from nothing and under pressure. The third is punishing the person who reports, which teaches everyone to stay quiet. The fourth is assuming a legal duty always applies or never applies, when it depends on the system and the data, and the person who owns data protection should decide.

How we use it

The ownership article’s thirty day plan ends by setting the route: agree who is told, and in what order, before anything goes wrong. The all-staff briefing on responsible and compliant use covers how to report an incident, and the banning article explains why reporting only works when it is safe.

If you want to know who would be told in your business today, and what they could do about it, a discovery call is a quick way to start.

Sources

• What happens when nobody owns the AI you’ve rolled out: the examples of an incident, the missing route, the deployer duties including informing the provider and the authorities of a serious incident, the owner’s four jobs and authority to pause the tool, and the week four step of setting the route.
• How to write an AI use policy for your team: the incidents section of a policy, naming who to tell and how fast, reporting never being punished, and review after any incident.
• What the EU AI Act requires of a small business: the deployer duty to monitor and to tell the provider and the authority of a serious incident, and most small business use sitting at minimal risk.
• Customer data and AI tools: what is safe to paste?: what to do if customer data was already pasted, and the 72 hour notification of a personal data breach.
• Why banning AI tools at work does not work: incident reporting only working when reporting is safe.
• How much does an AI workflow actually cost to run?: incident response as part of governance, and cheaper to design in than to bolt on.
• How to check AI output before it goes to a customer: review before, correction after and escalation.
• AI responsible and compliant use: the all-staff briefing that covers how to report an incident.
• AI ownership, Deployer, High-risk AI system, Drift, Audit trail, Review date, Data protection, Shadow AI and Human in the loop: the neighbouring terms this page links to.