Government examines stronger reporting safeguards for serious AI incidents

The UK government is examining stronger safeguards for serious incidents involving artificial intelligence, including how significant failures and near misses might be identified and reported, as increasingly autonomous systems are deployed across public services and the wider economy.

Measures announced on 12 October include work on national preparations for incidents involving AI agents and the wider use of advanced technology to improve government cyber defences. Ministers are considering how the Cyber Security and Resilience Bill regime could address emerging risks, but have not announced that new AI incident-reporting duties are already in force.

The programme builds on pilots that the government says have helped identify and address more than 1,000 potential vulnerabilities in public-sector systems. The reported number reflects results from those pilots and does not establish that the same approach would deliver identical outcomes across every department or private organisation.

AI Minister Kanishka Narayan set out the government’s approach during an event at Imperial College London’s science and technology incubator. The central concern is that the capability of the latest systems to perform more complex tasks may increase the potential impact of errors, misuse or the compromise of connected software.

An AI agent differs from a conventional question-and-answer tool when it is able to access applications, carry out instructions and take action within a digital environment. That capability can improve productivity, but it changes the risks faced by an organisation if permissions are excessive, oversight is inadequate or an agent behaves unexpectedly.

Failures can have markedly different consequences. A model producing a poor answer is not the same as software making an unauthorised change to a business system or triggering a harmful transaction. The appropriate controls depend on what the technology can access, what it can alter and how easily an organisation can detect and reverse an action.

Incident reporting could help authorities identify emerging patterns across organisations and improve responses to serious events. Effective reporting frameworks need clear thresholds, proportionate requirements and rules for handling sensitive information. Excessively broad definitions can result in large volumes of minor events obscuring those that demand urgent attention.

The government has indicated that existing cyber security structures may form part of the response. That approach recognises that AI systems are often integrated into conventional networks, identity management processes, cloud services and software supply chains. Good security controls can still reduce exposure even when the underlying technology is new.

Organisations deploying AI agents need an inventory of where those systems operate, what access they hold and which actions they can execute without a person approving each step. External model providers may change software or operating conditions, making internal monitoring, incident support and recovery arrangements relevant even when contracts contain security assurances.

Testing is another part of the programme. The AI Security Institute is continuing evaluations of advanced models, while the government considers how to prepare for failures involving greater autonomy. Such testing can reveal weaknesses before deployment, but it cannot guarantee that every use case or combination with other software will be safe.

The government will have to determine which incidents merit formal reporting, how serious harm is defined and how notifications interact with existing cyber security procedures. A narrowly defined duty could concentrate attention on incidents involving sensitive services or significant disruption, while a wider requirement would entail more reporting by organisations already managing routine software errors.

Any new statutory reporting or security duties would require further policy development and, where necessary, legislation. Existing cyber controls still provide a basis for limiting access, recording system activity and testing recovery. The government’s proposals remain at an exploratory stage, rather than imposing an immediate new reporting obligation on companies.