Why more AI near-miss reports can signal better monitoring

More reported AI near misses can reflect better monitoring

A financial firm that reports more AI near misses may have better monitoring than one that reports fewer. For executives assessing AI safety, disclosure totals can therefore mislead: they count detected events, while another firm may have missed similar activity. The UK Financial Conduct Authority’s call for reporting makes detection quality important when interpreting those totals.

Jessica Rusu, the FCA’s Chief Data Officer, called for reporting to cover significant AI failures and near misses in a Monday speech to a Bloomberg UK Policy Series audience. “Our rules already require firms to report major incidents, and that must extend to significant AI failures and near-misses. As the AI Minister has recognized, near-miss reporting for agentic AI is becoming a national security priority. It is a regulatory priority too,” she said.

Her call comes amid concern about continued deployment of generative AI, which produces content, and AI agents, which can take actions, alongside recurring reports of agent misconduct and inadequate consequences. Reporting would require firms to disclose consequential events even when safeguards prevented harm. The FCA had not answered requests to clarify its terminology by publication time, leaving firms with a classification question: what happened, how serious was it, and what stopped it?

Separate serious harm, blocked harm, and testing

The consequences offer a starting point for assessing significance. Pieter Arntz, a malware intelligence researcher at security firm Malwarebytes, gives four hypothetical financial-services examples: an AI system approving substantial fraudulent payments, releasing sensitive customer information, preventing customers from using essential financial services, or initiating transactions that threaten market integrity. Malwarebytes has a commercial stake in demand for security services, and Arntz approaches the question through the harm a system could cause.

For Arntz, those consequences determine whether a failure is significant. An incorrect answer becomes consequential through the action or outcome it produces. As he puts it, “The test should be the scale and severity of the impact, rather than whether the failure looks technically dramatic.” An ordinary-looking service interruption can therefore warrant close attention when it prevents customers from accessing essential services.

A near miss requires a further judgment: whether an event had a realistic path to serious harm before something intervened. Arntz illustrates this with an agent attempting an unauthorized transfer that a separate approval requirement blocks. The attempt exposes a problem with the agent’s behavior, while the approval requirement explains why the transaction stopped. Assessing the event requires both the attempted action and the successful intervention.

Arntz’s other hypothetical example shows how an attack can set that sequence in motion. Prompt injection uses instructions in material an AI system encounters to redirect its behavior; an attacker might use it to induce an agent to attempt to release sensitive information. An outbound security control could then prevent the information from leaving. The agent’s redirection creates the danger, and the separate control contains it.

Testing needs a separate classification because an organization deliberately seeks unsafe behavior to find weaknesses. Red teaming is adversarial testing designed to provoke such behavior, so an adverse result can mean the test has found a problem. Noah Kenney, principal consultant at Digital 520, says: “A model failing a red-team test is the test working, not a regulatory incident.” Digital 520 has a commercial interest in demand for consulting on these problems; Kenney’s interpretation and Arntz’s categories remain distinct from an official FCA definition.

Reporting totals depend on what firms can detect

These categories help explain individual events, but a reporting total also depends on whether a firm can detect them. Detection must precede classification and disclosure, so a firm’s count reflects both activity in its systems and its ability to observe that activity. Comparing totals therefore also means comparing monitoring capabilities.

Kenney illustrates the problem with two hypothetical banks. “A bank with good monitoring may catch hundreds of attempts by AI agents to act outside their authority,” he says. He contrasts this with weaker detection: “A bank with poor monitoring may report zero, because it never saw them. On paper, the safer bank looks worse,” Kenney says. The example shows how reporting can reward the appearance created by poor detection.

This makes a raw count hard to interpret. More reports could reflect more attempted misconduct, stronger detection, or both, and each explanation leads to a different judgment about a deployment. A low total offers little reassurance unless the monitoring behind it deserves confidence. The FCA’s reporting call establishes no policy of ranking firms by report volume; the risk arises if regulators or executives later use totals without examining how they were produced.

Examining an event also means separating the agent’s conduct from the controls’ response: in Arntz’s transfer example, an agent exceeded its authority and an independent approval requirement stopped execution. Combining those facts into a single total hides information that compliance teams need to explain events to regulators and executives need to compare business units. Successful containment still leaves a reason to report, because the attempt exposed a realistic potential for serious harm. Recording the attempted action, its possible consequences, and the intervention preserves that information without treating a blocked attempt as equivalent to completed harm.

Assurance must continue as models and suppliers change

A successful intervention shows how a control worked under particular conditions, and those conditions can change. Rusu called for greater investment in cybersecurity and operational resilience, the ability to withstand disruption and keep services operating, and linked it to ongoing assurance, or repeated checks that a system remains acceptably safe. “A system that is safe at deployment may behave differently at scale, under stress, or as the models beneath it change. Agentic systems therefore require continuous, conditional assurance,” she said.

Ongoing checks also require clarity about who or what can act. Rusu said the FCA would work with industry, other regulators, and frontier labs developing advanced AI models on a framework for agentic finance. Its proposed scope includes agent identity, delegated authority, accountability, and execution: which agent is acting, what permission it has received, who answers for its behavior, and how its actions are carried out.

Near-miss reporting would require firms to recognize and explain breakdowns in those arrangements before harm occurs. Frank Dickson, principal analyst at Dickson Research, sees this as the speech’s most important proposal and the area where financial firms are least prepared. “I love the near-miss idea. It is the most important thing in the speech. It is the part financial firms are least ready for,” he says. As a commercial research business, Dickson Research has an interest in demand for expert analysis of these risks.

Dickson’s examples extend the discussion from agents’ actions to the external services they depend on. He points to reliance on a single model and describes a hypothetical outage at one AI supplier that disables a customer-service channel. Customers can lose access because a supplier becomes unavailable, even when the firm’s agent has made no unauthorized decision. Supplier concentration therefore belongs in an assessment based on consequences.

A supplier can also alter behavior while its service remains available. A provider may update a model without informing every customer, leaving firms with behavior that differs from the version they tested. Dickson illustrates the risk this way: “A model can pass testing in March and misbehave in May without anyone at the firm changing a line of code, because the provider changed the model underneath.”

The March-to-May example shows why the value of initial testing depends on what changes afterward. A firm’s service relies on behavior supplied by another organization, so unchanged code on the customer’s side can still produce different results. Dickson’s prescription is direct: “A model version change should get the same scrutiny as any other production change.” That scrutiny lets firms revisit earlier safety judgments when the conditions supporting them change.

Enforcement must give firms a reason to act

Ongoing assurance requires information, but firms also need a compelling reason to act on it. Nader Henein, a UK-based VP analyst at Gartner, questions how much additional effect the FCA proposal will have because many UK enterprises already face EU disclosure requirements. Those obligations are separate from the FCA proposal and its unresolved terminology. Gartner has a commercial interest in demand for research and advisory services on regulatory and technology risks, which is relevant context for Henein’s assessment.

Henein places the proposal within a wider series of warnings from financial authorities. He names the Bank of England, the UAE Central Bank, and the Monetary Authority of Singapore, and says European central banks and the European Systemic Risk Board raised the cyber-risk assessment associated with frontier models to “severe” a few months earlier. In his account, the concern was simultaneous cyber incidents whose effects could spread across the financial sector and threaten its core stability.

Those warnings explain the concern, while enforcement determines the consequences of ignoring it. “Financial institutions tend to heed the warnings of the regulator that decides whether they can continue to operate, but without clear laws, regulatory oversight, and the obligatory fines, these disjointed statements are not likely to make a difference,” Henein says. His argument is that regulators must turn warnings into obligations with consequences.

Even an obligation backed by a fine can fail to change behavior if firms treat the cost as predictable. Arntz raises the hypothetical possibility that a firm could meet its reporting duty and absorb a fine while leaving the underlying problem unchanged. His concern differs from Kenney’s warning about misleading totals: one concerns whether enforcement changes conduct, while the other concerns how disclosures are interpreted. Effective enforcement must address harmful practices while allowing regulators to distinguish a firm that detects and contains attempted misconduct from one that leaves comparable activity unseen.

Key takeaways for leaders

• Interpret reports alongside monitoring quality: Higher AI near-miss totals can reflect stronger detection, while low totals may hide unseen activity. Risk and compliance teams need evidence of monitoring coverage before comparing incident counts across firms or business units.
• Record harm, intervention, and testing separately: Compliance teams need to document the attempted action, its potential consequences, and the control that stopped it. Keep deliberate testing results in a separate category, while FCA terminology remains unresolved.
• Reassess safety as conditions change: AI owners need ongoing checks as systems scale, face stress, or receive model updates. Clear records of agent identity, delegated authority, and accountability help explain where controls succeeded or failed.
• Include suppliers in resilience planning: A model provider’s outage or update can disrupt services or change behavior without changes to a financial firm’s code. Technology and risk teams need to assess supplier concentration and scrutinize model version changes as production changes.
• Connect reporting to corrective action: Reporting obligations and fines may leave underlying risks unchanged if firms treat them as routine costs. Boards and compliance teams need to track whether disclosed incidents lead to effective remediation.