How to prepare for and remediate an AI system incident

For all the possibilities AI gives us, there is always a chance of the technology malfunctioning or becoming compromised. In the event of an AI system crisis, new research from ISACA has found that the majority of organisations surveyed couldn’t explain how quickly they could stop an AI system emergency, or even report on what caused the issue.

According to ISACA’s report, 59% of digital trust professionals didn’t understand how quickly their organisation could interrupt and halt an AI system during a security incident. Just 21% reported that they could meaningfully step in in half an hour. The indicates a landscape where corrupted AI systems can continue to operate unchecked, leading to a risk of irreversible damage.

Ali Sarrafi, CEO & Founder of Kovant, an autonomous enterprise platform, said, “ISACA’s findings point to a major structural issue in the way that organisations are deploying AI. Systems are being embedded into critical workflows without the governance layer needed to supervise and audit their actions. If a business cannot quickly halt an AI system, explain its behaviour, or even identify who is to be held accountable, the business is not in control of that system.”

AI failures and risks

In all, only 42% of respondents expressed any confidence in their organisation being able to analyse and clarify serious AI incidents, thus leading to possible operational failures and security risks. Moreover, without explaining these incidents to regulators and leadership, businesses may face legal penalties and public backlash.

Proper analysis is needed to learn from mistakes. Without a clear understanding, the likelihood of repeated incidents only increases. It’s important is to manage AI responsibly, with effective AI governance, yet ISACA’s findings indicate this is often missing.

Accountability is another fuzzy area with 20% reporting that they do not know who would be responsible if an AI system caused damage. Just 38% identified the Board or an Executive as ultimately responsible.

Sarrafi noted that slowing down AI adoption is not the answer; instead, rethinking how it is managed is key. “AI systems need to sit in a structured management layer that treats them as digital employees, with clear ownership, defined escalation paths, and the ability to be paused or overridden instantly when risk thresholds are crossed. The way, agents stop being mysterious bots and become systems you can inspect and trust. As AI becomes more deeply embedded in core business functions, governance cannot be an afterthought. It has to be built into the architecture from day one, with visibility and control designed in at every level. The organisations that get this right will not reduce risk, they will be the ones that can confidently scale AI in the business.”

There is some reassurance, however, with 40% of respondents saying humans approve almost all AI actions before being deployed, and a further 26% evaluate AI outcomes. That being said, without an improved governance infrastructure, human oversight is unlikely to be enough to identify and resolve issues before escalating.

ISACA’s findings point towards a major structural issue in how AI is being deployed in different sectors. With over a third of organisations not requiring their employees to disclose where and when AI is used in work products, the potential for blind spots increases.

Despite more stringent regulations that make senior leadership more accountable, organisations are failing to implement and use AI safely and effectively. It seems many businesses are treating AI risk as a technical problem, not as something that requires careful management in the entire organisation.

Change to how the integration and actions of AI are handled is essential. Without proper governance and accountability, businesses are not in control of their AI systems. Without control, even the smallest errors could cause reputational and financial harm that many businesses may not recover from.

(Image by Foundry Co from Pixabay)

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and co-located with other leading technology events. Click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.