Cyber Insight urges businesses to test what happens when AI-enabled security gets a decision wrong

Confidence in AI should come from understanding how it operates within the business, including its limits.

---

An AI-enabled defence system should be able to make a mistake without giving that mistake control of the business.

That is the standard Cyber Insight believes enterprises should apply as they introduce more AI-assisted investigation and automated response into their security operations. Detection accuracy and speed remain important, but the company argues that the systems surrounding AI deserve just as much scrutiny.

Consider a hypothetical incident involving an account used to approve supplier payments. Suspicious activity is detected shortly before a payment run. Blocking the session could interrupt an attacker. Disabling the account could also interrupt legitimate work. Leaving it active carries its own risk.

The challenge is deciding which action is justified, what evidence supports it and how the consequences will be managed.

"Businesses should expect AI to help their security teams work faster," says Deon Smal, CEO of Cyber Insight. "They should also expect a clear answer when they ask what happens if the assessment is wrong. That answer needs to be part of the design, not something worked out during an incident."

Cyber Insight draws a distinction between using AI to interpret information and granting a system authority to change the environment.

A tool that summarises an investigation does not carry the same operational responsibility as a workflow that can disable accounts or isolate systems. Enterprises should understand where AI contributes an assessment, where a configured rule determines the next step and where an analyst makes a decision.

Generative AI introduces a further consideration when it interprets external material. Malicious instructions embedded in content can attempt to influence a model's response, a technique known as prompt injection. The UK's National Cyber Security Centre advises designing applications to reduce the consequences of compromised outputs rather than assuming such manipulation can always be prevented.

This does not mean every AI-assisted security product has the same exposure. It means businesses need to examine the specific technology, the information it consumes and the actions connected to its outputs. 

Cyber Insight's position is that access restrictions and approval requirements should be enforced by the surrounding systems, rather than depending only on instructions given to a model.

"An AI system may produce a convincing recommendation. That does not make the recommendation an authorisation," says Smal. "The business still needs to decide what the system may change, under which conditions, and who is accountable."

Cyber Insight believes a useful test of cyber defence is what happens after an individual control or assessment falls short.

An incorrect alert should not automatically trigger an unnecessarily broad interruption. Equally, a missed threat should not leave an attacker with unrestricted access to connected systems. 

The company recommends examining both outcomes when assessing a security design. Which permissions constrain a compromised account? What checks protect a sensitive transaction? What other evidence could expose suspicious activity? How would the business restore an affected service safely?

This approach builds on established cyber resilience engineering, which considers the ability to withstand, recover from and adapt to adverse events alongside prevention.

For Cyber Insight, the practical implication is that enterprises should test more than whether a tool detects a threat. They should also test whether the response remains proportionate and whether essential business functions have appropriate protection when something goes wrong.

Recovery must be treated realistically. Restoring access after an unnecessary restriction is a different problem from recovering information that has already left the organisation.

Keeping people involved does not mean placing every alert in a queue for manual approval.

Cyber Insight advocates agreeing response authority in advance, with clear conditions for immediate action, escalation and intervention. The risk of waiting should be considered alongside the risk of acting.

These decisions are already relevant to managed security services. Sophos MDR, for example, provides different arrangements for notification, collaboration and authorised response. Those settings govern the actions its operations team may take; they are not interchangeable with a claim that AI has unrestricted authority.

The same discipline should inform an enterprise's wider use of automation.

A critical server may require a different response procedure from an employee laptop. An unavailable primary contact should not leave the team uncertain about who can authorise containment. An analyst reviewing an AI-assisted recommendation needs access to the evidence behind it, not just a suggested action.

"Human oversight has to mean more than having someone available to click approve," says Smal. "That person needs the evidence, the business context and the authority to make a useful decision. Otherwise, we have added a step without necessarily adding control."

Cyber Insight recommends starting with one business-critical process and following a realistic incident through the existing environment.

The exercise should establish what would be detected, what could happen automatically, what would require approval and what the business would do if the initial assessment proved incomplete.

It should also consider change. An AI workflow approved to read information may later receive permission to update records or initiate actions. Microsoft's agent-security guidance recommends reviewing access as agents change and limiting permissions to the work they need to perform.

Cyber Insight combines locally based security expertise with Sophos technology and a global MDR network, supporting organisations with ongoing detection, investigation and response.

The company's message is that confidence in AI should come from understanding how it operates within the business, including its limits.

"The objective is not to make organisations hesitant about AI," concludes Smal. "It is to make them confident enough to use it properly. That confidence should rest on tested controls and clear responsibility, rather than an expectation that every decision will be right."

To discuss how AI-assisted investigation and response fit into your existing security environment, contact Cyber Insight at info@cyberinsight.co.za.

Local experts. Global intelligence. Trusted protection.