In the digital age, organisations are eager to adopt and use AI to benefit from the technology’s instant and pervasive capabilities. However, entrusting AI with access to systems and data without a comprehensive understanding of how it works is dangerous.
This is according to Garith Peck, managing executive: cyber security and cloud at BCX, who delivered a presentation at the Chartered CIO Conference 2026, hosted recently by the Chartered CIO Council in partnership with ITWeb and sponsored by BCX.
Addressing the theme Digital Trust and Sustainable Innovation, Peck said organisations continue to deploy AI at scale, either intentionally or unknowingly, often without considering the level of autonomy with which the technology operates.
He said there is no doubt about the power of AI. The real question, he noted, is how quickly organisations can engineer trust, given AI’s ability to influence and change bias, beliefs and mindsets.
Peck likened the rollout of AI and its associated autonomy to the role of a commercial pilot.
“With every flight, the pilot will engage autopilot at some point. Automation is a huge part of modern aviation… the autopilot manages all functions, the avionic sensors and central server within the autopilot automation system. However, the (human) pilot never leaves the cockpit. He or she is there to oversee, to ensure trust and accountability. When it comes to AI, enterprises don’t have a ‘pilot in the cockpit’,” he said.
Evolution of cyber security
Discussing the role of AI in cyber security, Peck said the discipline has evolved beyond the days of relying solely on anti-virus software, firewalls and patches.
“Cyber security remains one of the most complex disciplines in IT,” said Peck. “Confidentiality and integrity are the foundation of cyber security – its ‘DNA’. But AI is changing how we adopt things from a cyber perspective. AI interprets, generates and infers; we act autonomously.”
Peck said that with AI, the threat landscape is no longer solely technical, but also cognitive and psychological.
“Machines influence how we perceive reality… the biggest risk with AI is mass manipulation of human trust at scale. The cyber battlefield is not about machines attacking machines; ultimately, it is propaganda used to attack individuals,” he added.
Peck also cautioned that the speed at which AI can generate information makes it difficult for humans to validate reality quickly enough.
“Eighty percent of people who consume technology are not technology savvy.”
He added that advances in deepfake voice and video technology have made it increasingly challenging to verify authenticity.
“The future crisis will not be data breaches, it will be the collapse of confidence in what is real."
Peck said organisations are becoming increasingly dependent on AI despite lacking sufficient AI and cyber security maturity.
“The most dangerous thing about AI is that its failures can appear normal, so that workflow is still completed and the answer may appear to be the real outcome,” he added. “What happens when AI gets it wrong? The public will not blame the algorithm, they will blame the organisation and its leadership.”
According to BCX, while South Africa has made progress in its digital transformation journey, including the digitisation of some social services and advances in healthcare, education and telecommunications, the country continues to face several challenges. These include digital literacy, infrastructure constraints and a limited supply of skills.
Peck also highlighted the issue of shadow AI, describing it as a growing concern for cyber security experts, particularly with the rise of agentic AI.
“This means in emerging markets, trust has become more important than ever before,” said Peck. “The defining technology of our time is not intelligence, it is trust. Digital distrust is economic inequality… humans can and will adapt, but AI influences judgments, shapes perception, confidence and belief.”
Peck said the role of the CIO has changed and in the AI era, CIOs will serve as trust architects and governors of machine accountability.
“Technology leaders must ensure that progress does not outpace trust… we have to have ‘a pilot in the cockpit’, with the right processes and procedures in place. Technology must be secure by design.”
Share
