Johannesburg, 30 Jul 2020
In today’s highly competitive and rapidly evolving environment, businesses have to respond ever more quickly to market changes and customer requirements.
This has led to the growing adoption of what’s known as the DevOps methodology, which is increasingly employed to improve the quality and speed of delivering software. It is an approach that encourages communications, collaboration, integration and automation among software developers and IT operations in order to reduce the software release cycle and thus time to deployment, while minimising defects.
But developing software rapidly is one thing. Developing the right software applications to address the right issues requires insight into the business’ needs – and identifying those needs depends on having access to and understanding of data. In effect, what’s needed for realisation of the dream of the “data-driven” business is access to data insights in as close to real-time as possible.
The problem, says Johan Du Preez, Data & Analytics Architect with Ovations Group, is that current data teams are unable to deliver insights at the speed businesses requires.
To do so would demand a reduction in the cycle time of data analytics; the ability to move code and configuration from development environment into production; the ability to focus on important issues and create models and analytics that fuel business innovation; and the ability to catch incorrectly processed data and ensure the reliability and quality of the final output.
Enter dataOps
DataOps seeks to reduce the end-to-end time of data analytics, speeding up the time for the delivery of high-value, real-time insights for business decision-makers. It enables data scientists and analysis to develop more predictive models and better visually render data.
However, DataOps does not replace business intelligence (BI). “BI incorporates the insights that are delivered; DataOps is the framework within which to deliver those insights,” Du Preez explains.
While DataOps borrows heavily from DevOps to optimise code verification, it employs aspects of two other key methodologies – agile, to govern analytics developments; and statistical process control (SPC), to orchestrate and monitor the data pipeline – the transformation of data through a series of steps into reports, models and views.
“Effectively, as a modern, agile organising framework, DataOps strives to get data moving at the speed of business and still be accurate and reliable,” Du Preez says.
“Creating DataOps team structures and processes gives data and analytics teams a new approach to dealing with business requirements. Instead of trying to tackle a business requirement from a vague one-liner request, it allows for value to be delivered in smaller increments that can be provided rapidly and with accuracy. This leads to a reduction in the need for constant reworks.
“And because the data and analytics team is able to publish new or updated analytics in 'sprints' or rapid intervals, thus enabling the team to continuously reassess its priorities and more easily adapt to evolving requirements,” he adds.
In addition, the team is able to start implementing automated testing of BI deployments, enabling them to become proactively aware of report failures and thus set the pace of correcting issues.
Another benefit of DataOps is that with the adoption of agile development into data analytics, data teams and users can work together more efficiently and effectively.
According to Du Preez, although DataOps is a fairly new concept worldwide, it consists of industry best practices that some companies have been aware of for a long time. Growing numbers of companies are adopting this framework as it resonates with what is being experienced in data and analytics departments on a daily basis.
A South African bank, for example, is using DataOps not only to meet regulatory and compliance requirements with regard to data quality and governance, but also improve its business reporting and customer experience thanks to an aggregation of its data sources and a shift from batch delivery and static dashboard to persona-based, near real-time reporting.
This capability gave the bank much-needed agility to be able to navigate the pivot to the new market conditions wrought by the advent of the COVID-19 pandemic.
“There is no doubt that DataOps would benefit all decision-makers in an organisation as it allows more rapid access to data and insights, thus enabling more evidence-based decision-making,” Du Preez says.
“However, although DataOps requires collaboration across business functions, dealing as it does with the entire data life cycle from analytics, orchestration, data origination and requirements gathering, Ovations recommends that the transition to DataOps be led by the data analytics team.
“Nevertheless, regardless of which departments are involved in the process, the ultimate goal must be for the analytics teams to be able to deliver insights at the speed of business,” he concludes.
Share