Speed, trust, culture: The three biggest trends in data and analytics
Extreme data is one of three major trends driving data and analytics today, says tdglobal.
Enterprises are generating and collecting data on an incomprehensible scale. The IDC says data is growing at 40% annually and that by 2025, there will be 175 Zettabytes of data floating around the world.
Let’s put that into perspective. According to one analogy, if each Terabyte in a Zettabyte were a kilometre, it would be equivalent to 1 300 round-trips to the moon. Times that by 175 equals 227 500 trips to the moon and back.
tdglobal, a data and digital managed solutions and software provider, describes this as ‘extreme data’ and says it’s one of three major trends driving data and analytics today.
Trend 1: Extreme data
Wayne Borcher, Chief Operating Officer at tdglobal, says extreme data passes by so quickly that, if you blink, you’ll miss it. “Extreme data has to be analysed in real-time – literally as it streams by – if you want to gather actionable insights from it. Data that was generated even an hour ago is already old news.”
Around 20% of extreme data will be generated in real-time, mostly by the Internet of things. Once 5G connectivity becomes a reality, this percentage will increase exponentially, putting pressure on organisations to respond quickly, but also creating more opportunities than ever before.
“Businesses need to process streaming information in the moment, so that they can respond in the moment,” says Borcher. “Some businesses are already doing this, using new technologies or applying existing technologies in new ways.”
These newer solution platforms bring the capability to analyse vast amounts of streaming data in real-time, ingesting the data streams, analysing it and providing the analytics in sub-second response times. This enables near real-time analysis of streaming data, allowing businesses to make in-the-moment decisions. Together with cloud-based data warehouses that separate compute from storage to deliver consistently high performance, this reduces processing time from days to minutes.
“Consumers want what they want, when they want it. And so do employees. Technology has given them access to anything they want, at their fingertips. If you can’t provide that experience, they’ll easily find it elsewhere,” says Borcher.
But there’s a caveat. If you’re making decisions in the moment, you have to be confident that they’re the right ones. “Algorithms powered by artificial intelligence and machine learning are already making decisions traditionally left up to humans. Surgeons, financial advisors, crisis response teams, and every other industry will act on insights generated by these algorithms, which are trained on the data we feed them. We can’t afford to get it wrong.”
Trend 2: Trust in data is paramount
To get it right, businesses have to be 100% confident in their data, says Borcher.
“Even businesses that aren’t yet implementing AI and ML need to think about it. But many businesses are still not getting the basics right. And they have to get these absolutely right before they can think about AI and ML. Until they’ve been through the processes of engaging, cleaning, securing and governing their data, we wouldn’t advise that they base any business decisions on it. Inaccurate decisions, based on inaccurate data, almost always result in financial and reputational damage.”
He uses financial services businesses as an example. “These businesses need to comply with new and changing international regulations. The risk of not securing, protecting and appropriately using their data can be severe. But, if they trust that their data is properly stored and governed, they’ll achieve faster compliance.”
Once businesses are confident in the state of their data and data governance, they can start thinking about increasing data accessibility across the organisation and attracting AI talent, says Borcher. An important consideration is that trust in data needs to extend across the organisation, since data analytics is no longer only the realm of data scientists.
Trend 3: Data cultures
“Companies are focusing on instilling data cultures by embedding analytics into every department, empowering business users at all levels, and incorporating analytics into every workload. With the democratisation of data, everyone has equal opportunity to explore data, experiment with applications, and make decisions that could impact the entire organisation. But, if no one in your organisation can find the data they need to do their jobs, it’s time to redefine your data governance strategy.”
Getting to this point will be a challenge for many businesses, he says. “Data is still siloed across the organisation and there’s no single version of the truth. Imagine an orchestra that thinks they’re all playing from the same song sheet, but they actually all have different song sheets. When they start playing, it’s a cacophony. It sounds terrible and the audience walks out. The same is true in siloed organisations. Everyone thinks they’re using the same data, but the insights prove otherwise.”
Data access alone does not automatically translate to insight, he adds. Users first need to trust and understand the data, based on a shared vision for how the data should be used. Data should be easy to find, easy to understand and easy to use for a specific purpose.
“There’s a shift towards self-service solutions that are designed with ordinary users in mind, so that intelligence can be delivered and consumed by everyone, in the most natural way.”
The benefit, says Borcher, is that, when everyone has access to the same information, collaboration increases across the organisation. “Different people might ask the same question of the data, but in a different way. This will produce a number of potential solutions to a business problem. Deciding on the right one will require individual judgment and communication with others.”
Ultimately, the process begins and ends with people, he says. “Businesses need the right data and the right skills but, more importantly, they need direction from the top. Leaders must define business problems and ensure everyone is working together to solve them.”