BI faces architectural shift
Data warehousing has been around for 26 years. To give a little context, 26 years ago, in 1988, there was no big data, mobility was in its infancy, there was no real public e-mail or Internet, and storage for 100MB cost $10 000.
"The old data warehouse model had a single data model, where the data warehouse held all of a company's data, both private and public, in a single mode. This has turned out to be very impractical; we need private spaces that hold data not for public consumption. This should now be built into the model, but although it can be done, the books haven't been updated."
So said Mark Madsen, president of Third Nature, who was presenting a keynote at the ITWeb Business Intelligence Summit 2014 in Bryanston this morning.
He added that we are facing an architectural shift, and that BI is not about product, but about capability. "The way organisations handle data is changing. We've gone from a few sources of information coming into the database, to multiple sources. The very notion that you must know and model all the data first, is untenable."
Madsen said businesses must find ways of dealing with all the new external and event data that is flooding the enterprise. "We are in the middle of an architectural transition for data, and technologies such as NoSQL and Hadoop are allowing companies to collect and store massive amounts of data that we couldn't before."
He also said the ability to adopt new technologies is governed by people and organisations. "There are innovators and late adopters. Innovators view technology as a means to capitalise on future opportunities; late adopters tend to view technology as a means to resolve current problems.
"Some people can't resist getting the next new thing because it's new and new is always better. Many IT organisations are like this, promoting a solution and hunting for the problem that matches it. It is better to ask what is the problem for which this technology is the answer?"
He says the design point should be the passive consumer of information. "Self-serve BI is not like picking the right doughnut from a box.
"As practices evolve based on new capabilities, a new level of complexity develops over the top of the older, now better understood processes, leading to new data and analysis needs."
When forming a data strategy, companies must think of how to connect problems and opportunities with solutions. He added that businesses should focus on how data can be used as a resource to further the goals of a business strategy.
For one thing, he said, this means building the capabilities, organising the effective use of data, and building the necessary infrastructure. "Planning data strategy means understanding the context of data use so we can build infrastructure. We need to focus on what people do with the information as the primary task, not on the data or the technology."
Complexity will continue to increase, he explained. "Technology captures observations. These change our understanding. New understanding changes practices. Practices drive changes to technology, needing more data."
Essentially, he said it is about having an architecture that manages data as infrastructure. "With this in place, we can manage position data according to its different uses within the business, and store data in different places for different uses."
In this way, he says data use must be separated from the mechanisms of its delivery.