New tools promise end to information overload

The faster the world moves, the more we need to know about it. We're all swamped with information - and we're only going to need more of it in future. But do we have the tools we need to manage and make sense of the information that surrounds us?

Ideally we should be able to find just what we need, when we need it: but in practice that doesn't often happen.

In too many cases we have felt that more information hasn't helped us to make better decisions; it's just made us more confused, anxious and over-worked.

But despite the screeds that have been written about "information overload", the problem isn't that we have too much information - quite the contrary, we need more and better information!

Instead, the problem is that the information we do have is poorly organised and presented. Most organisations are simply not able to manage their information, because they don't have the right tools. Other areas of technology have evolved beyond recognition in the past 20 years, but data analysis and presentation have not kept pace.

Take the paper report, still the most common information analysis tool on the planet. As any regular user knows, reports almost always raise more questions than they answer. A glance at a report on sales figures by region, for example, often quickly leads to questions about what's going on with particular product lines, reps, clients, or types of clients. The report lists figures for the last year and you'd like to see more detail for the past six weeks, or vice versa. These are simple and obvious questions, most of the time - but getting the answers can involve hours of painstaking work and calls to a put-upon IT department which will probably want to know why you didn't ask those questions in the first place.

Business intelligence (BI) software based on Online Analytical Processing (OLAP) technology was supposed to change all this by making it easy to find and analyse relevant information. A typical OLAP implementation involves pulling data from various sources in the organisation into "cubes" with predefined hierarchies that make it easier to interrogate the data - so long as you stay within the hierarchies. The moment you ask a different question, you're straight back to the old problem of needing hours of expensive IT time to get the answers.

Now here's something you'll hardly ever hear an IT professional say: the problem isn't the users, it's their tools. Human intelligence is remarkably good at spotting possible patterns and trends, and new data will always raise new questions. It's never going to be possible to decide in advance exactly what questions you want to ask.

There are two reasons why our information analysis tools haven't been able to keep up with us so far. First, the number of possible data permutations rises exponentially with the number of variables you're working with. Given three variables - customer, product and sales person, you can answer any question about your sales with just six reports. Add just two more variables (customer type and area) and you'll need to wade through 120 reports, and by the time you get to ten variables you're facing several million possible permutations.

Apart from the fact that it is impractical to produce all these reports, it is simply impossible for the human brain to analyse information on that basis. It is therefore clear that traditional reports are not the best way of analysing the volumes of data and permutations that we have to deal with today.

Historically, BI has attempted to facilitate the speed with which we can generate reports. By doing so, it has focused on the wrong issue and has not alleviated our problems.

The second reason is that until quite recently, computer memory and processing power just wasn't up to the job of handling those millions of permutations simultaneously. OLAP technology was born in the 1980s and is designed to work around the hardware constraints that existed at the time. Those constraints have long since been broken, but users of OLAP-based software are still bound by them.

In the past few years, three technological advances have changed that picture dramatically. The first is massive increases in processing power and memory, including gigabyte-sized memories on standard machines. The second is improved data compression abilities, and the third is a patented technique called Associative Query Logic (AQL) that enables every piece of an organisation's data to be linked automatically to every other relevant piece of data.

The combination of these three advances has made it possible to develop what we call "in-memory associative BI" - business intelligence applications that can hold an organisation's data in active memory while you work. This makes data analysis and querying almost infinitely flexible and responsive, presenting new views on the data with a single click and making it possible to analyse the data along any possible dimension. In a world where you can never know exactly what questions you might want to ask until you've seen the data, this is an immensely powerful capability.

Instead of the few million reports you'd need to comprehensively analyse 10 sales variables, you can do it with two screens of an easy-to-use dashboard.

Information overload is not about too much information, it's about not having the tools we need to use it properly. Fortunately, we now have the technology to make information overload history. Data analysis tools that follow the intuitive leaps and turns of the human mind can turn us into fish in the information ocean, rather than tired swimmers clinging to our rafts.