Puzzle piece to big picture

In an age of data overload, organisations are increasingly faced with how to turn a mountain of puzzle pieces into a coherent whole, so they can make better, faster decisions.

Achieving this is the lifework of Jeff Jonas, IBM distinguished engineer and chief scientist of IBM Entity Analytics, who is working on advanced 'sense-making' systems to help companies “make sense of what they learn as they learn it”.

Speaking at the opening of the IBM PartnerWorld Summit, being held in New Orleans this week, Jonas stressed that “the data must find the data... and the relevance must find the user”.

He argued that, “while computers are getting faster, organisations are getting dumber”, as the growth of data outpaced businesses' ability to understand what it means - and respond accordingly.

Jonas dubbed this gap between the available observation space and sense-making algorithms “enterprise amnesia”, and said companies were heading for a dead-end by trying to squeeze data out of a puzzle piece. In the world of big data, said Jonas, it's all about context.

“You can better understand something by taking into account the things around it,” he said, using a puzzle metaphor to explain several concepts key to the sense-making process: Imagine an ever-growing pile of puzzle pieces of varying sizes, shapes and colours - until you start taking the puzzle pieces to the table to see how they relate, you don't really know where you are.

“With each new observation, one of three assertions are made: 1) it's un-associated; 2) it's placed near like neighbours; or 3) it's connected,” Jonas explained.

This process of accumulated context can get increasingly complicated as false positives and negatives emerge, or as new observations reverse earlier assertions, he added. But something important can happen once the 'edges' of the puzzle are determined and the middle pieces start to fall into place: “Given sufficient observations, there can come a tipping point where understanding is faster even though there's more data than ever before.” This development demonstrates just how key context is to the sense-making game - and how more data can mean better predictions.

Jonas explained that more data placed in the right context can help weed out false positives and false negatives. So, for example, if you have two pieces of data pointing to a John Smith, but with slightly varying information (one has his home address and telephone number, the other his home address and date of birth) then a third piece of data showing they are the same person reduces the overall complexity - the three John Smiths are now understood as being one and the same.

This form of identity resolution leads to another of Jonas' key points in data contextualisation, one which resolves even the most problematic of identity puzzles. He calls it 'space-time-travel'.

It is based on the precept that “the same thing cannot be in two places at the same time”. The question of when and where thus becomes the ultimate disambiguation tool, Jonas argued. He used the example of identical twins lying about their identities, claiming they are one and the same person.

Their date of birth would be the same; their fingerprints would be the same; a parent could even join in on the deception and argue that you are dealing with one person. How would a system resolve this dilemma of two persons with seemingly identical personal information? By relying on the fact that the same person cannot be in two places at the same time.

So for Jonas, when and where become far more important criteria in the standard list of identifying information (date of birth, ID, driver's licence, address, telephone number).

“Cellphones are generating a staggering amount of geo-locational data - 600 billion transactions per day in the US alone,” he pointed out. “This data is being de-identified and shared with third parties in real-time, and your movements quickly reveal where you spend your time.”

So, for example, simply by the system picking up that you spend 80% of your evenings in place X, it can reasonably assume that place X is home. Likewise, by registering that you spend your mornings travelling to place Y and afternoons returning from there, it can reasonably assumed place Y is work. All these conclusions can lead to powerful predictions about how people behave - and will behave.

“Space-time-travel data is the ultimate biometric,” said Jonas, adding that it will enable enormous opportunities but also unravel one's secrets and challenge existing notions of privacy.

Jonas is now working on a project called G2, which aims to help enable organisations “make sense of new observations as they happen, fast enough to do something about it, while the transaction is still happening”.

This will eventually lead to enterprises which can use what is happening in their observation space to make clear and informed decisions, through data finding data and relevance finding the user.

“The value of your data is proportional to the context it's in,” said Jonas, concluding that the organisations that can make better sense of the observation space and react faster will be the most competitive in the future of big data.

(Lezette Engelbrecht is hosted in New Orleans by IBM SA.)