As the big data phenomenon hits the enterprise, there has been the emergence of the disturbing trend of businesses storing vast amounts of personal data unnecessarily.
This was one of the biggest takeaways from the Technology Radar, a report by ThoughtWorks, a global technology company. The Technology Radar, now in its fourth year, is created by the Technology Advisory Board, which consists of 20 senior technologists at ThoughtWorks.
"These days there's a lot of hype around the idea of big data - and, with it, the notion that we should capture and store every bit of data we can get our hands on. The 'capture-it-all' approach raises serious questions of privacy," states chief scientist, Martin Fowler.
"We advocate that businesses adopt an attitude of "datensparsamkeit" and store only the absolute minimum personal information from their customers," Fowler says.
Dr Brain Leke, lead consultant at ThoughtWorks SA, says the main challenges organisations are facing right now is the kind of data they should be collecting and how much of that data to keep.
He adds that analytics has generally been driven by a need to derive significance from existing data rather than the business drivers aimed at. This leads to collection of unused segments of data which have no business need, he explains.
"We firmly believe that a user-value-driven approach in which high-valued business intelligence capabilities drive the evolutionary development of data warehouse components needed to support those capabilities. This helps avoid overbuilding the warehouse to support more than its intended purpose," says Leke.
He also notes that the "capture-it-all" concept leads to data capture that is not necessary for business decisions. There is extra effort involved in analysing such data, which can be avoided if just the necessary data is collected, Leke notes.
"Exhaustive searches require more hardware and, thereby, cost to derive significance and should be discouraged. Hackers are also more ferocious than ever, with one spectacular security breach after another in recent times. The recent expos'e by the American NSA contractor [Edward Snowden] revealed unprecedented mass-surveillance by numerous governmental agencies."
Leke believes that the decision on what is to be captured should be driven by only what the business needs rather than capturing every information available. This is safer due to the fact that information that is not stored cannot be stolen, he adds.
To Leke, organisations can maximise the potential of big data by using iterative, incremental and evolutionary methodologies driven by business needs and drivers. This will need a postulation of theory and incrementally evolving the data model with findings, he states.
"Organisations also need to focus on value-driven development approaches. The goal of analysing data should be to produce user-valued features. Users only care about access to and presentation of information that helps them either solve a business problem or make a business decision and this should be the main focus."
Organisations should also strive to automate their processes used to derive any significance from the data, which leads to more predictive behaviour, says Leke.
Share