Data explosion speeds up

Global data is accelerating at a 45-fold annual growth rate which, according to analysts, is faster than what businesses would be able to store and manage by 2020.

This is according to independent consultant and founder for 9sight Consulting, Dr Barry Devlin, who was one of the keynote speakers at this week's ITWeb Data Warehousing 2010 conference held at the Gallagher Estate in Midrand.

Devlin pointed out that soft information, also known as unstructured data from Web-based content, is growing at a 60% compound annual growth rate. He also indicated that soft information first overtook hard information volume in the late '90s.

He revealed how 1 200 exabytes of information consists largely of images and video, while less than a quarter of that number (240 exabytes) is generated by enterprises.

“However, over 950 exabytes of information is touched by enterprises, and it's becoming increasingly important for businesses to access that information,” noted Devlin.

“Interestingly, there is not enough storage in the world to cater for all of this information. Only 65%, down to 50% in 2016, can be stored permanently and approximately 75% of data is duplicated.”

Devlin explained that the implication of the global data explosion is that an enterprise has decreasing control over its content and the quality of information it uses.

“Traditional data warehousing is becoming less and less relevant because of the proportion of overall soft and hard information.

“We've moved from a hard disk environment to solid state disks and now to in-memory. This means that the bottleneck in data warehouse design goes away. In-memory enables organisations to access and index information very differently from the past.”

“Businesses today are demanding both flexibility and integration, which are almost exact opposites. It's very difficult to do both of them together and yet that is what business demands. Dramatic changes in business needs and technology in recent years demand a new type of architecture,” he added.

According to Devlin, soft information will be the new focus of data warehouse managers. “Business requires a fully integrated store of hard and soft information. Most of the information that we deal with in the future will have different characteristics. Organisations need to check the level of duplication in their IT systems.”

According to the EMC-sponsored IDC Digital Universe Decade study, the amount of digital information created in 2010 is 1.2 zettabytes and this equates to every single person tweeting continuously for 100 years, or 75 billion fully-loaded 16GB Apple iPads, which would fill the entire area of Wembley Stadium to the brim 41 times.

IDC states that by 2020, more than one-third of all digital information created annually will either live in or pass through the cloud. In addition, 35% more digital information is created today than the capacity exists to store it. This number will jump to over 60% over the next several years.