Subscribe
  • Home
  • /
  • TechForum
  • /
  • Big data is not just a fashion trend - it is the real thing, says Cherry Olive

Big data is not just a fashion trend - it is the real thing, says Cherry Olive


Pretoria, 26 Aug 2013

The software industry is often seen as being the heart of the hardware industry, with partakers often creating a lot of hype about new software developments, such as cloud computing. Another, seemingly powerful, software trend is big data - something that is being spoken about increasingly, from various quarters.

"There is a current belief that there is nothing more fashionable than big data," said Johann Evans, chief technical officer at unified data management company, Cherry Olive, "but I am one of those people who believe it is far more than a hype, or a fashion - it is a growing phenomenon in the technology market that has to be seriously addressed.

"Generally, master data (shared data such as customer, product and location) is usually relatively small in volume. Here you are talking about a few million product data records, and, taken further, this amount could increase to tens of millions or hundreds of millions. But we are looking at these volumes only when one considers B2C (business to consumer) companies - where you get customer records in these volumes. These volumes can be comfortably managed by current database technology."

Big data is about much larger volumes - and current databases are battling to handle it. Big data is not just accounting data; it is data that is being spat out by the Web or via machines, such as Web logs or sensor data.

"It is frightening to note that a jet airliner generates a massive 20TB (terabytes) of diagnostic data each hour. Volumes like this are why current databases are almost haemorrhaging themselves in an effort to cope."

In the latter part of the 1990s, a three terabyte database was considered the norm. In 2003, the largest data warehouse in the world was 30TB in size. By late 2012, Teradata had 25 customers with petabyte-sized data warehouses, an increase of over 30 times in a decade.

"Clearly," said Evans, "big data is not just a whimsically acronym. It is here. It will continue to grow. And it will need to be managed. If companies cannot handle it themselves, they will have to look outside of their own walls."

Share

Editorial contacts

Johann Evans
CherryOlive
(+27) 12 259 8096