Subscribe

A solution for heavy data lifting

Matthew Burbidge
By Matthew Burbidge
Johannesburg, 26 Mar 2018
Amit Dave, CTO for IBM Systems, MEA: The challenge with storage architecture is how to remove latency.
Amit Dave, CTO for IBM Systems, MEA: The challenge with storage architecture is how to remove latency.

While many enterprises grapple with the challenges of big data, others are putting their infrastructure to work, lifting 'heavy data'.

Leon Van Wyk, head of media asset management at MultiChoice Video Entertainment, told a data storage solution forum hosted by IBM in Johannesburg this week that his organisation used IBM's Elastic Storage Servers (ESS) to process thousands of 'heavy' files, often coming in at 70 or 80 gigabytes at a time, and send them to over a hundred thousand clients every month.

Van Wyk says MultiChoice has now increased its online storage to four petabytes, from 380 terrabytes, but its footprint in its data centre had decreased. For other, older data, it has 42 petabyes on tier two facilities, where it employs a tape storage solution.

"You really have to break away from the traditional concept of storage. This (ESS) was a game changer," he says.

The ESS solution has also meant that MultiChoice has been able to embark on a rationalisation exercise on its Randburg campus to collapse its arrays into one unified storage facility.

IBM ESS is an implementation of software-defined storage and combines the company's parallel file system with its POWER8 processor-based servers. It provides greater throughput and simplifies data management by consolidating storage requirements. The company's Spectrum Scale file system also means it provides a single name space, regardless of whether it was residing on tape, disk or a competitor disk.

Ronnie Moodley, Sales Executive in the IBM Systems Group at IBM South Africa, says infrastructure remained a key factor in any organisation. The challenge for many however was how to best leverage the insights that could be gathered from data. These insights, in turn, rest on pillars such as storage, security and the speed of compute.

"You really have to break away from the traditional concept of storage. This (IBM Elastic Storage Server) was a game changer.

Leon Van Wyk, MultiChoice Video Entertainment

Van Wyk says a challenge for MultiChoice was not only finding the right solution, but also ensuring it will still be relevant in a few years' time.

"Technology is changing extremely fast. The refresh rate on IT hardware is roughly three to four years. As the technology evolves, you get bigger throughput, bigger scale, more consumers, and we have to keep up no matter what industry you're in. We try and calculate what the future is going to hold."

"We had to find something in the industry that would measure up to our requirements. And also try and build something that would last for at least five years."

The battle against latency

Amit Dave, CTO for IBM Systems, MEA, says its customers are gathering increasing amounts of data and performance of infrastructure is becoming increasingly important.

"Some transactions in the AI world such require instantaneous performance, even faster than flash, and that's where Non-Volatile Memory Express (NVMe) comes in. It's able to significantly reduce I/O overhead between CPUs and storage, resulting in performance improvements compared to previous interfaces such as SCSI that were originally developed for use with far slower hard disk drives."

Dave says the challenge with storage architecture was how to connect hundreds of terabytes, or even petabytes of capacity to analytics engine and still remove low-latency. This is where NVMe delivers without compromising any of the reliability and data services provided by mature network protocols.

NVMe-oF (over fabric) as an architecture was developed by optimising all of IBM's storage stack, starting with IBM FlashSystem 900.

"Our strategy is to enable NVMe-oF capabilities through a software upgrade, thereby allowing clients to carry on doing acquisitions of our all-flash arrays today, knowing very well that it is ready for NVMe-oF as the capabilities roll out across the family."

Dave says IBM has also recently added a new product called Spectrum NAS (network attached storage) that is a software layer sited on existing disk storage.

"If you want to create a NAS environment, you don't have to go out shopping for NAS hardware. Now you take the software, part of our Spectrum Storage Suite, and create your own NAS."

Share