As open source software framework Apache Hadoop and its community turns 10 this year, enterprise adoption of the platform is unprecedented.
This is according to market analyst firm Forrester, which anticipates firms will spend nearly $800 million globally on Hadoop and Hadoop-related services in 2017.
Hadoop is used for distributed storage and processing of big data sets using the MapReduce programming model. It consists of computer clusters built from commodity hardware. All the modules in Hadoop are designed with a fundamental assumption that hardware failures are common occurrences and should be automatically handled by the framework.
Forrester says the R800 million figure is likely due in part to firms deciding to build a Hadoop-based data lake.
The firm explains Hadoop started as a meritocracy of open source coders overseen by the Apache Software Foundation, who were driving Hadoop to solve a specific set of common business problems.
It adds the situation is more complex now because enterprises, independent software vendors and systems integrators have embraced Hadoop.
"These firms increasingly demand more control over its direction. Today, they can exert that influence by joining the ODPi or sponsoring contributors to open source projects - but will that be enough to keep the Hadoop market stable?"
If Hadoop follows the trajectory of other hot emerging technologies, Forrester expects some market consolidation through acquisition soon, possibly for control of Hadoop.
"The question is - how long can Hadoop sustain this kind of growth? We think Hadoop's preteen growth spurt faces some challenges. Enterprise architects should look at the big, open questions surrounding the evolution of this remarkable technology," says Brian Hopkins, Forrester's VP and principal analyst serving enterprise architecture professionals.
Forrester notes most of the cloud Hadoop services, including Amazon Elastic MapReduce, Azure HDInsight, Google Cloud Dataproc and Hortonworks Data Cloud for AWS, use the Hadoop Distribute File System (HDFS) as an interface layer onto their respective cloud object stores directly.
The market analyst firm adds that other big data analytics software, like Apache Spar, can use HDFS but also draw from any file system or even databases.
"So in the cloud, what is left of Hadoop? YARN still has value as a resource manager, but that could diminish over time, too, as projects like Mesos gain prominence or resource management becomes less important because it is abstracted by the cloud vendors," says Hopkins.
"Vendors whose strategy and revenue base for Hadoop solutions are primarily on-premises knew that cloud adoption among enterprises was coming, but it came faster than everybody expected. As Hadoop has become a mainstream enterprise technology, it's natural for enterprises to seek cloud adoption options. The result - public cloud Hadoop options are changing the very nature of Hadoop itself."
Forrester points out that as enterprises move further into the public cloud, they will naturally adopt more of vendors' cloud object stores and vendor-specific analytic processing frameworks instead of Hadoop alternatives.
"This means Hadoop distribution vendors need to stop their fierce competition and cooperate, or else they'll face losses in relevance and revenue," Hopkins notes.
Share