The big data exploration
Sometimes it takes a leap of faith and a fair amount of uncertainty to produce big data success stories.
The concept of big data is now well established in the hearts and minds of business, and most large companies have started (or are in the process of) investing in big data platforms.
Despite this, there is still a lot of uncertainty in the minds of some companies as to how they can benefit practically from big data. It's a case of: "I know what it is, I know that I need to have it, but I am not too sure what to do with it once I have it!" This then prompts businesses to state/ask:
* "We need you to come in and tell us what big data can do for our organisation."
* "Where does big data fit into our organisation?" or
* "What should our big data strategy be?"
These are only a few of the big data related questions practitioners are currently faced with. However, there is a perception that answering these questions correctly will provide a fantastic opportunity for businesses to embark on a new and exciting journey into big data. But, I find myself asking, is this really the case? Or, can answering these questions now be a critical mistake, one that will lead to an unsuccessful ending? Let me explore further.
Old rules still apply
Through experience, IT professionals have learnt IT should never drive business strategy. The early years of business intelligence (BI) and data warehousing (DWH) were littered with the discarded remains of many unsuccessful DWH and BI initiatives that failed due to them being technology-driven exercises.
When technology defines the use case, the business drivers and value, ultimately, what the business gets are solutions that are out of touch with what it really needs.
Significant changes to infrastructure requires major contract revisions. Bring in the lawyers!
So, the short answer to these questions is that, while data practitioners can tell companies what some of the technical capabilities of big data are, they cannot (and shouldn't) tell companies how it can benefit business, or how they should be using it.
The old rules of system development must always be remembered, starting off as usual with clearly understanding the business problems and opportunities. Instead of IT telling business what to do with big data, business must tell IT what its problems are in order for IT to determine how and where big data can be of assistance.
Bitter pill to swallow
When talking about business problems and opportunities, IT will typically get very excited about how big data leverages off commodity hardware, most commonly using open source software. As a result, there are massive cost savings for business regarding data storage, compute power and software licensing.
In IT terms, cost savings is the number one business opportunity. However, through my experience, I have come to realise this is not what the business necessarily relies on or cares about when making such decisions.
Consider the following:
* Most businesses are locked into IT contracts with a fixed term. Significant changes to infrastructure requires major contract revisions. Bring in the lawyers!
* Migrating data solutions onto big data platforms implies major costs, and more importantly, risk.
* There is also a significant human resource impact, implying massive training and recruitment requirements, with an increased risk of staff attrition.
The cost savings must be truly staggering for business to be willing to take on the risk and expense of such an implementation.
Disruptive use case
If the standard IT use cases should not lead the charge, what then? Big data is a set of disruptive technologies and capabilities. The companies that are going to benefit the most from it are those that are looking to be disruptive in the industry. The best, most successful use cases are often partnered with the development of entirely new business processes.
So, start by exploring the business. Speak to the business users, find out about the ideas that employees always talk about at staff functions. What is on the company's wish list that is never tackled because it is "impossible to do"?
Lock into the dreams, aspirations and ideas of the thought-leaders in the business. What was previously considered impossible might very well be possible when looked at through the lens of big data.
It must be said that this is not going to apply to all businesses. Conventional businesses that are unable to adapt to the new era are going to receive limited benefit from this technology and approach. It is the digitally-enabled, data-driven company that has disruptive business strategies that is going to truly benefit.
In early times, explorers would head off to discover uncharted lands and territories. They would only have a general sense of where they were going. They did not know what dangers they would encounter, nor what potential riches they would discover. Despite this, they set off, convinced there was value to be uncovered.
It is critical for business to adopt this same mindset. It is often difficult to motivate big data financially, as companies are never 100% clear on exactly what they are going to uncover in the data, or what additional possibilities are going to emerge. The technology is complex, which implies steep learning curves. Like those early explorers, this involves risk, and there will be highs and lows along the way. It is therefore extremely difficult to accurately predict cost and time-frames for big data.
Business, therefore, needs to be like the underwriters of those early explorers, with the foresight and vision to believe in the journey, and the guts to take on the investment.
Julian Thomas is principal consultant at PBT Group, specialising in delivering solutions in data warehousing, business intelligence, master data management and data quality control. In addition, he assists clients in defining strategies for the implementation of business intelligence competency centres, and implementation roadmaps for a wide range of information management solutions. Thomas has spent most of his career as a consultant in South Africa, and has implemented information management solutions across the continent, using a wide range of technologies. His experience in the industry has convinced him of the importance of hybrid disciplines, in both solution delivery and development. In addition, he has learned the value of robust and flexible ETL frameworks, and has successfully built and implemented complementary frameworks across multiple technologies.