Time to get savvy
BI struggles to respond rapidly to the business need, even when attempting agile development.
In order for businesses to be competitive decision-makers, they need timely access to information. The vehicle for this is business intelligence (BI) teams and their supporting processes. The challenge for IT BI teams is that acquiring data from source systems (internal and external), and transforming this into information for decision-making, is a complex process that requires scarce resources and expensive tools and systems. Throw delivery methodologies into the mix and you have the modern-day dilemma of BI teams delivering information too slowly.
In recent years, many have tried to implement agile methodology in this area, but with mixed success. As a consultant, I have been involved in a number of these initiatives, and have noted a few common factors that largely contribute to the limited success of agile methodologies in data warehousing and BI.
The changing nature of business:
The very nature of BI is that the end-user requirements evolve frequently, and with an increasingly data- and digital-driven landscape, businesses are changing at an even faster pace. Today's businesses often have small windows of opportunity, where access to information is required within days in order to take advantage of an opportunity.
The fact is BI often cannot respond rapidly enough to the business need, even when attempting agile development. The upfront project negotiations, planning and design phase alone very often exceed the window of opportunity a business might have.
The paradigm shift:
The traditional methods for managing a BI engagement provide a solid foundation for solution delivery, eliminating much of the stakeholders' risk. These focus on in-depth, exhaustive analysis, design and implementation. All this is done with the best of intentions, but the result is long, drawn out and can be costly.
Implementing agile methodology requires a significant change in thinking, stripping away a lot of the safety and security people have grown accustomed to. The process is heavily dependent on multi-skilled people, which conversely, leaves many of them feeling exposed.
This creates a real challenge, as there is no one-size-fits-all solution, and the success of agile in BI is going to be largely due to the ability of the people involved to adapt and embrace the new paradigm.
This situation is made worse by the fact that the various agile methodologies are not prescriptive, and leaves a lot to the teams to decide. These individuals in the teams must be able to develop the mindset required to think and execute in an agile framework and approach, and not just according to an agile recipe book.
The relay team is only as fast as its slowest runner:
The simple fact is that vanilla agile is not enterprise aware and cannot scale to accommodate the constraints the enterprise places on it, eg, enterprise architecture and release management. When this happens, the "hurry up and wait" scenario occurs.
User requests are logged, and significant time is spent on prioritisation, analysis and design before landing with the delivery team. At this point, progress speeds up rapidly under agile development practices, but then slows down once again when the code is passed onto release management processes.
The unique nature of BI:
BI delivery is a complicated beast, interacting with many different systems and stakeholders throughout the company. In addition, there are vast complexities in successfully analysing data from disparate systems, designing a formal data warehouse and BI solution that can consolidate these disparate data sources, and successfully presenting the resulting information to the client.
BI delivery is a complicated beast.
Even with the best of tools to speed up data analysis and development, the design and implementation of these solutions will always be a significant obstacle that even agile methodology just simply cannot overcome.
So, what is the solution?
In the BI world, it is information, not code, which is valued above all other things. This implies a fundamental shift in thinking. Instead of thinking in terms of design and code, teams need to think in terms of data and information. The question should not be how to design and code a solution, but rather how to get the data or information to the end-user as quickly as possible.
As such, everything from engagement model to solution architecture to delivery methodology needs to change to address this concept. Once a team adopts this change in focus, it can have a profound and fundamental impact on the way a team delivers value to its end-users.
A comprehensive hybrid agile framework (Strategic Architecture for Value Yielding BI) should build on various agile practices and strategies, and include a strategic, enabling architecture and business-centric delivery team structures.
Some key principles:
1. Avoid large, difficult to motivate projects. Focus on short, bite-sized deliverables that add to a larger vision, but with far less attached risk and expense.
2. Rapid information delivery is the single biggest goal of the BI competency centre (BICC) - prioritise this!
3. Existing teams must be on-boarded to the new principles slowly, allowing sufficient time for training, mentoring and developing the resources, so they can learn how to think and be agile.
4. The entire demand management function for the BICC needs to run on a single agile framework, from end to end. Only in this way can a steady, reliable stream of work flow through from the idea phase to successful delivery.
BI needs to become more savvy to catch up with the pace business requires in order to be relevant and start yielding business value.
Julian Thomas is principal consultant at PBT Group, specialising in delivering solutions in data warehousing, business intelligence, master data management and data quality control. In addition, he assists clients in defining strategies for the implementation of business intelligence competency centres, and implementation roadmaps for a wide range of information management solutions. Thomas has spent most of his career as a consultant in South Africa, and has implemented information management solutions across the continent, using a wide range of technologies. His experience in the industry has convinced him of the importance of hybrid disciplines, in both solution delivery and development. In addition, he has learned the value of robust and flexible ETL frameworks, and has successfully built and implemented complementary frameworks across multiple technologies.