Barriers to BI
Governance is still the biggest hurdle in the race to effective business intelligence.
Whether talking traditional big stack business intelligence (BI) solutions or new visual analytics tools, it's an unfortunate fact that enterprises still buy into the candy-coated vision of BI, without fully addressing the underlying factors that make BI successful, cost-effective and sustainable.
Many companies wrongly assume that in data, nothing changes.
Information management is a double-edged sword. Well architected, governed and sustainable BI will deliver the kind of data business needs to make strategic decisions. But BI projects built on ungoverned, unqualified data/information and undermined by 'rebel' or shadow BI will deliver skewed and inaccurate information. Any enterprise basing its decisions on bad information is making a costly mistake. Too many companies have been doing the latter, resulting in failed BI implementations and investment losses.
For more than a decade, I have been urging enterprises to formalise and architect their enterprise information management (EIM) competencies based on best-practice or industry standards, which follow an architected approach and are subjected to governance.
EIM is a complex environment that needs to be governed and which encompasses data warehousing, BI, traditional data management, enterprise information architecture, data integration, data quality management, master data management, data management life cycle, information life cycle management, records and content management, metadata management and security/privacy management.
Effective governance is an ongoing challenge, particularly in an environment in which business must move at an increasingly rapid pace where information changes all the time.
For example, to tackle the governance issue in context of data quality starts with the matching and merging of historical data to ensure design and storage conventions are aligned and all data is accurate, but according to set rules and standards.
It is not just a matter of plugging in a BI solution that would give the results. It may require up to a year of careful design and architecture to integrate data from various departments and sources in order to feed the BI system. The conventions across departments within a single organisation are often dissimilar, and all data has to be integrated and qualified. Even data as apparently straightforward as a customer's ID number may be incorrect - with digits transposed, coded differently between source systems, or missing - so the company must decide which data source or integration rule to trust in order to ensure data warehouses are compliant with quality rules. Data warehouses must also be compliant with legislation standards needed to build the foundation of the 360-degree view of the customer that executive management aspires to. But, integrating the data and addressing data quality is only one area where effective governance must be applied.
Many companies wrongly assume that in data, nothing changes. But, in reality, the company must cater for constant change. For example, when reporting in a bank, customer records can be dramatically incorrect if the data fails to reflect that certain customers have moved to new cities, or that bank branch hierarchies have changed. Therefore, linking and change tracking is crucial in ensuring data integrity and accurate current and historic reporting.
Automation can only take the company so far: it can automate to the nth degree, but it still requires data stewards to carry out certain manual verifications to ensure the data is correct and remains so. Companies need to know who is responsible and accountable for their data and be able to monitor and control the life cycle process from one end to the other. The goals are to eliminate multiple versions of the truth (results), have a trail back to sources and ensure only the trusted version of the truth is integrated into systems.
Rebel without a cause
Another challenge in the way of effective information management is the existence of 'rebel' or shadow data systems. In most companies, departments frustrated by slow delivery from IT or with unique data requirements start working in siloes, creating their own spreadsheets, duplicating data and processes, and not inputting all the data back into the central architecture.
This undermines effective data governance and results in huge overall costs for the company. Instead, all users should follow the correct processes and table their requirements, and the BI system should be architected to cater for these new requirements. It all needs to come through the central architecture. In this way, the entire ecosystem can be governed effectively and data/information can be delivered from one place, also making management thereof easier and more cost-effective.
The right information management processes also have to be put in place, and they must be sustainable. This is where many BI projects fail - a company builds a solution and it lasts only a year, because no supporting frameworks were put in place to make it sustainable. Companies need to take a standards-based, architected approach to ensure EIM and governance is sustained and perpetuated.
New BI solutions and best practice models emerge continually, but will not solve the business and operational problems if they are implemented in an ungoverned environment, much the way a beautiful luxury car may have all the features a driver needs, but unless the driver is disciplined, it will not perform as it should.
Mervyn Mooi is a director of Knowledge Integration Dynamics (KID), and also a key resource within the company's information management, data warehousing and business intelligence teams. He has been in the IT industry for 36 years, beginning his career as an operator at the CICS bureau in Johannesburg in the early 1980s. Thereafter, he was appointed as a programmer at state-owned oil exploration and production company SOEKOR. In 1986, Mooi joined Anglo American's head office IT department where he remained for almost 12 years. Here he progressed to become a senior programmer, analyst, database administrator and technical support specialist. After completing his degree in informatics, he then left to join Software Futures, where he worked as a senior consultant for 18 months in the data warehousing and business intelligence arena. Mooi joined KID in 1999 as a data warehouse and business intelligence specialist. Mooi's experience in ICT disciplines includes operations, business and systems analysis, application development, database administration, data governance/management, data architecture/modelling, production application and systems software support, data warehousing and business intelligence. He now focuses on enterprise information management, information governance and cloud solutions.