Johannesburg, 21 Jul 2004
Without a doubt, business intelligence (BI) applications have successfully made their way into the hearts and minds of business users. BI systems are no longer the domain of IT departments. Each day, BI-based corporate dashboards are facilitating well-informed decisions by synthesizing, filtering and serving up bottom line numbers, facts and trends to users across the organisation.
With strategic BI tools at their fingertips, enterprise business users can establish, modify and tune business strategies and processes in relative real-time, helping them gain competitive advantage, improve business operations, drive profitability and achieve larger corporate goals. In the process, BI has become an everyday tool for customer service and revenue enhancement.
Said Mark Cooper, managing director of MigrationWare, distributor of Trillium and Discovery software in SA: "Even though there are signs of an economic upturn, that doesn`t mitigate the need for tighter controls and close scrutiny of operations by executives. Executives need to maintain this vigilance to squeeze out every ounce of productivity possible, though with less resources - less personnel, less budget and certainly much less time. Further complicating matters has been the real-time craze of the Web as both a competitive force and a productivity-boosting asset to BI systems through emerging Web services capabilities."
Yet, for all of the sophistication behind today`s BI suites and systems, their effectiveness within each enterprise hinges on the intellectual raw material they are asked to process - incoming data. "The application`s ability to help executives base decisions upon outputs is ultimately dependent upon the relative quality of data within the enterprise application," continued Cooper. "Whether the data depicts inventory, sales, suppliers, financials or customer metrics, without correct, comprehensive and timely data that accurately reflects an executive`s universe, BI systems will ultimately fail.
"As the Achilles` heel of BI, data quality has proven time and time again to be an indispensable step in ensuring the ultimate success of multiple enterprise systems - most particularly business intelligence. Without it, the system can lose credibility among users, and in the utmost worst-case scenarios, lead to disastrous business decisions. And with so much focus on BI, nowhere do you find attention being paid to the quality of the company`s data.
"For instance, if you are an insurance carrier that wishes to learn from your BI dashboard, who the top 10 companies that employ your customers are, your premise would logically be that the names of those top companies were spelled consistently in all of your databases. However, if one of those top customers was First Manufacturing Company, and one data mart had it stored as `1stManufacturingCo,` another had it as `FirstMfg` and a third had it correct, each would show up lower on your list than the one company would, if all data references were consistently accurate."
Cooper points out the case of a large organisation with millions of data files on inventory items sourced from tens of thousands of suppliers. Data needed to be organised for easy access to facilitate rapid shipments to virtually anywhere in the world at a moment`s notice. However, when the organisation closely examined the state of its inventory data, comparing records and files in multiple systems, it discovered numerous duplicate serial numbers. One particular serial number had mistakenly been applied to both a common item used in trash collection and to a vital electronic circuit board that held the brains behind a sophisticated piece of hardware. When the BI dashboard was queried for the number of deployments of that circuit board to gauge what costs an upgrade or recall might incur, the answer came back in the millions rather than hundreds. The solution to this problem called for a data quality implementation that ensured data for each serial number was distinct, and the rest of the data feeding their decision-support and inventory management systems was accurate and up to date.
Data quality problems typically result from seemingly inconsequential, easy to miss errors - usually human in origin. For instance, type-Os among the few lines of descriptive information from a legacy system that now need to be loaded into a new BI system, or in fixed fields originally hard coded into an order entry system forcing data entry operators to type into another field, irregardless of what that field was truly supposed to contain. Or perhaps the error resulted from the creative manner in which a support person used the status or comments field to capture valuable information. Several years later that creative interpretation of information is being tapped as the centrepiece of a new application. As so often happens in IT, the original intent was noble and accurate, but as times and requirements change the data must change with them.
Quite obviously, correcting data anomalies can go a long way towards recouping your BI investment and ensuring that the project will meet its stated goals. To that end, there are 10 steps to follow that will take the data quality risk out of BI.
1. Profile the data in all source systems to capture data content, quality and interdependencies. Data profiling automation solutions can automate this tedious, often error-prone process, reducing time required by up to 90% over manual assessment and profiling.
2. Standardise all relevant data to ensure consistency. This usually requires updating definitions and gaining consensus among top executives regarding how to define different data elements.
3. Match records across sources with statistical logic to eliminate duplicates and create relationships for a 360-degree view of all customer activity. In doing this, apply consistent processes and business rules around data quality to batch data cleansing and all new data entering the enterprise.
4. Enhance existing data from outside sources, such as database services that supply financials and demographics on consumers and businesses to further enrich the 360-degree view.
5. Share common meta data to gain consensus among managers on what the master data means. This way all parties can get to that information about the data from a central storage location.
6. Completely avoid hand coding the means of moving data. There are many sophisticated automation tools that extract data from disparate sources, transform it into appropriate shapes, and load the data into designated target systems and applications. This automated process is essential to being able to repeat and understand your data and the ONLY way to ensure that your data quality system can easily adapt to unpredictable business changes.
7. Allow for scalability. Do not limit the system to handle only the data volumes you need in the immediate foreseeable future. Parallel processing systems will allow your data quality system to accommodate growing data volumes by easily scaling over increasing hardware resources without costly re-architecting the data quality solution.
8. Go with best of breed solutions. While some vendors offer a "one-suite-does-all" approach, closely examine the functional capabilities and limitations of the offering, particularly regarding data profiling and data quality systems functionality.
9. Standardise as much as possible. When integrating best of breed solutions with older systems, open standards systems - like Web services, XML and J2EE - facilitate rapid connection of multiple data sources with BI applications to extend data management capabilities throughout the enterprise.
10. Follow a data quality mantra to allow for an unpredictable future. To ensure the integrity of data in the coming years and decades, enterprises need to create a culture of data quality that makes the topic a high priority for everyone.
"There is no question business intelligence will continue to gain momentum as a strategic business tool for enterprise organisations. In order to maximise the investment and derive true value out of the implementation, organisations need to focus on the quality of the data input within the BI application. Without strategic data quality initiatives in place, organisations will leave end-users and ultimately the future of the business vulnerable to bad decisions based on bad data," said Cooper.
Share