Plan, standardise, transform data for better BI

In a global economic environment characterised by periods of sharp decline, steady growth and unpredictable periods of stability, organisations are turning to business intelligence and corporate performance metrics to help improve productivity and cut costs across the business. In the process, CIOs are learning that the success of any business intelligence implementation hinges on the accuracy, reliability and timeliness of the corporate data feeding the system.

Before any business information can be analysed, relevant corporate data must be refined into actionable information using data integration. When applied to business intelligence, data integration provides the necessary infrastructure to ensure that information is accessible, reliable, complete and consistent.

There are four steps an organisation can follow when preparing data for a business intelligence system.

Plan ahead and assume nothing about the state of your data when you begin a project.

During the business requirements phase of a project, it is imperative to determine the true state of the data from all of the contributing sources in your organisation. Specifically, assess and profile the data in terms of availability, consistency and the validity of embedded business rules. This process should be automated, and use as much data as is statistically important to you.

Profiling data provides an immediate payback by quickly defining what format manipulations and standards are needed to ensure downstream accuracy and reliability.

Before any business information can be analysed, relevant corporate data must be refined into actionable information using data integration.

Julian Field, MD, Centerfield Software

Standardise, match and enrich the data. Once you have identified any inconsistencies in data, it is essential to correct the data; standardise the names, addresses, product codes, birth dates, ID numbers and locations into consistent and reliable formats.

There is a good chance the data will require some level of enrichment to complete missing or inaccurate values. This may mean merging external information sources with your own data.

Many organisations store their customer data in numerous systems, so it may be necessary to remove duplicated data. This process should use statistically proven matching techniques to ensure the consistency and successes in standardising any sort of data, not just customer information.

To populate your business intelligence system, you will need to pack up and move the data from its source to the target data warehouse in a reliable and timely manner.

This step requires the ability to extract, transform and load data from the myriad hardware platforms where it resides and deliver it to the consolidated data warehouse that feeds your business intelligence systems.

This can be a very time-consuming exercise and you may want to investigate the benefits of parallel processing systems. Alternatively, if you need immediate visibility as soon as a customer places an order, then you may be able to "trickle-feed" the information instead of taking the bulk approach.

The last step is to ensure that you can trust your data. This means giving end-users the ability to track and understand data composition and continuously monitor data quality.

After delivering the data to the target system, you will need to ensure that business users have access to the source, definition and history of that data, as well as an understanding of how that data relates to other information throughout the organisation. Business-friendly explanations of the definition, origin and relationship of data can be presented through a browser and can remove ambiguity, lessen the IT support burden, and provide end-users with information that can aid better decision-making.

Periodic data audits will also help ensure the continued reliability of your data. Data audits provide management with graphical assessments of data quality based on business rules, and allow you to recognise anomalies as they develop and rectify them immediately.

The first key to ensuring a useful and successful business intelligence system is to make sure you have a data integration system that can handle the full range of functions from profiling to data quality, transformation and delivery.