Making a meal of healthy data

While technology systems grow and grow, there is often very little possibility for viewing all company data from an integrated, single, trustworthy data store.

It is with this ideal in mind, that data integration needs to be a critical component of an overarching business intelligence (BI) and even storage strategy.

Companies that have embraced a BI policy need to be able to quickly get to and manage data that is gathered throughout the organisation. This data has to be transformed to make it consistent and trustworthy in order to provide credible, strategic BI as opposed to ad hoc reporting on data that is obtained from silos in storage areas.

A good data integration tool can remove the headache of moving, accessing and transforming data throughout the enterprise.

Most companies are looking for a series of data integration tools that not only respond to data integration and data quality improvement requirements but also enable the enterprise to consolidate the number of vendors they use by standardising on a single integration tool platform. This ensures enterprise-wide data integrity, reduces the cost of data integration, enhances the sustainability of integration projects and ultimately reduces the overall cost of downstream data exploitation.

The key word for data integration is elimination. The elimination of arduous and time-consuming custom hand coding, the elimination of documentation backlogs, and the elimination of reliance on specialist skill sets. For this reason, companies should seek out a set of data integration tools to assist them in ensuring the data in their organisations meets with all the required health checks.

The issue of data quality should dovetail data integration efforts as opposed to being an afterthought in the integration process.

Annemarie Cronje, product manager for the SAS Data Integration Server product suite at SAS Institute South Africa.

When deciding on data requirements, the first element to look at is the elimination of delivery delays and the high costs associated with having customised systems built for every integration project. Not only is it expensive to develop customised software tools, but it can be extremely dangerous if the intellectual property which developed that software leaves the company. In many instances these tools are developed to handle only a single element or integration challenge of the business, and often prove ineffective when needed to execute data integration across a myriad of disparate systems.

A data integration tool which allows integration tasks to be developed in a collaborative, self-documenting, re-usable manner and can run across disparate systems, being able to read and transform a variety of different data streams, can prove to be more effective. This also eliminates time delays in execution and potential hold-ups in the business environment due to ineffective data integrity tools.

Bad data input results in bad data propagation. Data integration is not only about getting any data into a single system for data exploration to be able to occur. It is also about improving the accuracy and relevance of the data contents, using standardising and matching techniques, during integration. Data that is inaccurate or contradictory can have devastating effects on reporting, analysing or forecasting. Imagine investing in a company with five million subscribers, only to find out that one million were duplicate clients.

The issue of data quality should dovetail data integration efforts as opposed to being an afterthought in the integration process. By embedding data quality within the data integration process, the end result is one which is clear and consistent, allowing problems to be identified before the fact, as opposed to wasting time and money after the fact to track down inaccurate or inconsistent data. The company is then also ensured of having an accurate, trustworthy data platform to support future analytics and business performance management across the enterprise.

But projects change, and when one data integration task has ended, there is often cause for a new one. In the past, companies had to view each data integration project as a separate occurrence and develop a budget and project scope for each instance. With a single enterprise-wide data integration tool, the company can avoid the spiralling costs of each new project, because existing tools can be recycled to work across a variety of projects as they are able to interface with all of the existing systems within the organisation.

Real BI is not merely achieved in the analytics or presentation of the data after extraction. BI starts with having the right data, which is clean and healthy, in place right from the onset of every project. Therefore it is important to use standards-based data integration tools in conjunction with any BI solution to ensure the right data, first time and every time.