Coping with the third force

Once in-house data has been correctly ordered and made available in the enterprise data warehouse, it will typically need to be augmented with third-party data which reflects the broader universe in which the company operates.

Ralph Kimball, regarded by many as the "father" of data warehousing, writes on this issue: "Pressure to include external data in our data warehouses bears on us now from a combination of forces that have arisen in the last few years. First, we want to leverage the wealth of information we have about our customers. Second, much more data has become available: We work more closely with our business partners and third-parties, who provide us with data about our products, customers and markets that we are unable to generate in-house. The third driver is the Internet. The ready availability of data transfer tools for the Internet has lowered the barriers to exchanging data with business partners."

Another item on the wish-list of most large, customer-centric organisations is the desire to analyse key business metrics.

Julian Field, GM, Ascential Software SA

As examples of this pressure, detailed sales reports might need to be supplemented by weather reports, broken down by region. Customer data, along with recent purchase records, might profitably be overlaid against geo-spatial or census data so as to generate propensity-to-buy models based on external inputs, narrowing and intensifying the focus of outbound campaigns, reducing wastage and enhancing their chances of success. Or, as is emerging with ERP II and collaborative commerce, the extended supply chain needs to be informed and inform, in real-time or as close to it as possible.

Or the nature of the business can change and the company outsources to a third-party distributor. Unexpectedly, and without too much forewarning, the business must adapt to accommodate the new data reality.

Another possibility, which we`re seeing fulfilled by the day, is that databases can grow to such an extent, and so fast, that it is beyond the ability of the business to cope with the data flow and therefore it misses the business opportunity presented by very large databases (VLDBs). These can hold limitless volumes of data which can be trawled at high speed thanks to the power of today`s servers.

In this, the latest Industry Insight on data quality, I focus on the importance of dealing with change at the second level: that where as-yet unidentified data needs to be integrated with core production data, often at very short notice (the previous Industry Insight looked at the primary impact on the business of companies writing their own data interface scripts).

Take the example of Telkom, which is replacing some 600 core systems. It has not specified which systems it will buy, and it doesn`t want to prescribe or be prescribed to, but it does want freedom of choice, along with the discipline that delivers this freedom. Its solution was to lay an integrated, enterprise-wide data platform, one that permits any source to any source, and any source to any target. The result: the latitude to choose applications that are best for the business, rather than being hamstrung by technology issues, such as data proliferation.

The elegance of this solution is self-apparent, and its benefits will be felt for years to come.

To get there, Telkom management had to undergo a vital change in mindset: to move away from the internal culture of writing its own routines, scripts and data interfaces and to automate the process. It is a point all large organisations with complex data structures reach at some stage, typically after wrestling with the issue for years and finding business opportunity constrained by architectural challenges.

Another item on the wish-list of most large, customer-centric organisations is the desire to analyse key business metrics (analytics on demand) and profile customer activities in real-time: what has been referred to as the zero latency enterprise. The key restriction here has been the time consumed in the batch process currently involved in updating the data warehouse. It is in this space that we see the convergence of data integration, VLDBs and real-time analytics: the ability to choose, on the fly, from any source and map to any target; to populate databases of limitless size; and to interrogate these in real-time so as to permit the business an ongoing, current view of what is happening under the hood and in the extended universe.

To do this, you need an architecture that can collect and analyse information continuously, in real-time, capturing changes in source systems as they occur, and provide the results without delay to business users. Blue sky? Not really; the pressure from the business for competitive advantage in today`s increasingly e-enabled business world means most of these requirements are on corporate wish-lists already, and it`s only a matter of time before they are drilled through to IT architectures.