Cor Winckler, technical director of Prescient Business Technologies, recently attended the six-day long TDWI (The Data Warehouse Institute) Spring Conference in San Francisco.
It is more of an in-depth "BI University" than a conference. A series of full-day and half-day courses were offered each day, and in the evenings attendees could select between visiting the 45 data warehouse DBMS, ETL and information delivery vendor exhibitions, one-on-one "guru-sessions" with the lecturers or more "night school" classes.
With in-depth sessions by industry leaders and data warehousing experts like Larry English, Margy Ross (co-author of Ralph Kimball`s latest textbook and president of the Ralph Kimball group), Claudia Imhoff, Colin White, well known author on information management Clive Finkelstein, metadata guru David Marco, and many more, it is the ideal event receive in-depth instruction on the latest developments in the data warehousing world.
One of the key principles, which were confirmed by the keynote sessions and many of the presentations, is that a data warehouse that is not based on a sound need by business, with strong business ownership and sponsorship, is bound to fail or deliver sub-optimally. Despite technological advances, the data warehouse cannot be bought off the shelf or driven by the IT function - it needs to address specific prioritised business needs. Data warehouse implementation must be seen as a series of strategic long-term projects for the company.
A very hot topic at the moment is real-time or near-time data warehousing. As data warehouses and their users mature, faster turn-around times are required to get the data in the data warehouse. For example, detecting credit card fraud five days after it started happening is of no use, since the card in question has long reached its limit. The anomaly (ie the fraud) needs to be detected within hours to be of any use. The "mining engine" that searches for anomalies in the credit card transactions needs to access near-time data in the data warehouse. Operational transaction systems have to trickle-feed the transactions into the ETL pipeline as they happen. One of the enabling technologies for this is so-called enterprise application integration (EAI) messaging systems. The EAI would typically be used as a queuing mechanism - with standard ETL tools doing the more complex transformations before loading the data in the data warehouse. The ETL tools need to be EAI-aware, and should able to handle small amounts of rows that are frequently being fed at high speed into the data warehouse.
The latest trend in information exploitation is XMLA (XML for Analysis). Already various vendors that are part of the XML for Analysis council are inter-operating. XMLA is positioned to become the standard API/Protocol for various front and back-end systems to communicate seamlessly with each other. Instead of publishing a schema, a data provider would publish an SMLA interface, and other analytic front-end applications will be able to issue standard calls to the data provider using XMLA.
On the vendor side, we see a lot of OLAP integration into the major DBMSs - such as Oracle9i with OLAP built-in, IBM DB2`s cube-view (code-named Aurora), DB2 OLAP Server and Microsoft SQL Server`s Analysis Server (especially on 64-bit architectures where cubes with up to 64 million dimension rows are now possible). Microsoft is fast becoming a major player in the data warehouse field with 64-bit SQL server benchmarks showing faster and cheaper responses than equivalent DBMS`s on Unix. The ETL vendors are widening their product offerings as the DBMS vendors start shipping ETL tools with their base product. Some ETL vendors are starting to play in the end-to-end market, including data cleansing, data profiling and even front-end capabilities. More are offering EAI capabilities, to interface with existing EAI technology, making a message queue the starting point for the ETL process.
However, proponents of the Kimball dimensional model bus architecture and the Inmon 3NF EDW architecture agree that metadata is the most neglected integral success factor for any data warehouse implementation. Both agree on the importance of metadata in well-defined and mature data warehouse- and information architectures. As yet, no single vendor covers the whole spectrum of metadata, as required in a full data warehouse architecture. This is partly due to the lack of an accepted metadata. The CWM (Common Warehouse Meta model) is positioned to become that standard. As this matures, metadata tool vendors will lose differentiation in the marketplace as they start supporting a single standard to store and interchange metadata.
"Using the TDWI conference as a barometer of data warehousing as it is done in SA, nothing that I saw at the TDWI conference made me believe that we have any reason to stand back to the rest of the world," said Winckler. "Certainly at Prescient Business Technologies, the best practices we have adopted are well aligned with the best in the world. Technology-wise we may need exposure to a wider variety of tools, but as the younger vendors mature and move into the international market, we may have a wider selection of products available. The products that have made it into the South African market are certainly staying ahead of the game in the North American market."
For more information on Prescient Business Technologies, visit www.pbt.co.za.
Share
Editorial contacts