Integration blues?

With hindsight, the 1990s was the decade for implementing packaged business applications. With foresight, the first decade of this century is the decade for integrating those applications, enterprise-wide.

This fact is what is driving the unprecedented investment in technologies that can help companies extend the value in existing systems; take these systems to the Web; integrate systems to those of other companies; and clean up the data into a common and consistent format for analysis and manipulation.

With the alphabet soup of acronyms in this sector of the market seemingly thicker than ever, there is significant confusion.

Johan Cloete, CEO, GBI

To put the size of this market into perspective, various analysts such as Gartner, Meta and International Data Corporation (IDC) estimate its value at more than $15 billion a year, and growing rapidly. IDC makes enterprise integration its second fastest growing market sector, after the Internet!

But, with the alphabet soup of acronyms in this sector of the market seemingly thicker than ever, there is significant confusion. (Consider, for instance, a market ruled by acronyms such as ORB, Corba, COM, DCOM, MOM, MQ, J2EE and more, and you can understand why some corporate customers are not easily able to make a choice!)

There has been a significant blurring of the lines between ETL (extract, transform and load) and EAI (enterprise application integration) technologies. This blurring can lead to customers making the wrong buying decision, or holding off due to indecision. So I thought I would take some time out to consider the two options, and when each is indicated.

In theory, process and data integration across applications provides broader access to accurate, consistent and complete data by employees, suppliers and customers, resulting in more efficient operations, more satisfied customers, and faster, more effective decisions.

In line with this, companies are moving large data sets between applications, and performing complex transformations on the data, a task ideal for ETL, but not always for EAI.

ETL tools were originally designed for building data marts and data warehouses, and updating them in batch mode with a predetermined frequency. Some have since evolved to meet the requirements of enterprise application integration.

These new-generation tools have been termed ETL++ (batch ETL + real-time + packaged integration). They provide capabilities considered to be the reserve of EAI tools, such as bi-directional adapters for leading packaged applications, guaranteed delivery, and real-time data movement.

The role of ETL++ for application integration both complements and competes with EAI.

As ETL++ tools progress beyond core batch data warehousing, the choice as to when to use EAI or ETL++ has become confusing. Gartner concurs, and has produced a report which poses the question: "Integration brokers and ETL tools: is the line blurring?"

It`s no longer just a question of ETL for batch/bulk data movement, and EAI for real-time. While you`ll still use EAI more often than ETL++ for real-time integration, the new generation ETL++ tools also cope with real-time data movement.

Understanding that ETL++ technology uses some of the same technology as EAI to provide real-time application integration helps to overcome the confusion between EAI and ETL. The basic underlying technology for EAI, message-oriented middleware (MOM), uses store-and-forward (queuing) technology to guarantee delivery of messages. Both technologies make use of MOM: EAI builds workflow and process integration on top of MOM; ETL++ builds data integration on top of MOM.

A simple definition for MOM is that it provides guaranteed once-only message delivery. You provide a message to MOM, it places it in a message queue, and then makes sure it gets where it needs to go.

Performing data synchronisation, interactive processing and multi-step processing requires a mix of all the technologies mentioned so far: ETL++, EAI and MOM. ETL++ performs many operations by itself, but it relies on MOM in certain circumstances.

MOM provides EAI and ETL++ tools with once - and only once - guaranteed delivery, in addition to features such as publish and subscribe or broadcasting. The difference is the graphical application built on top of MOM.

EAI workflow products provide graphical development and management of workflow and business process management on top of MOM.

EAI uses MOM for interactive processing, most distinctively when large numbers of transactions or one-to-many or many-to-many distribution is required.

ETL++ uses MOM for guaranteed delivery for interactive processing. ETL++ on its own handles data synchronisation and certain interactive processing scenarios (plus tasks traditionally handled by ETL such as batch and real-time data warehousing).

Gartner notes: "More than 80% of companies which lead their respective industries in revenue growth during 2002 to 2004 will have implemented a real-time enterprise nervous systems (ENS) for integrating applications within and outside the enterprise." (The Enterprise Nervous Systems arrives. Gartner, December 2001)

As for when to use EAI or ETL, Gartner boils it down to bulk data movement versus real-time: ETL for batch, information brokers (EAI) for real-time. However, bringing ETL++ technology into the equation modifies the conclusion, in that enterprise nervous systems for many companies will rely on a combination of ETL++ and EAI.

Bringing an ETL++ tool into the mix adds capability beyond that which can be provided by traditional ETL. It enables enhanced integration productivity and performance for the batch and real-time data integration tasks of data synchronisation and interactive processing above and beyond what EAI can offer.