Extraction, transformation and loading (ETL) is one of those areas of IT that few people know anything about, but that every company with even a small data warehouse or data mart relies on. It is also a fiercely competitive market in the process of consolidating, but not yet dominated by one major player.
ETL is defined as the process of moving data from transactional and other data stores into a data warehouse. As the name suggests, far from merely transferring data from one source system to another, ETL consists of extracting the data from the operational source, transforming or cleansing it to achieve accuracy and consistency, and then loading it into a warehouse.
In the past, large corporations used to develop their own ETL code, but they quickly realised that this was a never-ending, thankless job: the ETL scripts constantly needed updating because of changes to source data file formats, validation rules and the addition of non-traditional data sources.
It is simpler to leave the coding of data access technologies to a third party, some of which also include products and services for augmenting cleansed data with external information, such as customer demographics or purchasing behaviour.
According to a 2003 ETL report by research firm CurrentAnalysis, the market for ETL products is growing rapidly and will continue to grow. "Data quality has been recognised as a major concern as the ramifications of `garbage in, garbage out` quickly become obvious in near real-time applications that require linking customer records across multiple, disparate database environments and there will likely be even higher growth in this market segment," reports CurrentAnalysis.
It also notes that the demand for data marts to support analytic applications and the use of data augmentation services to enhance customer records with demographic, socio-economic, and lifestyle data will also contribute to this growth.
As with most areas of IT, the ETL market is not standing still. Spurred on by intense competition in a slower economy, vendors are consolidating their offerings through additional products, many gained by acquisition.
CurrentAnalysis adds that many ETL vendors "continue to form alliances or partnerships with other vendors in order to offer end-to-end data warehouse solutions. Some vendors have even started to offer industry and/or functional data marts and analytic applications as part of their own product sets."
While Ascential Software is the leader in end-to-end solutions, CurrentAnalysis notes that the wave of acquisitions and collaboration will result in new rivals in this market space; Ascential will need to stay a step ahead of the pack.
The research firm has placed Ascential in the first tier of global ETL vendors. It also ranks the firm as "positive" in the areas of current perspective, momentum and vision. This has positioned Ascential as a clear leader.
"There are many specific areas of competition in which vendors are battling to outdo each other, one of these being the ability to access data from additional, diverse sources," says Julian Field, MD of CenterField Software, the local Ascential distributor.
"Gone are the days of simple accessing of data from other relational databases via ODBC," continues Field. "Vendors now need to offer solutions to access additional data sources such as non-relational databases as well as semi-structured data, such as spreadsheets, and proprietary enterprise applications."
Additionally, as Web services become a critical part of the corporate application and data landscape, the ability to access and transform XML tagged data is becoming an important enabler for Web services. Including the functionality to track and integrate Web site behaviour is also important, but it is not the competitive differentiator it was three years ago. Field also notes that enterprise information integration (EII) is emerging as a focus area for Ascential, and consequently other ETL vendors.
CurrentAnalysis says EII is driven by the recurring need in enterprise customers for short-term data marts that are created for specific projects and require historic and current data, possibly from operational databases and existing data warehouses.
As for the long-term future of the ETL market, CurrentAnalysis speculates that the focus on consolidation via acquisitions or alliances will continue.
"While we have divided the ETL space into data extraction, data cleansing, data transformation and data augmentation, vendors realise that customer solutions require components of each and have moved to expand the scope of their offerings to accommodate these requirements."
Additionally, as the lines between enterprise application integration (EAI) and ETL continue to blur - due in no small part to the need for near-time information - traditional ETL vendors are targeting the integration of these two markets in their product line-up.
"ETL technologies have played an important role in the enterprise for many years," Field concludes. "Far from backroom systems anonymous to the rest of the world, these systems are encroaching on new turf, expanding the reliability and trust inherent in ETL to EII, EAI and even the world of analytics. As product offerings expand to provide broader solutions, we can expect to see a dominant player emerge in the near future with an enterprise-wide suite of integrated data solutions."
Share
Editorial contacts