Traditional extract, transform, load (ETL), or ELT (extract, load, transform) approaches no longer meet the needs of data-driven businesses, particularly where data is dispersed, and organisations move to harness AI in more of their processes.
The old ways are too slow, rigid and cumbersome as data volumes explode and organisations require access to real-time data, faster, and in any format.
Changing data needs are driving the emergence of a new approach to making data readily accessible using a virtualised data layer to expose data in a ‘data marketplace’. Instead of traditional data lakes and data warehouses designed to bring data together from various sources, data virtualisation connects to data where it resides and makes it available to users in an agile and open data marketplace.
Data virtualisation combined with the data marketplace concept exposes business-friendly models across multiple access channels without moving or copying data.
Flexible, open data self-service
In traditional approaches, ETL developers must connect to a source, copy the data and run the job − spending a great deal of time simply watching loads, watching data move, testing and checking that it's correct. With a virtualised data layer, nobody wastes time watching data move.
Data virtualisation establishes a single data access layer for finding and using all enterprise data from physical data sources like data warehouses, data lakes, transactional and analytical databases, cloud and enterprise applications, APIs and data files.
With a virtualised data layer, nobody wastes time watching data move.
With a centralised, logical layer and management, data integration and delivery costs are reduced, plus efficiency and agility are improved. Organisations quickly gain a unified view of data from disparate sources, and faster data delivery in the format each consumer needs.
In this environment, users can run queries, request access to data, and perform ad hoc exploration directly within the interface, without needing IT intervention.
Ideally, users should also be able to see whether a dataset is already used in other reports or dashboards in tools like Power BI or Tableau, enabling them to reuse these instead of starting from scratch. They should also be able to trace lineage and data ownership to understand where the data came from and how it’s been transformed.
Whether users are analysts, data scientists or business leaders, they need access to curated datasets and data products that are trusted, governed, searchable and presented in business-friendly terms. The data products should comprise rich metadata with business context, while security, access controls and compliance are seamlessly enforced in the background.
The new online shopping-style data marketplace meets their needs, offering a unified and open semantic layer that maps business concepts to underlying systems and publishes them via ODBC/JDBC, REST, GraphQL APIs and much more. This virtualised model supports open access via any tool and data retrieval method, avoiding the need for bespoke API coding and centralised ETL pipelines.
Simplified data management
By adding a virtualised data layer to enable a data marketplace, organisations can streamline data management across complex hybrid data environments, and various departments, business units and branches.
They can also avoid the duplication of datasets, with the associated additional costs and complexity of this. For governance, risk teams and auditors, the virtualised data layer simplifies control, management and monitoring, plus there’s only one layer to audit.
No matter how complex the environment, the data virtualisation layer can act as the glue that connects and exposes everything, giving organisations the insights they need − whether it’s a single view of operations, or a 360-degree view of their customers.
While many users still focus on the tools for creating reports, they overlook the fact that generative AI will soon be creating any report needed and directly from the data. This means the data itself is critical for the data-driven enterprise of the future and access to data should be their primary focus.
Therefore, organisations need to go beyond serving reports; they must enable business to self-service using data products that democratise data.
There is also a lot of focus on bringing data into one pot and then analysing it. But very little effort goes into making the data truly accessible, in a meaningful and ready-to-use format.
By transforming datasets into products that can be used by anyone, with whatever front-end tool they wish, a data virtualisation allows organisations to take raw data ingredients and transform them into useful data products that can be access in a data marketplace. This latter is akin to finished ‘meals’, which can be easily searched for, retrieved and consumed as needed by end-customers.
Share
