One of the key components in Cach'e 5, the latest version of the new post-relational database from InterSystems, is bitmap indexing. Henry Adams, country manager for InterSystems South Africa, analyses the new technology and its implications for the market.
Real-time business intelligence has long been the Holy Grail of vendors and customers in this market. They need it for a variety of reasons:
* The ability to obtain a view into critical business data while it is current enough for meaningful decisions to be made against it.
* The need for management to be able to align its business intelligence systems with customer-facing systems such as customer relationship management (CRM), and partner-facing systems such as supply chain management (SCM) and collaborative commerce. These new-generation systems operate in real-time, and decisions made against them and information delivered from them to customers and partners should also be real-time in nature.
Another goal of business intelligence customers - but this time not necessarily vendors - has been to use the same database for recording transactions securely and for management information systems/decision-making. Both goals have proven beyond the scope of traditional databases. The reasons given to date are simple:
* InterSystems holds the view that transactional systems need to be protected at all costs. Under no circumstances, so conventional wisdom goes, can core business systems and their associated databases be put at risk by running complex queries while records are being updated. Accordingly, data needs to be streamed off on a regular basis, and made available for query in a separate database.
* This separate database has many of its functions disabled - such as the ability to update, along with two-phase commit, rollback, stored procedures and triggers. Yet, while these functions may not be invoked, they are paid for.
The upshot of this situation is that customers cannot have real-time business intelligence, even if they want it; they can only view and review historical data; and they must have two infrastructures, with all the attendant cost in servers, database, administrators, analytics tools and more.
There has to be a better way: customers want it, and technology can deliver it. The answer, InterSystems has decided, is bitmap indexing, a technology which has been around for some time but which, conventional wisdom holds, is not suitable for OLTP systems.
Bitmap indexing is an alternative to standard indexing. In standard indexing, the indexed value is linked back to those rows in the database table that contain that value. In bitmap indexing a record of bit values is created for each value of the element being indexed. Each bit in the string indicates whether the row pointed to by that bit has the element with that value. For example, if you have a column `Colour_Of_Hair` then for the value `Red` you may have a bit string that looks like this "0111000110" which indicates that rows 2,3,4,8,9 have `Red` as the value for `Colour_Of_Hair`. So the query `select count(*) from Table where Colour_Of_Hair=`Red`` only requires a bit count of the relevant bit index string in order to return the answer 5. Needless to say, this is significantly faster that counting the individual entries in a standard index.
This can speed up certain types of queries by an order of magnitude.
Bitmap indexing - which has been around for some time but is not used in OLTP systems - offers an intriguing option. In a metadata-like manner, it uses individual bits to point to live data, allowing interrogation of the database without the burden of trawling through vast amounts of irrelevant data. And because not all the records in the database are being accessed, there is not the traditional overhead and performance drain associated with the query process which would hamper a transactional database.
InterSystems` research into bitmap indexing has seen it re-engineer the concept to the point where it can be deployed in a hybrid transaction-query situation.
The key is to ensure the bitmap indexing is fast and nimble enough for deployment in transactional systems. And so it has proven: benchmark tests on Cach'e 5, the latest database from InterSystems, indicate significantly enhanced performance relative to its predecessor, even given the presence of the bitmap indices.
The implications for customers are intriguing: major cost savings, greater agility in reporting and analysis, and a tighter integration with customer and partner-facing systems.
Share