In the world of business intelligence (BI), much has been said about architecture and designing the right solution to meet an organisation's specific needs. However, people tend to quickly forget about how fast the technology constantly changes, and as such, often don't always realise or understand the immediate impact that technology and the consequent change thereof may have on best practices for design and the modelling of spectacular technological solutions.
In the 1980s and early 1990s, BI solutions (then mostly referred to as data warehousing solutions), were heavily restricted by the availability and affordability of storage. For example, many of us grew up on the standard 'Sales' star schema - where in those days it was typically advocated to store Sales by Product by Day.
The problem with this kind of summarisation, however, is that it removes all the interesting bits of dimensionality that are only available at the transaction level. Luckily, hardware over the years became more affordable, allowing most retail BI solutions to be able to go down to the lowest level of available grain, which is the sales transaction line item. This has been possible since the 1990s, as a result of retail volumes not being typically high.
Other industries, however, are not that lucky. In the telecommunications sector, for example, the 'transaction-level' grain implies that one has to keep track of each and every call made by each and every subscriber. Some solutions even require the individual events that make up the call to be stored. For the most dimensional data, this is the correct level of storage.
Be precise
However, in the 1990s, and even early 2000s, it was not advocated to store and query this data at a granular level. This usage data was mostly only summarised. Best practice nowadays still suggests the usage summaries as the quickest way to get to common answers, but advanced analytics now requires more detailed level information to get an exact answer.
If one considers the average laptop, for example, it now has several hundred times more processing capacity than the Apollo spacecraft that put a man on the moon just 40 years ago.
Cor Winckler is technical director at PBT.
Examples of these include usage 'patterns' as analysed by high-end data mining solutions, which cannot function without access to detailed information. Considering that this is a never-ending stream of hundreds of millions of complex records every day, this often remains a challenge for most companies involved within the telecommunication space.
The surprising factor is that details, which as little as five years ago one would not have dreamt of loading or using, never mind processing and allowing query access to, can now be stored and processed effectively. This is due to the leaps and bounds with which the hardware manufacturers have, and continue to keep making capacity higher and more affordable for any organisation, no matter the industry they are involved in.
Dream come true
If one considers the average laptop, for example, it now has several hundred times more processing capacity than the Apollo spacecraft that put a man on the moon just 40 years ago. As a matter of fact, most modern cellphones have more processing capacity than the computing equipment that was on that same Apollo spacecraft. In fact, the first data warehouse in which I was involved in, in the early 1990s, could now probably fit on my current laptop hard drive, with some space to spare! This very same laptop could also be fitted, quite easily, with half a terabyte of storage. In the 1990s, users used to dream of being involved in the first terabyte data warehouse project. Now, most clients have databases that exceed 1TB - how the times continue to change.
The continued growth in storage capacity is, of course, linked to higher CPU counts (and speeds), more RAM that is made available on medium to large database server installations, as well as further sophisticated bandwidth, to transfer these large amounts of data. To put it into perspective, my first hard drive, which was bought in the late 1980s, was a full 10MB. I now have a hundred times more storage available on my iPod, as well as most memory sticks. That same PC had 256K of RAM. My laptop now has 2GB of RAM and a dual core CPU. The server that I recently conducted work on has an impressive 96GB of RAM, and 24 quad core CPUs (essentially 96 CPUs), and 15TB of storage, soon to be upgraded to 45TB of storage. This curve is a steep and rather exponential one.
The world is changing right underneath our feet, almost faster than technology users can keep track of. This increase in capacity certainly bodes well for the BI industry, as it allows for the tracking of aspects on a level of detail that was not possible a mere five years ago.
As trends come and go, it is hard to fathom where the technology industry will be in another five years. With continuous changes and developments taking place, it seems to me that the sky is no longer the limit!
* Cor Winckler is technical director at PBT.
Share