User acceptance testing in BI projects

The importance of proper user acceptance testing in business intelligence projects cannot be stressed enough. This is the step when technology and business really meet for the first time.

Any business intelligence project is in effect a massive collaborative effort between business and IT. The business has to drive the project from the sponsorship, ownership and requirements points of view. IT has to drive the process from the technology and methodology angles.

During requirements specification and report prototyping it is a collaborative process for the business to specify what they require and for IT to document it in technology-implementable terms. However, it is during the user acceptance testing process that the rubber really hits the road.

IT is expected to perform unit and integration testing as part of the BI development methodology. They have to ensure each of the BI solution`s components - be it the data warehouse, the ETL components or the reporting and analytics components - operate correctly in isolation and also when integrated into the larger integrated and to-be-automated processes.

However, these are very mechanical and technical-level processes. Numbers of rows read and numbers of rows written have to correspond. Source system cross totals have to tie up with data warehouse cross totals and data warehouse cross-totals have to align with report cross totals. However, nowhere in this process is any business interpretation or business view correctness and sanity check tests applied. The IT personnel simply do not have the business insight to perform that on their own.

Enter user acceptance testing (UAT). This is first time that the result of the development effort is scrutinised for any real business meaning and business-level correctness. Yes, sure, the rows read and written may tally up, and the cross totals may match, but if the real business-focused measurements do not make sense, then the underlying implementation, how technically correct it may be, is not of any value.

It is during UAT that the users for the first time see for example, net written premium, per company, per line of business or per client, in real terms. Similarly, it is the first time they see gross sales, VAT paid, commission paid, etc, in real unit and monetary terms. Business users may not know or relate to BI concepts and processes, but one thing is sure, they know their business! They will very quickly identify measures and counts that are out of line. They know very well when the profit figures before or after commissions are wrong, or whether the units shipped per product line are accurate.

UAT is an iterative process. Do not expect to have a single session with the users and expect to walk away with a signed off delivery certificate. Especially in the first phases of a BI project, there will be comebacks; there will be corrections and adjustments. Make sure on the project plan that there is ample time and budget for a few iterations of the UAT process. Make sure the users understand this process and are committed to work together through the ups and downs of UAT. Not managing their expectations can lead to serious negative impressions, especially during the first few iterations of a new project, when everyone is still unfamiliar with the processes and data.

Such negative impressions are often very difficult to set right - one negative impression takes 100 positives to cancel it. However, managing the expectations and process well from the beginning can assist in a well-structured close-knit teamwork process, where the adjustments and corrections can be streamlined to speed up the process.

UAT is a crucial process in any BI project. It is the first time when business and IT together really see the results of a lot of combined and separate efforts. It is the process through which the necessary corrections and adjustments must be made to make the difference between project success and project failure. To ensure this takes place, sufficient time and resources must be scheduled up-front and everyone`s understanding and buy-in must be obtained before the process is kicked off.