Swimming through murky waters: Implementing proof of concepts

Implementing a business intelligence proof of concept is very similar to swimming through deep murky waters - there are many big things lurking there that can bite you in the back, some with serious consequences.

The first thing we have to establish is why we are doing the proof of concept. Is it a proof of product or a proof of concept?

The former is a technical evaluation whether a particular product set has suitable functionality to implement a desired solution, and whether it is stable, easy to use, efficient, etc. In this case, the evaluation is driven from a technology point of view, and the functionality and characteristics of the tool are evaluated.

The second is a business-focused evaluation to determine the value of the solution in business terms - in other words, whether we will get a suitable return on the investment should we implement the solution. In this context, the emphasis is on business value - ie, if we implement this solution, how much more profit will we make, or how much costs can be reduced.

It is very important to distinguish what the proof of concept is set out to prove - the product or the business concept. The former is relatively straightforward, and can be done on smaller and easily managed subsets of data. The second is often quite complex, as often the economies change as the scale increases, and justification is hard with a small subset of data - which can make the exercise extensive, time-consuming and costly.

In the process of managing all expectations - of the supplier, the technical stakeholders and the business - it is important to specify the scope and outcome of the exercise up-front. It must be phrased as a definitive conditional: if the product meets at least 85% of the given functional criteria, we will acquire 12 licences; or, if the solution decreases our marketing spend by more than 12.5%, we will employ it for the next three years` marketing drives.

By doing this, all stakeholders, including management and the vendors, will know what is being evaluated and what the corresponding outcome would be. It doesn`t leave holes open for interpretation and possible future misunderstandings.

It is very important to establish up-front what the budget for the exercise is. The budget can severely restrict the time spent, the depth (level of detail) or the breadth (number of aspects evaluated) of the investigation.

In order to avoid cutting corners halfway through the investigation in order to meet the budget, the financial implications must be out in the open before the exercise is even attempted. Having a fixed budget or rate will then determine what is feasible and based on what detail a call must be made.

As mentioned before, it is often very hard to determine real business benefit with a low budget proof of concept.

There is a massive difference between a proof of concept and a pilot project (or a prototype). With a pilot project or prototype, the road is being paved for the future real project or implementation. In such a case, we spend more effort on the design and implementation to ensure it is strong, flexible and robust enough to be taken into the future - and the business expects to pay more, as it is an investment in a future implementation. But a proof of concept is just a single once-off test. The price we pay for a quick turn-around or low budget investigation is that we have to (re-)do it properly, before we can build anything substantial on it. The proof of concept is merely that - a simple test to determine whether something works or whether it has business benefit.

Organisations must expect that when a proof of concept is completed successfully, it merely buys the route to go forward - the road must still be constructed. It would be crazy to try and force the quick-and-dirty proof of concept into a permanent solution. Pritt and bubblegum can only withstand so much pressure!

When attempting a proof of concept, especially in the business intelligence space, it is very important to set the expectations right up-front: what is being evaluated (product or concept), what is the scope and budget of the exercise, what happens based on the outcome, and most importantly, how it will be taken further in a real production environment.