Why do we need metrics? Why should we care about measurements? As engineers, builders, or entrepreneurs, we try to create something that is as good as it can be. We strive for perfection.
Yet, perfection is a misleading ideal. We tend to delay the release of a product because it is not ‘perfect’. In theory, that makes sense. So, why do I call perfection misleading?
To define ‘perfection’ we need to understand what quality means to customers. If we do not have a clear definition of what a customer needs, then we lack a definition of quality. If we lack a definition of quality, then we can never reach perfection. To strive towards perfection, we must first strive towards measuring and validation. Note, I say strive towards, not achieve. Perfection is a direction and not an exact outcome.
It is very easy to get lost in the noise of potential metrics competing for our attention. We need a strategy to determine what exactly we should measure. What we measure should give us valuable insights to confirm or deny our assumptions.
I will give an example business case as well as an accompanying hypothesis. This example will serve as a reference point throughout the rest of the discussion.
You have an innovative idea for a new kitchen appliance rental platform. You believe customers want to rent an appliance, then return it after using it. You know that people do this with clothes and technology.
You believe it is possible to extend this to include kitchen appliances. You write your hypothesis based on these assumptions. “There is an interest in renting an appliance, like a coffee machine, for some time rather than buying one.”
Let us start by discussing the hypothesis stated in the example. The hypothesis is a nuanced statement directed at a specific experiment. Understanding the hypothesis is the first step towards determining what to measure. Based on the risk that we are addressing, we get three hypothesis categories:
- Desirability hypothesis: Risks related to appeal or desire for the idea. Are customers interested in the product?
- Viability: Risks related to the realistic viability of the idea. Can this business model make money? Will we be able to get a large enough market share?
- Feasibility: Risks related to the ability to execute the idea. Do I have the skills to build this platform?
The hypothesis I shared is an example of a ‘desirability hypothesis’. We want to confirm there is a sizable interest in the proposed platform. Now that we understand the hypothesis, we can get closer to identifying what we need to measure.
Perfection is a direction and not an exact outcome.
In our example, we need to measure something that can prove if customers are interested. Imagine having a website where potential customers first encounter our platform. On the site, there is an option to subscribe for updates on the development of the platform. What could we measure to confirm our hypothesis? Here are two options:
- The total number of people that visited the site over the last month.
- The ratio of people that visited the page compared to those that subscribed to total page visits.
Selecting the first metric is a common pitfall for an entrepreneur. The first metric is an example of ‘vanity metrics’. Vanity metrics look cool. They tell you about the ‘size’ of the business. They are flashy and make shareholders happy.
We might learn something about the quality of our marketing from vanity metrics, but we cannot draw conclusions, or gain knowledge, about the desirability of the platform.
Vanity metrics lack a cause-and-effect relationship. You look at the number of people visiting your site and think: “Wow, people love my idea!” But people stumbled onto your site while searching for an appliance to buy, or because of very good marketing. They visit your site, then leave. Vanity metrics are misleading. They are not suitable for hypothesis validation.
Compare this to the second metric. We call this type of metric ‘actionable metrics’. Actionable metrics help us learn. The behaviour of individual customers ties into actionable metrics. Visiting your site and subscribing to receive updates shows interest. Here there is a very clear cause-and-effect relationship. We can use the actionable metrics to confirm our hypothesis. We can confirm or reject the desirability of our offering.
However, this is an indication of interest and not a perfect guarantee. What customers say they want and then what they do when the offer is available may differ. The principle of measuring actionable metrics is an ongoing process. The lean start-up cycle depends on iterative development, measurements and learning.
Gathering actionable metrics is the key to continuous learning and hypothesis testing. When we confirm, or invalidate, a new hypothesis with an actionable metric, we take a step closer in our strive towards perfection.
Share