All of us typically do test estimation, but may not even realise we are doing it. We may, for example, think of a test to execute, and then because we judge that it would take about two hours to complete, we plan it for tomorrow because we only have an hour until we have to travel home.
Without consciously doing estimation we risk doing it badly; and we will find it difficult to answer most reasonable questions in a confident manner.
How many hours will you need to do the testing? How expensive will testing be? How much testing should we do to ship the product with less than five medium severity faults? Can you complete the testing in four days? Why do you need three testers? These questions all relate to test estimation. As testers, and particularly as test managers, or test consultants, we are required to know how to estimate testing costs, resultant product quality, and test effort and schedules.
What is a test estimate?
According to the British Computer Society International Systems Examinations Board Test Practitioner course syllabus, "a test estimate is an approximate calculation or judgement, typically based on the professional understanding of experienced practitioners".
At least three test cycles are required merely to reach all of the faults in the system for the first time.
Wayne Mallinson is chairman of Test and Data Services.
Test estimates are difficult to do because so many variables and dependencies have to be taken into account. Think about it, which would take longer to test: software from development with very few remaining faults, or software where - on average - 30% of the coded user requirements fail when tested for the first time? When the delivered application is of poor quality, testing can take much longer: some faults may be masked by others (initially not discoverable).
If a screen cannot be displayed because of a missing menu item, for example, then the tester cannot possibly encounter a problem on this screen until the first problem has been resolved.
At least three test cycles are required merely to reach all of the faults in the system for the first time. Apart from the testers taking time to identify and report the incidences arising from these faults, the developers need time to investigate and repair the underlying faults before the tester retests the failed application area and does regression testing.
To make matters worse, it is often difficult to repair underlying faults and so a percentage of faults are not properly repaired or new faults are created by the repair process, resulting in unexpected side effects.
The poorer the software quality (related to the number of faults) the more test, repair, and re-test cycles there will have to be to reach minimum software release (or exit) conditions. The more test cycles there are, the higher the number of regression tests that will need to be run.
Delivery
We can now understand how dependent test estimates are on delivered software quality. Another more obvious test dependency is the delivery date of the software. If software is delivered late it is not always possible to fit the required testing hours (effort) into the reduced schedule. One tester could easily do 100 hours of testing in three weeks, but even two testers may not be able to 100 hours of testing in two days!
The testing effort and schedules are also impacted by delays, or by poor quality of the test environment infrastructure. Internal testing factors - such as the effectiveness and efficiency of the test process, and the skills, experience and physiological state of the available testers - also impact testing estimates. Testing is often delayed by external causes and the testers may work at sub-optimum levels if they are over-tired, rushed or stressed.
Test estimation and test planning need to take these project and product dependencies - and likely resultant scenarios - into account to set realistic test expectations for all stakeholders. Test estimation would need to be revised whenever test planning changes as a result of changing project or product circumstances.
Now that we have covered exactly what test estimation is, issues surrounding when and why test estimation should take place as well as what it is we need to estimate will be discussed in future Industry Insights on ITWeb.
Share