Each of us has very likely had to do an estimate in the past, whether it was for a set of assigned tasks, for a project, or for an entire organization. As a tester, the question is commonly presented as, "How long will it take to test this product - and what resources will you need?" and then the person asking stands there, likely somewhat impatiently, and waits for the answer.

One common approach is to not base test effort on any definitive timeframe. The testing simply continues from when the code is ready until some pre-decided timeline set by managerial personnel is reached. Another common approach is to estimate testing effort as a percentage of development time. Development effort is estimated using some techniques such as Lines of Code or Function Points and the allocated test effort is derived using a pre-determined ratio. Both practices rely heavily on the test team's ability to work to the strategy and uncover the significant defects up front. However, with this approach there is little ability to invest in planning the test effort or creation of tests. These methods are not based on any assessment technique that takes into account the additional complexities of the test effort, such as deployment configurations and human language support.

As noted by Capers Jones in Assessment and Control of Software Risks, most projects overshoot their estimated schedules by anywhere from 25% to 100%, but some few organizations have achieved consistent schedule-prediction accuracies to within 10% and 5%. Just as it is critical to offer something more than an off-the-cuff answer for the development activities, it is important to know how to perform an estimate of the testing effort for a project.

A good simple definition of an estimate consists of a description of the size or scope of the undertaking, the level of confidence or uncertainty in the estimate at the time that it is made, and a description of the technique used to arrive at the estimate. "It is very difficult to make a vigorous, plausible, and job-risking defense of an estimate that is derived by no quantitative method, supported by little data, and certified chiefly by the hunches of the managers," according to Fred Brooks in The Mythical Man-month.

Getting a Starting Point

The basic elements to consider when performing an estimate for test effort is the size of the system to be tested, the components of the system, the quality requirements for each component, the resources available, and the level of productivity of those resources. From these elements we are first able to determine the overall size of the test effort in terms of test cases or verification points (eg: within a Use Case). Then considering the resource availability and level of productivity the total effort and schedule can be determined.

Regardless of the uncertainties and risks that may come into play at the beginning or during the project, we still need to get a starting number around which we can base our estimate. A well-defined requirement or specification, being a structured document (or documents) that likely follows certain standards in authoring and describes the scope of the system to be produced, is the best source for producing an estimate. These qualities allow the system to be sized using a variety of techniques that can quantify both the systems' functionality and its complexity (eg: performance, stress, security, and other non-functional requirements).

With an algorithmic approach to generating an estimate, the first step is to enumerate the collected requirements. If a requirements style guide has been used it should be easy to identify the number of requirements captured in the text. Also consider the number of screens and input fields for each. You may find it useful to group the requirements by type: imperative, weak phrase, list, etc. and weight them with the number of estimated tests for each type. Next further group and weight the requirements by complexity and the ability of developers to implement. Here you can make use of historical information for the organization or team from past projects - perhaps look at bug counts for the modules that are similar to those in your project or the estimate overruns for the different phases of the project.

That sounds straightforward and simple; just count the requirements, group them, and apply a set of formulae. But is it that easy? As Steve McConnell comments in Rapid Development about the accuracy of the first estimate of the project, "Some organizations want cost estimates to within plus-or-minus 10 percent before they'll fund work on requirements definition. Although that degree of precision would be nice to have that early in the project, it isn't even theoretically possible. That early, you will do well to estimate within a factor of 2."

More Than Test Execution

Depending on information and time available, your formulae can be made increasingly complex to factor in different influences and trade-offs in scope, resources and schedule. As more understanding of what influences your estimates is gained and more iterations of the estimate are completed you may find your model increasing in sophistication; similar to the increase in understanding gained between Dalton's and Bohr's atomic models.

However, you can still rapidly complete a list to account for all the activities you perform in a project that are related to actual test execution such as:

• Reviews of requirements and designs
• Test strategies and test plans (including test cases)
• Test analysis/matrices and test data preparation
• Test automation

These "overhead" factors to test execution depend on the quality requirements and extent of investment in upfront planning. The percentage of effort as it relates to test execution can often be directly tied back to the number of test cases or verifications calculated earlier and therefore be 'formularized'.

Uncertainty Factors, Multipliers, and Other Influences

Counting the requirements and applying formulae is certainly the basis of the approach, however there are a number of uncertainty factors, multipliers and influences to be considered when examining the project for test effort.

• Are requirements, designs, plans, and so forth available and are these documents clear, concise, and accurate?
• Do project stakeholders have realistic expectations in terms of schedules and functionality?
• Are there clearly defined milestones during the project for testing? (eg: Alpha, Beta, Gold)
• How well managed are the change control processes for project and test plans, requirements, designs, and code?
• Does the project team have the skills, experience, and tools needed for this project?
• Is the project team established or is there expectation of ramping up or turnover during the life of the project?
• To what extent can the project re-use test assets from previous projects?
• What is the required investment in the test environment set-up and maintenance?
• Have meetings, vacations, and sick times been built into the schedule?
• How many builds are planned to be delivered to testing? What if there are additional builds required or what if one is delayed?
• How many deployment configurations are to be supported and need to be tested? Do all of them need to be tested to the same degree?
• How many human languages are to be supported? Are special skills required for this type of testing?
• What amount of non-functional testing is required or planned?

All of the above uncertainty factors can be mitigated through upfront planning and investment. But if you don't have the time to address training issues, requirement reviews for clarity and testability, or change control standards make sure to take this into account when considering the certainty of your estimate.

Finally, don't just have one person do the estimate. Discussion of differences in numbers can make visible and clarify assumptions or advantages of approaches. Don't forget to include a level of confidence in each phase of the estimate and the final overall number. A Guide to the Project Management Body of Knowledge (PMBOK), Project Management Institute defined an estimate as: "An assessment of the likely quantitative result. Usually applied to project costs and durations and should always include some indication of accuracy (+- x percent)." As an estimate is refined as the project progresses, the effort may change but the confidence level should rise.

Summary

Benefits of Quality and Costs of Quality are in a balance and it is important to find and commit to the right equilibrium when facing the continuing challenge of not enough time for testing. There are many approaches to estimation of the test effort of a project and though the outlined approach described above is in no way rigorous, it offers the ability to approach the task in a systematic manner with a defined technique and supporting data - a significant practical advantage over ad hoc techniques that allows further research and experimentation to improve the methods used to arrive at valid estimates.

For more information...

QA Labs' test experts construct effective test strategies and realistic estimates tailored to your timeframe and resource realities. Our standards and best practices have been honed with experience on hundreds of projects. QA Labs paves the way for immediate and on-going quality improvements by establishing critical test artifacts, practical automation - all while getting those bugs logged.

Our clients hire us knowing they have hired a company, not an individual or group of individuals. We bring our comprehensive experience to the table and can provide the exact resource skill sets needed at exactly the right time. QA Labs offers a clear, cost-effective, alternative to hiring individual contractors, and even to hiring new permanent full-time employees.

Contact us at services@qalabs.com or visit our website for more information on our QA and Testing services.