Testing – The Neglected Child of our Industry
Testing seems to be a sore point in most development organizations. There never seems to be enough time, nobody knows how much testing is enough, and once the application is shipped and some bad bugs show up, the test team is blamed for not having done their job. Testing has been a problem for software development organizations for decades. Why is that?
There are many reasons and they range from subtle organizational and social issues to hard core technical issues. This paper describes some of the most frequently observed sources for test related problems. I will then propose solutions to address each of these problems.
Reasons for poor testing
- Lack of planning is arguably the number one problem leading to insufficient testing. At the beginning of the project, a lot more thought is normally invested in estimating the effort for design and code than into planning a detailed test strategy. This problem gets compounded when design changes are made at some later point in the project. It is quite common that corresponding changes in test effort are not considered.
- A more insidious problem is created by schedule overruns. Most projects work against a fixed deadline for delivery. The deadline may be imposed by a customer, a trade show, or similar. It typically is very difficult to move that end date and in some cases severe financial penalties or opportunity costs are associated with a slip. When design and development activities overrun (and they almost always do), testing gets squeezed between the end date for development and the immovable delivery date.
- Related to the above phenomenon is the widely held view that the job is done once coding is complete. Some feel that testing is only there to catch problems that were introduced by incompetent programmers. And since most companies are proud of their programmers, the feeling is that testing is more of a luxury or a safety net rather than the integral part of the software development process, which it is.
- Testing is often (rightly or wrongly) much shorter than design and development. Accordingly, test resources tend to be required during certain peak periods and there may be little or no need for testing staff at other times. Especially small and mid size companies find it difficult to keep a staff of dedicated test experts. It is more convenient to use development staff for testing. Developers are shifted from coding to testing which makes perfect sense from a staff balancing perspective. It is, however, a frequent source of quality problems. First of all, developers see testing as a lowly job compared to design and development. They are not very motivated and try to get the job done as quickly as possible (and therefore are unlikely to argue that the testing period is too short). Second, it is ineffective to ask somebody who coded a piece of software to then try to find errors in it. Many companies try to avoid the problem by allocating people to test code they did not write. But these developers are still part of the development team that, as a group, created this application. Even with the best of intentions, they tend to test along the same thinking path which introduced a defect in the first place.
- While development occupies a good part of the Computer Science curricula at Universities, testing is hardly ever mentioned. Most companies send developers to a variety of training courses but testing courses are much less popular for some of the reasons mentioned above. There is also a considerable shortage of good test training.
In the following I will argue that outsourcing the test function or using external contractors can solve all of the above problems. Obviously I am biased, but I hope that the logic of the argument is appealing.
The Solution Space
- One of the most powerful arguments for external support is the opportunity to involve experts early on in the project life cycle. Test experts can give feedback to the development team about test requirements, testability of their specifications, and a reality check for the test plan and schedule. This early involvement reduces time for test planning and execution considerably.
- Another argument for external test resources is the desire to obtain an objective and independent view of the product quality.
- Traditionally, the software industry assumes that testing is about testing code. In reality, all artifacts that are created during a project are testable. A simple example is the user manual (it is frequently overlooked when it comes to testing). Testing requirements specifications and design documents is a powerful technique to catch errors early on and move the cost of rework to the less expensive part of the cost-of-rework curve.
- There is a limit to how many design activities can be performed in parallel. But testing is a perfect candidate for schedule reduction by starting on test case design during the design and coding phase.
- Not many companies specialize in the provision of testing services. But those that offer the service made a conscious decision to make testing their profession. That means they train their staff to stay at the leading edge of test technology. The people they hire are interested in the subject. They thrive on breaking things (i.e. crashing software). That sounds scary but it is much better to have a tester crash your app than a customer.
- Test tools are sophisticated pieces of software. They are also expensive. To decide if and what test tools to buy is often a difficult task. Test contractors are in the business of knowing what test tools work well in certain situations since they work with these tools all the time.
Summary
Here is a simple test: do you have problems with the testing activities and/or do you feel there are too many bugs that show up after product release? If the answer is “yes”, the next question is: do you want to hire a full time person to address these problems, use one of your own scarce software developers and detract them from developing the next release to investigate the test problems, or do you want help from a contractor who can give you focussed experience in as much or as little time as you want to allocate?
© 1999 QA Labs Inc.
Author: Wolfgang Strigel
Email: strigel@qalabs.com
Back to Resources.
