Upcoming Course Announcement: Register now for QA Labs' two-day Quality Assurance course presented through the University of British Columbia Continuing Studies on October 30th and November 7th.

Pot-holes on the Road to Automation

Introduction

Testing costs can be a significant part of the project with software project managers spending up to half of their project budget on testing. But how do you make testing more cost effective so that you are getting more done with less?

One effective solution is automated testing or "tool assisted test activities performed with the objective of evaluating the software against pre-defined results/expectations that require no operator input, analysis, or evaluation." [QA Labs Inc., 1999]

Many people try to add automation to their projects, only to end up frustrated, and annoyed. After one or two disastrous attempts, many just give up and stop trying. However, implementing automated testing is a basic cost-benefit analysis.

It is well-recognized that an automation undertaking will require significant investment up-front before actual savings can be experienced. At the same time, the positive effects of having automation can be experienced by the organization in advance of the anticipated actual break-even point.

To help approach the undertaking of automation from a realistic perspective, keep in mind that Automated Testing is not:

• Immediate effort reduction
• Immediate schedule reduction
• A silver bullet to find all the application defects
• Automatic Test Plan Generation for 100% test coverage
• One tool fits all and is easy to use
• Cheap to implement

The savings and benefits will come, though, if you recognize and plan for the following.

Avoiding Pot-holes

The following are some typical issues that those implementing test automation run into:

• Pot-hole: Test automation is not treated as a project with proper project planning and design
  Solution: Treat test automation as you would a development project and manage the scope, resources and schedule appropriately. Implement a pragmatic approach to testing such that: the project can be decomposed into modular, defined tasks with assigned resources and timelines; others can easily carry forward the process that has been defined; the effort and results are quantifiable; each test cycle becomes more efficient in uncovering defects; and the most critical test types and application functionality are targeted.

• Pot-hole: No reusability (use of functions and utilities) in automation scripts
  Solution: Implement an effective test automation framework through abstracting navigation, data access, verifications, reporting, and other common functions into libraries to modularize scripts thereby minimizing maintenance costs as there are changes to the application functionality.

• Pot-hole: Testers untrained in programming techniques are assigned automation tasks
  Solution: Testers performing test automation must be able to create and maintain automated test scripts. This requires strong knowledge of software development practices, experience with procedural programming languages, and experience with the test automation tool to be used.

• Pot-hole: Automation test suite is not maintained
  Solution: Test suites need to be maintained with each new build and release of an application. Maintenance of robust scripts typically requires ~10% of the time of originally creating the automation scripts, assuming an automation framework is firmly in place and that major additions or redesigns are not being done to the application.

• Pot-hole: Testing is typically performed at the end of the project life cycle
  Solution: The test process should begin where the development process does, at the beginning. Moving testing up the life cycle increases the ability to find defects sooner and provides more time for effective test planning, design, execution and tracking, and stability for automation.

Scripting Best Practices

Test Automation Scripts are software. And even if the current intended use is for testing the current project only, you never know where those scripts could end up, or for what other purpose they could be used.

You may develop a framework and individual scripts for one version of the product, but require subtle modifications for a version of the product customized for another customer.

You may have large automation effort ongoing, including potentially multiple applications or versions of your product and have multiple people modifying the core framework scripts.

To help in succeeding with an automation undertaking, keep the following best practices at the front of your mind:

• Document
• Manage expectations
• Keep in mind the overall project testing goals
• Plan to control progress and recovery of the execution of the testing
• Control the scope of the automation
• Use coding standards when writing your scripts
• Version Control is important
• Get early feedback
• Test your scripts!

Summary

"... when all the pieces come together - the right people, the right processes, the right time, the right techniques, the right focus - then we can achieve truly impressive returns on our testing investment." [Investing in Software Testing, Rex Black]

If implemented thoughtfully, the automated test suite will prove to be much more efficient than manual testing in terms of hours spent and defects uncovered in previously manually tested functionality. The automation suite can be left unattended to run at night, on weekends and holidays. The tools never get bored or tired and never assume the application/architecture works while emulating as many users as needed, accessing the application and performing any mix of transactions desired.

Therefore, the return on investment (ROI) of automated testing can be tremendous, as long as you can avoid the pot-holes along the way.

[Reference - "Test Automation - Pot-holes, Best Practices, and Cost-Benefits", QA Labs, 2003]