QA Labs brings you practical tips and tools on testing, quality assurance (QA), and related topics through this monthly e-newsletter.
This month's article author: Philip Yeo, editor: Trevor Atkins.
Automation - Free Tool Alternatives
Many companies desire to have a set of automated scripts testing their product. This could ultimately save money in the long run if done right. See our previous newsletter for more information: Potholes on the Road to Automation.
However, many companies, especially smaller development shops are unable to afford the high costs of a commercial automation tool. First there is the purchase price of the tool and the necessary number of licenses. These licenses can cost in the range of thousands of dollars per seat with the annual support costs as a significant ongoing expense. In addition to the cost of the actual tool there is the necessary training time, either with the implicit productivity costs of a tester learning the tool as they go or taking an official training course from the tool vendor.
It is therefore good to know that there are other automation solutions available. Many of these alternative solutions are free for use and even open source.
Types of Tools
If a person cares to look there is a multitude of free testing tools available. Below is a list of some of the different types that can easily be found:
- GUI Testing
- Web Testing
- Load and Performance Testing
- Test Suite Management
- Unit Testing Tools
- Test Coverage
- Defect Tracking
For a small testing department looking to setup and create an automated test suite the tools to investigate will likely be the first two items in the above list; tools to test an application through the user interface from an application or a web browser. These two categories of GUI tools can be largely broken down further into those that are used to test websites, Java applications, and Win32 applications. With the proliferation of Java applications and websites there is a reasonably large number of free automation tools that are geared towards them. Of course this leads to the potential problem that if the testing group is or becomes experienced with a tool for Java applications, a Win32 application that may require testing will not be able to benefit from this tool and expertise.
There is Free and there is Free
There are different kinds of "Free" licenses under which programs are released . These types of licenses go from "You may run the program" to "Here is the source code, please modify". The first type of "Free" allows the user to execute the program from a binary but the source code is not provided. Varying levels of "Free" may allow the program to be distributed without permission and minor modifications to be made to fine-tune it to your needs. The most "Free" type of tools are those with licenses that provide the user access to the source code and give the freedom to improve and modify the feature-set as needed. Some of these licenses may require that any changes be released back to the public so ensure no proprietary information or work goes into those tools.
Caveat: Before starting to use a tool make sure that the license agreement is known and understood by all parties involved and that any articles are not accidentally broken that may come back and haunt the testing team.
Pros of Free Tools
There are a number of advantages to using free automation tools:
- Cost: Perhaps the largest advantage of using free tools has to be the fact that it is free. It is easier to get the proper sponsors and go ahead if the initial overhead cost is kept low.
- Just what is needed: The testing group gets what they need/want in an automation tool. For example, if the testing group can do a lot of programming to add to the automated scripts, then maybe a relatively expensive full "Record and Playback" automation tool is not required.
- Personal Learning: The resource(s) using the free automation tool are learning new skills that make them inherently more valuable to the company and it is a skill-set they can take away with them and apply elsewhere.
- Open Source: Many of the free automation tools available have their source code accessible for all to see. This makes it possible for the more technical people in the testing group to further 'tweak' the tool to behave exactly as they wish.
Cons of Free Tools
Of course, there are a number of disadvantages to using free automation tools:
- No Record & Playback: For the majority of the freely available automated tools there is at best a limited "Record & Playback" functionality. If the testing group is less technical it is possible they will be unable to create scripts in a timely fashion.
- Little or No Support: The major tool vendors have 24 hour support lines that can be used to help solve any problems encountered. Getting help for free tools comes down to web searches, forums and emails to the creator. All of these avenues of assistance are hit-or-miss.
- Limited Documentation: Sometimes the documentation provided with free tools is limited and the tester has to learn how to use it by trial and error.
- License Jargon: If a free automation tool is used, the license agreement must be carefully read so there will be no surprises down the road if the use of the tool is ever reviewed or audited.
Samples of Free Tools
The following is a brief overview of a selection of free automation tools. For a larger list of free automated testing tools please see the page maintained by Danny Faught (http://testingfaqs.org/t-gui.html).
| Name: |
AutoIT |
| Type: |
Win32 GUI |
| Description: |
AutoIT is a freeware automated testing tool that executes BASIC-like scripts and interacts with windows objects. |
| |
AutoIT can: |
| |
Simulate keystrokes |
| |
Simulate mouse movements |
| |
Manipulate windows |
| |
Manipulate window controls directly (i.e. set/get text from a text box) |
| |
The scripts used by this tool can be compiled into stand-alone executables for greater portability across machines. |
| |
AutoIT does not have a recording tool so the scripts need to be written in a text editor but it does have an “Active Window Information” utility included. This utility is used to identify the properties of a window for scripting purposes. |
| Website: |
http://www.autoitscript.com/autoit3/index.php |
| |
|
| Name: |
Abbot |
| Type: |
Java GUI |
| Description: |
Abbot is a framework for testing java GUI components and improves upon the basic functions provided by the class “java.awt.Robot” |
| |
It comes with a script editor “Costello” that includes a “Capture” option allowing record and playback of scripts. It was easy to install this framework and quickly create an automated test script. |
| |
Once an automated script is created the tool allows users to insert items through a dropdown list. Items that can be inserted include (but not limited to): |
| |
Assertions |
| |
Actions (AWT Menu Selection, Button Press, Drag, Keystroke, etc) |
| |
Function Calls |
| |
Expressions |
| |
Comments |
| Website: |
http://abbot.sourceforge.net/doc/overview.shtml |
| |
|
| Name: |
Badboy |
| Type: |
Web Test |
| Description: |
Badboy allows the easy creation of scripts to test web pages. It comes with “Record” functionality so all the user needs to do is step through the website under test to create a script. |
| |
The features of Badboy include: |
| |
Capture all requests, responses, parameters and other components in a complex website. |
| |
Modify parameters in a treeview representation. |
| |
Create script variables used throughout the scripts |
| |
Add assertions |
| |
Configure browser before running test |
| |
Badboy can also do performance testing of the scripts it creates. It can also export a recorded script as a JMeter script for use in that tool. |
| Website: |
http://www.badboy.com.au/ |
| |
|
| Name: |
Samie/Pamie |
| Type: |
Web Test |
| Description: |
Samie (Simple Automation Module for IE) & Pamie (Python Automated Module for IE) are two very similar tools that allow a script to control navigation. The tool Samie uses Perl as the scripting language while Pamie uses Python. |
| |
Both of these tools require the automated scripts to be programmatically created in a text editor (though Samie does provide a script editor with additional features if a donation is made). |
| |
Scripts written with these tools require knowledge of the language used (Perl or Python) and the inclusion of the appropriate module (sam.pm for Samie and cPAMIE.py for Pamie). |
| Website: |
Samie: http://samie.sourceforge.net/ |
| |
Pamie: http://pamie.sourceforge.net/ |
| |
|
| Name: |
Selenium |
| Type: |
Java Web Test |
| Description: |
Selenium is another tool for testing websites that executes the tests directly in the browser. Selenium works with Internet Explorer, Mozilla and Firefox, allowing browser compatibility testing to be automated. |
| |
Selenium works by using Javascript and Iframes to embed a test automation engine in the browser and uses that to perform the test. |
| |
The commands used to control Selenium are contained within a three column test table with a column for “command”, “target” and “value”. It is possible to use these test tables to perform actions and assertions depending upon the needs. |
| Website: |
http://selenium.thoughtworks.com/index.html |
Other Considerations
There are other considerations that need to be taken into account when investigating free tools. The questions that need to be asked range from the technology to the available resources and personnel:
- Technology: Is the tool being considered even compatible with the specific technology used by the target test application? If the tool is to test Java and the development team is using Flash, then maybe another tool should be used.
- Needs: What are the needs of the testing team? If a way to automate the new build process is needed, then a tool to test websites is not going to be of much assistance.
- Objectives/Goals: What are the objectives & goals for the testing team? If a fully automated (lights-out) GUI regression test suite to be used and maintained over a number of years and/or product releases is the team's mandate then maybe a commercial tool is a better choice.
- Resources: What are the resources available to the testing team? This is a reference to both people and money. If money is an issue then the wise choice of a free tool will help all involved. If the individuals on the testing team are not very technical then a simpler tool with record and playback may be required.
Summary
There are a large number of free tools that can be used for automation. It will be a while before they have the same functionality and "Ease of Use" as the commercial tools of today, but with the much lower cost (i.e. free) they are still worth including in any investigation of options available. Why buy a deluxe SUV when a compact hatchback will still get you to where you are going?
Links
QA Labs is a powerful player on your team supplying the critical competitive advantage you need today. Our mission is to help you make your software products succeed in the marketplace, whatever the climate. We work with you to make wise choices that reflect project constraints, industry trends, and business considerations. We are the largest independent software quality assurance and testing service provider in Canada. For more information, please visit www.qalabs.com.
Reduce costs - Take advantage of lower labour costs and today's 
Best wishes from QA Labs to you in this