Upcoming Course Announcement:

Register now for QA Labs' two-day Software Quality Assurance - Beyond Testing course presented through the University of British Columbia Continuing Studies on September 10th and 17th.

Register now for QA Labs' two-day Introduction to Practical Test Automation course presented through the University of British Columbia Continuing Studies on September 15th and 24th.

Visit the course links above, http://www.tech.ubc.ca/softeng/curric.html, or call 604-822-1420 to register.

This month's article author: Erick Van Selst, editor: Trevor Atkins.

Mobile Devices and Embedded Systems Testing

Mobile devices and embedded systems have a number of unique challenges that are not usually seen when testing desktop and web applications. Probably the most challenging of these are the lack of any solid baselines and the short lifespan of prototype hardware.

Most desktop and server software is developed through the evolution and integration of existing tools with off-the-shelf hardware. For example, a sophisticated online market may use Apache and Tomcat running on BSD Unix hosted on x86 servers. Despite the size and complexity of each of these individual parts, their widespread adoption allows developers and testers to work on the assumption that those parts have been previously tested, are in wide use, and any defects are well known. This allows both developers and testers to focus on the unique aspects of their project.

By contrast, each cell phone can be considered to be a completely unique implementation. Applications may be reused and, when possible, a company may release a line of phones based on similar hardware or software configurations but, even in these cases, there can be few assumptions about what works and what doesn’t.

Developing and testing such a product presents a daunting task as problems can be hidden literally anywhere in the system from the UI software right down to individual chips and circuits. Problems in the hardware are the worst in that they often cannot be addressed by a software workaround. In those cases, the tester simply has to work through the problem until the next version of the hardware comes out. Even when it does, high cost and low manufacturing yields may present resource availability issues.

Working with prototype hardware presents a number of new challenges for someone with a software testing background. Foremost amongst those is the simple fact that prototype hardware is not stable. Like the software it runs, it is under development and certainly has bugs. Unlike software, hardware is relatively immutable. Due to the complex ways that hardware design problems manifest themselves, neither hardware nor software development groups are usually interested in test results from “obsolete” prototypes. Consequently, these prototypes have very short lifespans: a particular version of a prototype handset may be useful to the test group for as little as three weeks.

Protoypes are also expensive: since they are made in small runs and the manufacturing process has not been optimized, the final cost can work out to several thousand dollars each. This leads to a balancing act on the part of the manufacturer: they want to provide enough handsets to uncover the maximum number of defects over the life of the version but that has to be balanced with the cost of providing those prototype handsets.

One way that the manufacturers address this is by doing successively larger runs. The initial small runs are used to test the manufacturing process: the devices produced are only tested for electrical function. As the cost of manufacturing comes down, the yield of working devices increases and the usability increases, software for individual functions and devices can be added and tested. Eventually, as the manufacturing process stabilizes further, more devices can be built and more functionality can be tested effectively.

For the black box tester, this means that it is usually unprofitable to do more than minimal testing on early stage prototypes. Aside from any testing required to assess stability, the test team is probably better off spending their time developing test plans and exercising those plans on established devices. As the new design stabilizes and become usable by the team, there will be pressure from both within and without the team to being testing in earnest. Some software testing on these prototypes is needed but anything more than the minimum to determine product stability may not be as productive as further test development.

For a variety of reasons including those given above, most embedded projects end up abandoning all defect reports and starting over clean at least once during the project. This typically follows a new hardware or system release that fixes major, widespread symptoms and is done under the belief that it is cheaper to wipe the slate than it is to confirm every bug report. Teams engaged in early stage testing can prepare for this possibility by maintaining a regression test set for all defects identified and by focusing on test planning and development over test execution. Test automation efforts should also be focused on the bigger picture: at an early stage planning test cases and designing required functionality is going to have a bigger impact than implementation of specific cases for specific devices.

When the team is expecting to be receiving useful devices, the test leads need to ensure that they are scheduled to receive enough devices at each manufacturing stage. With a fairly stable device and a system in an early test phase, a single tester will probably require two to three test devices to be maximally productive. This can vary depending on the type of device, the degree of automation and amount of time and effort required to establish testing conditions. Since most computers and devices can restart and/or reconnect themselves when turned on, a second device may allow the tester to remain productive while a crashed device is recovering. In other cases, it can take the tester considerable effort to configure the test but once it is running, it will continue unattended allowing the tester to move onto the next device.

In addition to the above, the team must plan for device failures. A recent study of commercially released digital handsets indicated that, on average, 1 in 8 broke within a year and most of those were defective out of the box. One can expect the figure for prototypes to be much worse than that - experience suggests that one wouldn’t be out of line in ordering two devices for every one actually needed to account for manufacturing issues and the inherent fragility of prototype hardware - or four to six devices per tester.

The devices also need to be configured and installed. In most cases, this means that the tester must be familiar with the firmware flashing procedure and have the tools to execute it. In some cases the devices may need to be specifically customized or instrumented for monitoring or control. Examples include control connectors, protocol analyzers, packet injectors and other test interfaces. Note that some of these features may be more or less invasive and may only be able to be used with specific hardware versions or for specific types of testing. The testers need to be aware of what is required and available and also need to have the resources and training to use the tools provided.

When thinking of hardware, one also needs to consider all of the accessory components like cables, power supplies, batteries, chargers, headsets etc. Even the most powerful embedded system is useless if you can’t turn it on or connect it. Compared to the operating costs, these items tend to be relatively cheap and will often remain useful over the entire project. One or two extras of each of these critical items may be the cheapest insurance that you can buy.

A modern embedded device rarely exists in isolation. Whether it is a router connecting two IP networks or a cell phone with the dazzling array of data services currently available, much of the functionality of these devices relies on the presence of external devices and services. For a router, this might be as simple as a few PCs and some software to generate various sorts of traffic. For a cell phone, this can involve real or simulated networks for every wireless data protocol supported ranging from W-CDMA to IR and servers or emulators to provide every kind of data traffic to be tested. For a the latest generation of media devices one may need to provide DRM and open content as well as servers for content control, streaming media, games and messaging. All this is in addition to the basic telephone, web, SMS and MMS services. The options are staggering. To avoid being overwhelmed by the possibilities and the potential labour involved, each test organization needs to assess what specifically they need and, more importantly, what they don’t.

A test group that doesn’t need to be overly concerned about how the device may react to degenerate traffic may be able to avoid the enormous expense of setting up an entire simulated network by arranging to use existing public or private networks. Simple messaging loads can be provided by reference devices without the cost of setting up a private SMS server. At the other extreme, for entirely new services, the testers might well find themselves heavily involved is a significant infrastructure deployment. The test team needs to determine early on what kinds of services they are going to need, where they’re coming from and who is going to configure and maintain them. If not carefully planned and managed, this can lead to a chicken and egg scenario where the test teams for each component of a new service plan their tests on the assumption that all of the other components are already available.

Automation of device testing can present additional challenges. As is so often the case with software automation, a significant portion of the interface may have to be mirrored in the test harness. The difference is that many embedded devices depend on "real world" as opposed to electronic interfaces. In a cell phone this might include an electro-mechanical keypad and a display. In control systems, this can involve a wide variety of sensors such as temperature, pressure, sound, light etc. In order to effectively automate testing, one needs to be able to control those inputs with software. Depending on the testing being done, one may be able to use a pure software approach, perhaps replacing a driver with a packet injector. When this is not the case, the automator must find ways to feed the device with real inputs. Similarly, one may need to instrument the actual outputs. The core software and the test drivers may tell you that the screen is showing the time, date and temperature but that is unlikely to satisfy a customer looking at a blank display.

In summary, it cannot be overemphasized that each of the practices and suggestions above need to be observed in addition to rather than in leiu of the best software quality practices. All of the risks, costs and challenges of software design and testing are not only still present but greatly exacerbated by introducing hardware to the equation. Developing and rolling out a patch for a live server in field trial may be considered fairly expensive by some but the cost can pale in comparison to rolling a similar patch out for 5000 embedded devices. By the same token, a good review process may save many thousands of dollars by reducing the number of prototypes that need to be discarded due to minor and avoidable issues.