Life Sciences & Healthcare Insurance & Financial Telecom & Wireless Media & Games Software
Featured Customer Testimonials Case Studies
Test Automation Test Planning Testing Consulting Training
Join Our Team News & Events Management Team Partner Contact Us

Quality Improvement Roadmap Series

1. Introduction
2. Requirements
3. Modular Delivery of Components
4. Test Planning
5. Modular Test Strategies
6. Integrated Module Test Strategies
7. Reusability of Data
8. Codebase stability and builds
9. Automated Testing
10. Test Case Management
11. Quality Assurance Plans
12. Negative Feedback Development
13. Code Freezes
14. Defect Reporting and Tracking
15. Improve Communication between Development and QA
16. Independence and Authority of QA
17. QA Resource Levels and Organization
Appendix A. Sample ARM Tool Output
Appendix B. Risk Factor Matrix
Appendix C. Sample Test Plan Outline
Appendix D. Sample Test Cases
Appendix E. VSS Workflow
Appendix F. Sample Quality Assurance Plan Outline

1. Introduction

Purpose

The purpose of these pages is to outline the common areas of difficulty in performing software testing and quality assurance (QA) activities, to propose strategies for addressing those problem areas, and to outline an improvement plan for those areas.

References

[1] Software Assurance Technology Centre (SATC), National Aeronautic and Space Administration (NASA), Automated Requirements Measurement (ARM) tool, http://satc.gsfc.nasa.gov

[2] The Institute of Electrical and Electronics Engineers, Inc. (IEEE), Standard for Software Quality Assurance Plans, Std. 730.1-1989

Back to Top

2. Requirements

Most companies would benefit greatly from assigning the required Development and QA resources to completing their requirements to a much higher degree of formality, accuracy, and reusability. Improving the requirements allows for better design, implementation, test planning, and test execution. Every defect not corrected in the requirements phase is estimated to cost 15 to 100 times the time and resources ($) to correct once it reaches the testing phase. Once the requirements have been rewritten in a complete and correct manner, extending the set of requirements for future versions of the product should be fairly trivial in nature.

One good method of analyzing requirements is by use of NASA’s free Automated Requirements Measurement (ARM)[1] tool. This tool examines requirement documents for specific problem areas, such as “TBD”s, vague or ambiguous wording, as well as counting the number of requirements (the number of “shall”s) and comparing that to the number of lines of text (i.e.: is the document overly verbose?). Refer to a sample output in Appendix A.

Other requirement tools are available, most of which are used for management of requirements. A simple database (such as one created in a database application like MS Access or in a test management tool like Mercury’s Test Director) would probably be adequate for most situations. The requirements could then be stored in such a database, and exported into a Word table to create a requirements document (such as would be sent to a client). The database would allow Project Management to track any changes to and other metrics related to the requirements.

Implementation Issues

Finding resources to complete this task may be difficult.

QA should peer review the requirements as they are being written. (Product) Management, Development, and QA should be in agreement on all requirements before the project moves to the next part of the development lifecycle.

Caution should be taken to not use a requirement management tool that does not allow for easy import or export of data, as this reduces the reusability of the data.

[1] Software Assurance Technology Centre (SATC), National Aeronautic and Space Administration (NASA), Automated Requirements Measurement (ARM) tool, http://satc.gsfc.nasa.gov

Back to Top

3. Modular Delivery of Components

Each component should be testable on its own. This may require harnesses/stubs written to allow for stand-alone testing. The most critical modules should be delivered first to allow for the early identification and removal of any serious (Priority = High; Severity = Inoperable or Major) defects discovered by QA. Each and every module should be testable

First we need to identify the modules themselves. Then we need to determine the criticality of each module, based on various risk factors (number of requirements, complexity of requirements, are there new technologies involved in the module, the number of corrected defects in the module, etc.). Lastly, we need to take a look at each module in relation to the others, to take into account any relationships or interdependencies between the modules that may affect a delivery schedule. Once we rank the modules in term of effort, criticality, and interdependencies, then we can schedule their development and resource allocation accordingly.

Module Criteria:

The high-level design itself and the answers to these questions will help identify modules. Project personnel should immediately begin the process of defining the modules contained in each of the projects, then estimate the risk factors associated with each of those modules.

Proposed List of Risk Factors:

Once the modules have been ranked according to a list of risk factors (refer to the matrix in Appendix B), such as the list above, we should be able to determine which modules should be scheduled for early delivery. Also, modules with little or no UI should be designed to have a test harness (to allow for complete verification and validation of the module prior to integration) to accompany this module. Critical modules that do not lend themselves to early delivery should be mocked up (i.e.: a “stubbed interface” should be created) for use by other modules, or they may be candidates for further decomposition into sub-modules that may have different delivery timelines.

Implementation Issues

Learning how to rate the risk factors will take time. Start with what you know, then estimate as best you can. The team’s estimation skills in this area will improve over time. Statistical analysis and experience will help find the best weighting for all of the risk factors.

Assigning Development resource(s) to create the harnesses/stubs to be used by QA. One method of allocating resources for this type of task is to have the Development resources assigned to development of a module requiring a harness responsible for the creation of the harness.

Back to Top

4. Test Planning

A test plan is a document that collects and organizes testcases in a form that can be presented to project personnel, project management and external clients. A solid, well-written test plan should allow a new tester to step in and easily execute the testcases by simply following the test steps. Writing a test plan early in the project lifecycle, and having it peer reviewed by Development, in general helps reduce the workload later in the project lifecycle. This allows QA to quickly and unambiguously complete the majority of the testing required, which will provide more time for “Ad Hoc”, “Real World”, and User Scenario testing of the product.

Once the modules have been defined, QA should create a test plan for each project, with well-defined groups of testcases that correspond to the modules of the project. Using a testcase database (refer to the Test Case Management section) to export the appropriate testcases for each module on a per-project basis will make this task much simpler.

A test plan should include:

A test plan may include:

In general, the project requirements can be used as the starting point for the creation of a project’s test plan. The majority of the testcases can be created using the requirements, with the remainder created incrementally as more information is received from Development during the design phase. As the module begins to be testable, the test plan itself will be tested, and any incomplete or under-tested areas can, at that point, have more testcases created.

A test plan should NOT include descriptions of methodologies, practices, standards, defect reporting methods, corrective action methods, tools or quality requirements (refer to the section below outlining Quality Assurance Plans). Refer to the sample test plan outline in Appendix C.

Implementation Issues

Some formal training for QA in the areas of technical writing and testcase creation would be very worthwhile.

Having Development peer review the test plans and having QA peer review the design documents would also allow QA to adjust their test plans based on information generated during the design phase.

Back to Top

5. General Testing Strategy for Integrated Modules

All effort should be made to complete as much testing of basic functionality of each module as early as possible. Once the basic functionality has been established as acceptable, QA/Testing can then move on to perform stress (load, file sizes, show sizes, etc.), error handling (“bad user” testing), performance (timing) and system (Operating systems, languages/locales, hardware, etc.) testing of each module. Once completed, with no serious defects outstanding, the source code for the module should be “frozen” (see Code Freezes in November 2000).

The basic strategy when testing individual modules is to begin by verifying that the module itself is testable. A small sub-set of the testcases for the module should be performed with each new build, to quickly and efficiently determine that in-depth testing of the module is possible. This testcase sub-set is commonly referred to as a Build Verification Test (BVT). One the module is determined to be testable, QA/Testing then proceeds to test as much of the core functionality as possible. As much of the input and output interfaces are tested (either via the UI or via the use of a test harness or stub), and user interface is fully exercised. All defects discovered are reported, and as many corrected as possible within a reasonable timeframe (based on the project schedule).

Once the functionality has been fully tested and all serious defects corrected, QA/Testing can then go onto perform stress testing and negative (error handling, or “bad user”) testing. This allows QA/Testing to identify serious problem areas early on in the project lifecycle, when they are simpler to detect, isolate and correct.

After stress and negative testing have been completed and all serious defects corrected, QA/Testing will move on to system and performance testing. Problems with hardware, operating systems, cards, drivers, and other peripherals should be identified and corrected as early in the project as possible, and modular system testing will allow QA/Testing to do just that. Preliminary performance testing should be broad and shallow, that is, it should focus on the performance of the basic functions. If problems are detected in the performance of a particular module, then more comprehensive performance testing should be performed.

Once the module shows acceptable performance, load and stress testcase results with no serious defects remaining, a regression test pass should be performed. In this testing phase, all defects identified in the module that have been closed (whether closed as fixed, not reproducible or won’t fix) should be retested to ensure that all closed issues are still closed, and that the corrective actions (if any) have not caused collateral defects elsewhere in the module.

Implementation Issues

QA/Testing needs incremental and scheduled delivery of each module from Development. QA/Testing would also benefit from having complete (robust) testcases that are accessible, usable, easily added to, and that can display the results of testing.

Back to Top

6. General Testing Strategy for Integrated Modules

QA needs to be able to integrate the “accepted” modules incrementally. If the product follows a somewhat linear workflow, this should be easy to implement for the next project lifecycle. The kind of testing that will yield the best results at this point is to follow something akin to User Scenarios (Use Cases), where the work flow is tested, as opposed to the unit functionality of each distinct module. Of course, one caveat here is if designed/planned module interfaces have undergone any kind of changes prior to integration. This could be problematic if this kind of information is not communicated to all users and implementers of the interfaces, and the interfaces are not kept consistent between modules (i.e.: module APIs should be implemented as per their design; any deviations from expected APIs could cause serious problems).

The basic strategy when testing integrated modules is to begin by testing as much of the complete workflow as possible. The testcases for this type of testing will take the form of User Scenarios/Use Cases, where the entire system is treated as one large black box. Linking together several testcases can create primitive User Scenarios as a starting point, and over time, more sophisticated User Scenarios can be developed. All defects discovered are reported, and as many corrected as possible within a reasonable timeframe (based on the project schedule).

Once the basic workflow has been fully tested and all serious defects corrected, QA can begin system testing of the integrated product, completing testing on all hardware and software configurations, as well as collecting performance data. Corrections to serious defects in this phase of testing are usually of moderate to high risk, therefore risk analysis should be performed by Development and QA prior to the implementation of any correction. This sort of risk analysis must include all stakeholders, including Support, Product Management, and other domain experts to allow for an accurate assessment of the impact of correcting or of not correcting the defect.

Next, QA can then go onto perform stress testing and negative (error handling, or “bad user”) testing. This will involve creating extremely large and complex files (“Real World” shows), large numbers of files, passing them through a complex scheduling routine, and transmitting the information correctly to a large number of players. This type of testing should be followed up by negative or “bad user” testing, where the tester seeks to abuse the functionality of the product in an effort to create odd program states. The majority of serious defects discovered in this last phase will require risk analysis, as they will likely have a large impact on the entire product. Please note that system testing is performed before stress and performance testing when testing integrated modules. This is a particularly useful approach for testing distributed product(s), where different modules are often located on physically separate machines.

Lastly, a complete regression test pass should be performed. In this testing phase, all defects identified in the integrated product should be retested to ensure that all closed issues are still closed, and that the corrective actions (if any) have not caused collateral defects elsewhere in the product.

Implementation Issues

One issue is training QA staff on how to design and document effective User Scenarios. Another issue is scheduling time for the creation of the User Scenarios. Since the majority of test planning is completed early in the project lifecycle but the testing does not occur until much later in the project lifecycle, there is a risk that the User Scenarios will not be developed until during the test phase of the project. This is precisely when QA resources have little or no additional bandwidth to take on this task.

Back to Top

7. Reusability of Data

Increasing the amount of data QA can reuse during the test/fix/retest cycle is a method to reduce the time required during the planning and the testing stages. One area that lends itself to increased reusability is project documentation. Requirements should be stored in such a manner that the information can be easily accessed, simple to update (where the data does not change in general, but the data, once approved for change, is modifiable), and can be reused by other departments. For example, QA should be able to grab a table full of requirements from a requirements document, export the table to a tab delimited file, import the data into Excel or Access (or any Database), and use that data to help generate and track functional testcases. To go one step further along this route, using two relational databases, one, which contains the requirements, and another, which contains the related testcases, QA and Development increase the reusability of the information contained inside the databases.

The requirements, if written in a certain manner or following a certain structure, can be reused by QA to aid in testcase generation. For example, the following requirement, written as such, could be reused quite easily to generate testcases:

Requirement ID Version Date-Phone Format

XXX-1234

1.1a

For all phone number entry fields the user will have a pre-formatted text box to help the user in correctly entering phone number information.

This table could be partially reused in a Test Plan, as follows (copy, paste, add one column, modify the headers, add the testcase steps, clarify the expected results):

Requirement ID Version Date-Phone Format Expected Result(s)

XXX-1234

1.1a

[Enter test case steps here]

  1. Launch X
  2. Bring up the Phone entry panel by clicking on the “phone” icon
  3. Enter a valid telephone number, i.e.: “1234567”, in the phone number field
  4. Click “Ok” to accept the phone number.

[Edit the following to yield the expected results for the associated test case]

The text field should reformat the data to read “123 4567”.

This is far simpler and will save a considerable amount of time as compared to creating a 100-page test plan from scratch. The test plan will also have the information structured in a manner similar to the requirements, which generally increases its readability. Another way to achieve significant savings in time spent generating testcases is to use Excel. Simply export the Word requirements table to a CSV (comma separated) or TAB (tab delimited) text file, then open the file in Excel. Use the fill and sort functions in Excel, create the remainder of the testcases and their expected values, save the Excel file as a CSV or TAB file, then, once opened in Word, convert the text to table (Table/Convert Table to Text). Select the same delimiter as used when saving the file in Excel, and the testcase table should be created correctly.

During modular testing, much of the output of many of the modules can be reused for future re-testing of the module, or as input for the following module (the next module in the product workflow). Retention of these testing assets will, over time, result in the creation of a large test suite that can be reused for future versions of the product. During final performance testing, all data should be retained for use as a “baseline” for future versions of the product. This can be done in an Excel spreadsheet, which allows for easy graphical representation of the data.

After the release of the product, the closed defects should be analyzed for reusability as testcases for the next version of the product. The defects that are not directly associated to a previously existing testcase can be seen as a new testcase, one that wasn’t included in the original planning, and thus it should be added as a testcase for future versions.

Implementation Issues

Companies need to avoid tools that do not support import and/or export of data, as this reduces the reusability of data and can force them to be “locked” into one particular tool or data format. Changing to another data format requires changing tools, which can be costly. Custom databases that are scalable, modifiable and provide import and export capabilities are more than sufficient.

Back to Top

8. Codebase Stability (a.k.a. “Daily/Weekly Builds”)

QA and/or Development should be producing incremental, scheduled builds as soon as possible in the implementation phase of the project lifecycle. This allows QA and Development to track when and where a defect was introduced into the source code.

Builds should be done on a daily or a scheduled basis, while still allowing for “on-demand” builds. Moving to a daily build situation allows QA to test very incremental changes, particularly defect corrections, very quickly as there will not be many other changes made to the codebase between successive build (i.e.: one day). Doing daily builds also allows Development to quickly isolate the introduction point of defects (determine which build introduced the defect, then analyze the code changes that contributed to the defect itself).

On-demand builds should be done only if there is a clear objective, which must be communicated to all parties. The easiest manner of doing this would be the creation of a “Build Request” form. This form would be filled out well before beginning each build, and would contain:

These requests, after their initial creation, would then be “owned” by the creator of the request. The request should then be e-mailed to all affected parties (project personnel on all three projects) which allows them a chance to add any information specific to their project. The build cannot be done unless all parties “sign off” on the request. The completed Build Request forms should be retained in VSS for future reference and for use in the project Post Mortem. This form could be something as simple as a Word template, which could then evolve to a more sophisticated build tracking system, perhaps via the use of a form on an Intranet that puts the entered data into an Access database. As the process becomes more robust, the form could be used as a means of collecting metrics such as lines of code changed in each build, number of defects addressed in each build, etc. Refer to Appendix E for an explanation of the building from VSS workflow.

Automation of the build procedure is one way to reduce the amount of resources spent on performing regular and on-demand builds. The Source Safe archive can be easily accessed via the command line, which offers a simple and effective option: scripting. Using scripts, a company could:

This could then be run every morning at a very early time, when it is likely that no one will have files that are critical to the project checked out. Alternatively, a staggered build schedule devised on a per module basis, one which is responsive to changes in the overall project and individual module statuses, would help alleviate some of the difficulties encountered by QA.

Implementation Issues

There is often no requirement that the source code must compile and must be testable (as testable as possible). Consequently, QA may be “forced” to continue testing an older build of the product rather than the latest, merely because the newer build is not functionally testable (i.e.: “bad build”). The amount of time required for determining if a build is testable or not can be quite lengthy. This may be reduced through scheduled delivery of builds combined with automation of Build Verification Testcases (BVTs).

Automating the build requires some domain knowledge of scripting, network transactions, SS command line interfaces, VB command line interfaces, and archival and compression techniques. This knowledge may exist in-house, but there may not adequate resources to perform this work in parallel with the current development cycle. This may be a good candidate for external (contract) work. All issues with third-party products will need resolution before the implementation of any automation can begin.

Designing a useable Build Request form.

Back to Top

9. Automated Testing

Product companies that have scheduled incremental releases can benefit enormously from adding Automation to their testing arsenal. The ability to reduce the amount of manual testing and re-testing is appealing, yet must be approached with some caution – it is not the “silver bullet” (magic solution) that it appears to be at first glance.

The fatal flaws in implementation of automation are trying to automate too much at once, and trying to automate indiscriminately. Although automation can be a time and effort saving mechanism, if it is poorly or partially implemented it’s usability (and thus it’s time saving properties) decreases. Automating modules of the project that are subject to frequent re-designs or re-implementations will require constant updating and re-automating.

Ideally, we want to start with a small set of automated testcases, and with each release, add to this set. Identifying a core set of testcases that should pass with every build (Build Verification Testcases, or BVTs) is the logical first step. At that point, we should automate those BVTs. The automated BVTs could be performed as part of the automated build process, while still being available for use for “on-demand” builds.

After automation of BVTs, the automation of the UI testcases is the next step. Since the User Interface of an application is generally one of the first areas to be “frozen” or less subject to design changes after initial implementation, the automation of the UI testing is one way to reduce manual testing and increase efficiency. UI testing involves using the mouse and the keyboard to navigate to all panes and windows, testing all keyboard mnemonics and accelerators, changing window properties (size, position, minimize, maximize, close). Once much of the UI testing is automated, any further automation should target those areas, components and modules that have a high degree of commonality between them.

Many third party tools exist to assist and manage automated testing. A good “capture-playback” tool that provides scripting capabilities and a command-line interface will more than likely suffice for the immediate to medium term needs, provided it can gracefully recover and continue testing after a serious failure. Other tools offer more functionality, including integration with IDEs and version control software, but in general, these tools are quite expensive, so some analysis of the requirements of a testing tool should be done prior to purchasing. Some care must be taken to ensure that, if an off-the-shelf tool is chosen for future use, the project personnel, particularly QA personnel have adequate training in the use of the tool.

Implementation Issues

Third party tools can be very expensive, and often offer limited functionality, particularly if the users are not sufficiently trained in the use of the tool.

Creating automation scripts that are maintainable and scalable is critical. QA must be able to easily modify the automation scripts to allow for variations from release to release, and ensure that all automation scripts are well documented for future users.

Back to Top

10. Test Case Management

Management of testcases will allow QA to consistently test the product in a reproducible manner, while generating useful metrics that can be used for tracking project test coverage and results or for predicting emerging trends. A means to reduce duplicate testcases (and thus reduce duplicate effort!) is by the use of a testcase database.

A simple database in MS Access should be created for use in the short and medium term. Each record should contain the following information:

Once the database has been created, QA should begin populating it with functional testcases, then focus on identifying BVT testcases from that set. The database and the record form must be extremely accessible and usable, otherwise it will not be used consistently. One key piece of functionality that will enhance usability of the testcase database is to ensure that users can easily import data from other sources to populate the database very quickly and efficiently.

Once populated to an acceptable degree, QA can record the results of their test passes in the database, including dates testcase performed, on which build the testcase was performed, the result (pass or fail), and IDs of any defects recorded against that testcase. This will allow queries to be run against the database records to determine such metrics as: % testcases pass; % testcases fail; % testcases Hi, Medium, Low priority; % testcases that overlap between multiple projects; and so forth. QA will also be able to generate reports that show, in varying degrees of detail, the amount of testing completed and the results of the testing completed, including associated defect IDs. This information can be valuable internally, as a means to collect quantitative data that can be analyzed and used to drive process improvement, as well as externally, as a means of demonstrating to a client precisely what tests were performed to determine the quality of the product (this can be extremely important for mission-critical/life-critical products).

Implementation Issues

Creating a usable database for the testcases will require some time and resources. Starting with the basic information, then increasing the level of detail and the scope of the information contained in the records incrementally will make the transition easier for QA.

Back to Top

11. Quality Assurance Plans

A Quality Assurance Plan (sometimes referred to as simply a Quality Plan or a QA Plan) lists in detail what the QA team’s responsibilities will be for the project. This document also includes what tasks the QA team will not perform and why. The main reason behind writing a QA Plan is it clearly outlines the boundaries of QA’s responsibilities on the project to the rest of the project personnel, including any external clients, sub-contractors, and co-contractors.

QA should write a QA Plan for each project while the requirements are being written. Here is a listing of the first-level contents of the IEEE Quality Assurance Plan:

  1. Purpose
  2. References
  3. Management
  4. Documentation
  5. Standards, Practices, Conventions and Metrics
  6. Reviews and Audits
  7. Test
  8. Problem Reporting and Corrective Action
  9. Tools, Techniques and Methodologies
  10. Code Control
  11. Media Control
  12. Supplier Control
  13. Records Collection, Maintenance and Retention
  14. Training
  15. Risk Management

A QA Plan that covers the key items of the list above (modified where appropriate) should be written for each project. A sample outline, with a brief description of each item, is included in Appendix F. A copy of the QA Plan should be circulated to all stakeholders, including Development (Project Lead), Product Management, External Clients, Support, and any Sub- or Co-Contractors. All parties should sign-off on the QA Plan once any issues have been resolved. QA should control the document, that is, changes to the QA Plan should not be commonplace, and not without agreement from all stakeholders.

Implementation Issues

Lack of training and technical expertise will make this a difficult document to create. However, having an incomplete plan is probably of greater benefit than having no plan, particularly when dealing with external parties, co-, or sub-contractors.

Back to Top

12. Incremental Testing combined with Slowed Feature Addition (a.k.a. “Negative Feedback” Development)

Logic dictates that fixing a problem in a simple system should be easier than fixing the same problem in a complex system. When serious defects are discovered in early testing, these defects should be addressed prior to the completion and integration of new code/functionality. This requires that Development be able to stop incremental development on new code to stop and return to correct any serious defects found in older code, while QA must be aware of incomplete functionality and avoid testing those areas of a module currently under initial development.

In the next project lifecycle, the modules that have been identified as both complex and critical should adopt a negative feedback development approach. After the implementation of any new functionality, that functionality should be tested and all serious (Priority = High; Severity = Inoperable or Major) defects relating to that functionality corrected prior to the implementation of any additional functionality to that module, and in particular, any additions to the defective functionality. Metrics can then be collected at the end of the project lifecycle and the rates and distributions of defects in the modules that used the negative feedback development approach can be compared against those that did not (i.e.: what is the distribution of serious defects? do more of them occur in the modules that used a negative feedback development approach or in those that did not?).

Implementation Issues

As the codebase increases in size from version to version, this becomes a more difficult task.

Back to Top

13. Code Freezes

As a module becomes feature complete, and once a module has been validated and verified against the requirements, the source code should not change, except in the case where a serious defect is discovered. In that case, the defect may be corrected once analyses of possible solutions and their associated risks have been completed and all parties (Product Management, Development and QA) are in agreement that the defect should be corrected.

In conjunction with modular delivery and testing, once a module has completed modular testing with no serious defects remaining unresolved, the module should enter a quasi code freeze, or a “code slush”. At this point, no new functionality should be added to the module codebase, so that integrated testing can begin. The module codebase at this point is determined to be “functionally complete”. Integrated testing will uncover more defects in the “slushy” module, and once they have been resolved in some manner, the module codebase should be frozen. At this point, no further changes to the module codebase should occur, with the exception of correction of serious defects. As each module becomes frozen, QA can perform full regression test passes on the frozen module. Once no more corrections are required, the module codebase is then determined to be “code complete”. Each frozen module will no longer require modular (functional) testing, and only those “slushy” modules that still have outstanding serious defects will require modular testing. At the point when all modules are frozen, QA can begin a regression test pass on the frozen (integrated) product. Once no further corrections are required for the integrated product, the product is then determined to be “code complete” and the product is ready for release.

Implementation Issues

QA should have knowledge of the Source Safe Administrator password and have the authority to remove check-in/out/add/delete permissions on modules that are ready to be frozen. If this is not feasible/palatable, then QA should have the authority to direct the SS Administrator to remove the related permissions.

Implementation of a Build Request system would allow QA to preview planned changes to modules that may be in a frozen state, and allow time for a response to the Build Request.

Back to Top

14. Improved Defect Reporting and Tracking

Defect reports have many uses: to record defects and their solutions, to provide quantitative data for post-project analysis and project trend predictions, to increase the effectiveness of regression testing, and to simplify the investigation of defects for Development. QA must begin to add more detail to the defect reports to allow for their use and reuse in the manners described above.

The “Description” field should always include a complete list of “repro steps” which, if followed, will allow any person to reproduce the defect. These repro steps are nearly identical to the test case steps listed for each test case and are critical when performing a regression test pass; without them, QA cannot verify that they are testing the same functionality that contained the corrected defect. A template that is automatically generated for the “Description” field of the defect form would help QA to fill out the repro steps for each defect.

QA should present the defect statistics at project status meeting, or perhaps publish them on the project Intranet. The data presented should include: the number of open defects; a breakdown of open defects by priority; the number found in the last week; the number resolved in the last week; and the “Bug of the Week” (the most serious, strange or interesting defect discovered during the last week). This would give the entire project team a good overview of project stability, QA efforts, and outstanding defect issues. These statistics should be saved to allow for statistical analysis as part of the Post Mortem phase. A defect “glide path” should also be created by QA for distribution to project personnel as the project enters the “code slush” phase. This glide path would show an ideal defect correction rate to achieve a state where all serious defects are corrected and thus the project is deemed acceptable for external release.

Implementation Issues

Adding the template to the “Description” field will have to be done by IT (most likely). QA should know how to make changes to the defect forms.

Back to Top

15. Improved Communication between Development and QA

The greater the flow of information between the various team members, the easier it is to resolve issues that arise during the course of a project. At many companies, QA, although they have access to project documentation and attend project status meetings, are usually not receiving enough information to properly perform their assigned tasks. And much of the information they do receive is either beyond their technical skill level or so far below it as to be nearly useless.

QA should use these status meetings to update the rest of the project team about such issues as current defect counts and their distribution by severity or priority, current risks that QA feels may crop up or need addressing, or even resource issues (hardware, software, people, etc.).

A way to collect project-specific information in a highly usable manner is via a project Intranet. Information such as how to set-up test environments, how to build the project, latest revisions of project requirements documents, project design and architecture documents, project quality assurance plans, project test plans, and so forth, can all be linked together through a single Intranet. VSS allows for “shadowing” of files from the archive to an administrator-specified location. The shadowed files could easily be linked via a central Home page. On the central Home page, other important project information could be listed, such as the availability of updated documents, module status, build requests, et cetera. Over time, the Intranet could grow to include user documentation in HTML format, particularly since on-line (HTML format) help files and web-downloadable applications are becoming the norm. QA could then create as many pages as they want to convey project-specific and QA-specific information to the rest of the project team. Limited access could also be given to external clients to connect and review documents on-line.

Implementation Issues

Having the hardware, software and resources to create and maintain a project intranet may be difficult to find.

Back to Top

16. Independence and Authority of QA

QA must be independent and have the necessary authority to perform the tasks associated with their role(s) properly. QA must be independent of Development to allow for impartial validation and verification of the code produced by Development. QA must have authority to raise issues at meetings, to reopen uncorrected defects, to enforce adherence to the QA Plan by all parties, and to escalate issues that QA feels are not being addressed appropriately.

The granting of authority to QA must come from management, and it must be in place for all aspects of the project lifecycle. The QA Plan should be the reference document to which QA can refer Development, Project Management, and external clients for the enforceable guidelines for the project. QA has the authority to inform members of the project team when they are not complying with these guidelines. QA also has the authority to inform project management when non-compliance continues. Examples of common guidelines: buildable and testable source code at all times; impact analysis prior to correction of a high-risk defect, particularly late in the project lifecycle; monitoring for adherence to requirements and design documents (watching for “feature creep”); prioritizing (with other stakeholders) defects for correction, particularly late in the project lifecycle (“bug triage”).

QA should write QA Plans for each project as soon as it is appropriate. QA should become the VSS Administrator and the defect tracking database/software Administrator. QA should be in charge of scheduled and on-demand builds.

QA must have the authority to make the “ship/no ship” decision. Features can always be removed or scaled back in order to meet ship deadlines.

Implementation Issues

This issue depends on the technical and professional competency of the members of the QA department. Training and management support (buy-in) is mandatory.

Back to Top

17. QA Resource Levels and Organization

The ratio of QA to Development varies from company to company, but in general, the target is to have a low ratio. Microsoft adopts a 1:1 ratio of QA to developers; Adobe is more along the lines of 1:2 or 1:3. Once a project reaches a ratio of 1:6 or 7, the amount of “catch-up” testing (testing required to keep up with daily changes to the codebase) begins to dominate the QA persons' time. The QA resources then have no extra time to work on process improvement, additional test planning and emergency situations that can arise, particularly near the end of the project lifecycle, let alone perform adequate testing of the product!

The QA team, as it grows, should be structured to allow for flexibility in times of high-testing demands. One attractive solution is to have one QA/Test Lead for each project, a person who is responsible for:

This structure allows each QA Lead to develop domain expertise in the technical area(s) that their projects utilize, as well as acting as extra test help on other projects when the need arises (i.e.: prior to release). A strategy such as this also allows staff to feel ownership of a product, which can be a great motivational tool, and which ensures the QA Lead for the project is always focussed on the “best interests” of the project.

In the long term, as the QA team increases in size (beyond four [4] QA personnel), one person should be nominated to act as QA/Test Manager. This person will be responsible for:

This structure provides enough resources to begin working on processes and procedures to incorporate continuous improvements in Development and QA. It also allows for the addition of new personnel quite easily.

Implementation Issues

Finding experienced QA personnel is difficult; training an inexperienced person is an acceptable, although more expensive solution in the short term (long term value is greater). As the team grows, senior team members will be able to take on the majority of the training of new personnel.

Management must buy-in to long-term quality improvement, and must commit the required finances to ensure a successful transition from an ad-hoc QA and Development cycle to a repeatable, continuously improving cycle.

Back to Top

Appendix A – Sample Output of the ARM Tool

Below is a sample output of ARM as applied to a sample requirement document:

The counts of IMPERATIVES found in file ReqSpec.txt are shown in the table below.
IMPERATIVE OCCURRENCE
shall 158
must 9
is required to 2
are applicable 0
are to 0
responsible for 3
will 33
should 2
TOTAL 207
The counts of CONTINUANCES found in file ReqSpec.txt are shown in the table below.
CONTINUANCE OCCURRENCE
below 0
as follows 0
following 0
listed 0
in particular 0
support 0
and 42
: 8
TOTAL 50
The counts of DIRECTIVES found in file ReqSpec.txt are shown in the table below.
DIRECTIVE OCCURRENCE
e.g. 0
i.e. 0
For example 1
Figure 1
Table 0
Note: 0
TOTAL 2
The counts of OPTIONS found in file ReqSpec.txt are shown in the table below.
OPTION PHRASES OCCURRENCE
can 21
may 14
Optionally 0
TOTAL 35
The counts of WEAK PHRASES found in file ReqSpec.txt are shown in the table below.
WEAK PHRASE OCCURRENCE
adequate 0
as appropriate 0
be able to 19
be capable of 7
capability of 0
capability to 0
effective 0
as required
normal 1
provide for 1
timely 0
easy to 0
TOTAL 28
The counts of INCOMPLETE words and phrases found in ReqSpec.txt are shown in the table below.
INCOMPLETE OCCURRENCE
TBD 0
TBS 0
TBE 0
TBC 0
TBR 0
not defined 0
not determined 0
but not limited to 0
as a minimum 0
TOTAL 0
The NUMBERING and SPECIFICATION STRUCTURAL counts for ReqSpec.txt are provided by the following tables:
Numbering Structure Specification Structure
DEPTH OCCURRENCE DEPTH OCCURRENCE
1 21 1 8
2 54 2 165
3 10 3 55
4 0 4 19
5 0 5 0
6 0 6 0
7 0 7 0
8 0 8 0
9 0 9 0
TOTAL 85 TOTAL 247
Total text strings: 724 Unique Subjects: 138
Total for SPECIFICATION STRUCTURE: 247
Total text strings: 724
RATIO = 0.341160220994475
Unique Subjects: 138
Total for SPECIFICATION STRUCTURE: 247
RATIO = 0.5587044

Back to Top

Appendix B – Risk Factor Matrix

We can easily create a matrix of modules compared to risk factors. Once we’ve filled in the various values, we should be able to add the numbers up. However, this would give a misleading result: we need to put more weight on certain risk factors than others. For example, the number of requirements should have the lowest weight of all the factors listed here, while complexity and testability should be most heavily weighted.

RISK FACTORS / MODULES Req’s Comp New Techs Work Type Test Fixed defects Imp Total
Module 1                
Module 2                
Module 3                
Module 4                
Req’s: Number of requirements; this can be computed by using the ARM tool (see section 2)
Comp: Complexity, where 1 is associated with modules of least complexity and 10 is associated with modules of highest complexity.
Techs: New technologies; for each new technology used in a module, score 1.
Work Type: The type of work being done where 1 is associated with work that is extremely familiar (i.e.: identical projects) and 10 is associated with work that is not familiar at all.
Test: The testability of the module where non-testable modules score 4 and extremely testable modules score 1.
Fixed defects: The number of defects (in each module) that have been fixed since the last risk calculation.
Imp: The importance of the module relative to the other modules, where a score of 1 is associated with relatively unimportant (trivial) modules and a score of 10 is associated with key or critical modules.

One suggested weighting would be:

Total = (A ´ # of Requirements) a + (B ´ Complexity) b + (C ´ New Technologies)c + (D ´ Type of Work)d + (E ´ Testability) e + (F ´ Importance) f + (G ´ # of Fixed defects) g

where {A - G, a - g} are real numbers that should be determined by way of statistical analysis of historical data.

Back to Top

Appendix C – Test Plan Outline

Back to Top

Appendix D – Sample Testcases

A sample of a table of testcases with listed explicit expected results is given below:
Testcase ID Testcase Steps Req ID Expected Result(s)
  Install XXXX Software    
[TST-0001]

Install XXXX Software – Destination Directory

  1. Begin installation of the XXXX software
  2. At the “Select Destination Directory” pane, click “Ok” to accept the default drive location
 ABC-0001 Clicking “Ok” should allow the user to proceed to the “Backup” pane.
[TST-0002]

Install XXXX Software – Destination Directory

  1. Begin installation of the XXXX software
  2. At the “Select Destination Directory” pane, enter a non-existent drive for the folder location (i.e.: “z”)
  3. Click “Ok” to accept the drive location
 ABC-0002 An error dialog is presented to the user: “Invalid directory name specified” and the drive letter is not accepted.
[TST-0003]

Install XXXX Software – Destination Directory

  1. Begin installation of the XXXX software
  2. At the “Select Destination Directory” pane, enter a drive for the folder location that contains illegal characters (i.e.: one of < > ? / \ | ;)
  3. Click “Ok” to accept the drive location
 ABC-0003 An error dialog is presented to the user: "Invalid directory name specified" and the directory name is not accepted.

Back to Top

Appendix E – Visual Source Safe Workflow

Builds are often done by locking out all developers from the VSS archive, performing a recursive “get” on the project, and, once the build is completed with no errors, a label is placed on the project in VSS. If the label was placed on the project first, then a recursive “get” done from that label, the project could be built without preventing the developers from “checking in” code for any period of time. The only possible way a check-in of a file could cause problems for the get from the label is if the developer’s system clock is incorrectly set. VSS has one strange quirk: the versioning for individual files is done numerically, but the versioning for the labels is done by means of time-date stamping. Thus a developer could check in a file after a label has been made, but if the developer’s system clock is 3 hours behind the time of the label, the file will be checked-in under the label. Thus any “get” from that label will unknowingly and erroneously get the latest version of the file, even if it was not intended to be included in that label’s build. The quick solution to this possible problem is to designate one internal server as the clock for the project, and for all project personnel to add a script to their start up folder which synchs their machine time to the server’s time.

VSS Workflow:

Of course, if the build itself is broken, the label from which the build was made should have a comment added which indicates that the build was broken in file X with error(s) Y. A new build could be performed once the defective file is corrected simply by placing a new, distinct label on the project, performing a recursive “get” on the label and re-building the project. During the Post Mortem of the project, a quick look at how many builds were broken as a percentage of total builds will be a clear indication of overall codebase stability, while any trends towards improvement or degradation of stability can be detected and addressed. Enforcement of a codebase that is always buildable AND testable at all times will help reduce the number of broken builds. Following this workflow will always allow QA to rebuild any previous build at any time with no fears of missing one updated file or including one broken file. This can be crucial when creating builds targets for external release(s).

Back to Top

Appendix F – Quality Assurance Plan Outline

This is a modified version of the IEEE QA Plan outline. Included are brief descriptions of each of the heading and sub-heading contents. It is still quite comprehensive, however, and could be pared down even further if needed.
QA Plan Headings Brief Description of Contents
1. Introduction
1.1 Purpose The purpose of the document
1.2 Quality Goals Any goals pertaining to overall quality of the product should be listed here.
1.3 Quality Requirements A listing of any requirements pertaining to QA, testing and overall quality.
1.4 Software Requirements A listing of any software requirements pertaining to QA, testing and overall quality.
1.5 Hardware Requirements A listing of any hardware requirements pertaining to QA, testing and overall quality.
1.6 Product Requirements A listing of any product requirements pertaining to QA, testing and overall quality.
1.7 Assumptions A listing of all assumptions made in the creation of this QA Plan.
1.8 References A list of all documents referred to in this QA Plan, either directly or indirectly.
2. Management How is the management structure for the project is organized?
2.1 Interfaces Who interfaces with whom for what?
2.2 Responsibilities What is QA responsible for on this project?
2.3 Project Org Chart (optional) Shows relationships/interfaces; usually only needed for large project teams
3. Documentation  
3.1 Quality Documentation What quality documentation is QA responsible for creating/reviewing/approving?
3.2 Other Project Documentation What additional project documentation is QA responsible for creating/reviewing/approving?
3.3 User Documentation What user documentation is QA responsible for creating/reviewing/approving?
4. Standards, Practices, Conventions and Metrics  
4.1 Standards What standards, if any, will be used on the project (all areas of the project, not just QA)?
4.2 Practices What are the general practices, if any, that will be used on the project (all areas of the project, not just QA)?
4.3 Conventions What are the general conventions (naming of builds, for example), if any, that will be used on the project (all areas of the project, not just QA)?
4.4 Metrics Will metrics be collected by QA on this project? If so, what kind of metrics will be collected? Note: no need to say what will be done with the data, just state if they’re being collected or not.
5. Reviews and Audits  
5.1 Reviews Will reviews be done? Formal? Informal (peer)?
5.2 Audits Will audits be done? If so, on what parts of the project (i.e.: project plan audits, milestone audits, etc.)?
6. Test Outline of the basic plan/strategy for testing as a whole for the project.
6.1 Unit Testing Will QA perform Unit Testing? If so, describe the basic strategy for Unit Testing. If not, who is responsible for Unit Testing?
6.2 Module Testing Will QA perform Module Testing? If so, describe the basic strategy for Module Testing.
6.3 Integration Testing Will QA perform Integration Testing? If so, describe the basic strategy for Integration Testing.
6.4 Build Verification Testing (BVT) Will QA perform Build Verification Testing? If so, describe the basic strategy for Build Verification Testing.
6.5 Functional Testing Will QA perform Functional Testing? If so, describe the basic strategy for Functional Testing.
6.6 User Interface Testing Will QA perform UI Testing? If so, describe the basic strategy for UI Testing.
6.7 Error Handling, or "Negative" Testing Will QA perform Negative Testing? If so, describe the basic strategy for Negative Testing.
6.8 System Testing Will QA perform System Testing? If so, describe the basic strategy for System Testing.
6.9 Stress Testing Will QA perform Stress (load) Testing? If so, describe the basic strategy for Stress (load) Testing. What are the acceptance criteria for Stress (load) Testing?
6.10 Performance Testing Will QA perform Performance Testing? If so, describe the basic strategy for Performance Testing. What are the acceptance criteria for Performance Testing?
6.11 Reliability Testing Will QA perform Reliability Testing? If so, describe the basic strategy for Reliability Testing. What are the acceptance criteria for Reliability Testing?
6.12 Security Testing Will QA perform Security Testing? If so, describe the basic strategy for Security Testing. What are the acceptance criteria for Security Testing?
6.13 Installation Testing Will QA perform Installation Testing? If so, describe the basic strategy for Installation Testing. What are the acceptance criteria for Installation Testing?
6.14 Internationalization and Localization Testing Will QA perform Int’l and Loc’l Testing? If so, describe the basic strategy for Int’l and Loc’l Testing. What are the acceptance criteria for Int’l and Loc’l Testing?
6.15 Regression Testing Will QA perform Regression Testing? If so, describe the basic strategy for Regression Testing.
7. Problem Reporting and Corrective Action  
7.1 Problem Reporting How problems are reported
7.2 Corrective Action How problems are fixed
7.3 Escalation Procedures How problems that QA feels are not being addressed appropriately get addresses and escalated, if necessary.
8. Tools, Techniques and Methodologies  
8.1 Tools A listing of all tools used on the project (all tools!), and a brief description of how/where/when they will be used.
8.2 Techniques Any standardized/published techniques that will be used on the project (i.e.: statistical techniques that will be used on collected metrics).
8.3 Methodologies Any standardized/published or custom methodologies that will be used on the project (i.e.: spiral lifecycle model).
9. Code Control  
9.1 Code Control Guidelines What are the measures in place to ensure control of the codebase?
9.2 Archiving Source Code How is the source code archived incrementally? What is done with the archives?
9.3 Archiving Compiled Code (Binaries) How is compiled code (binaries) archived incrementally? What is done with the archives?
10. Media Control  
10.1 Internal Media How is media created internally controlled, if at all?
10.2 External Media How is media received from external sources controlled, if at all?
11. Supplier Control What third-party tools/products/libraries will be used on the project? How will they be controlled? Are they tested prior to use?
12. Records Collection, Maintenance and Retention How will records (defects, Test pass results, etc) be collected, maintained and retained? How long will they be retained?
13. Risk Management What kinds of Risk Management techniques will be used, if any?
14. Glossary A listing of all specific terminology with their definitions

Back to Top

Our Clients
“…have the experience and the knowledge of tools and processes that you need…”
Case Studies
…reduced a test case set that numbered in the hundreds of trillions down to a single work day…

QA Labs Focus

QA Labs Focus Diagram Consulting Training Testing Test Automation Test Planning Our Expertise