Upcoming Course Announcement: Register now for QA Labs' two-day Quality Assurance course presented through the University of British Columbia Continuing Studies on May 7th and 14th.

Visit Software Quality Assurance: More Than Just Testing or http://www.tech.ubc.ca/softeng/curric.html to register.

Establishing Effective Metrics

Introduction

The biggest challenge in establishing an effective metrics programme is not the formulas, statistics, and complex analysis that are often associated with metrics. Rather, the difficulty lies in determining which metrics provide valuable information to the project and/or organization, and which procedures are most efficient for collecting and applying these metrics.

IEEE Standard for a Software Quality Metrics Methodology IEEE Std 1061-1998 (Revision of IEEE Std 1061-1992) relates software quality to metrics in the following: "Software quality is the degree to which software possesses a desired combination of attributes. This desired combination of attributes shall be clearly defined; otherwise, assessment of quality is left to intuition. For the purpose of this standard, defining software quality for a system is equivalent to defining a list of software quality attributes required for that system. In order to measure the software quality attributes, an appropriate set of software metrics shall be identified."

It is important to be aware that a common misapplication of software metrics is to use them to measure team members productivity against an industry standard. Such comparisons do not earn support for the metrics programme and in fact are likely to cause resentment among the project staff. A metrics programme must focus on much more than productivity measures.

Software metrics are used to quantify software, software development resources, and/or the software development process. Consider these areas of software development that can benefit from inclusion in a planned metrics programme:

• Product quality
• Product performance
• Schedule and progress
• Resources and cost
• Development process

Goals of a Metrics Programme

A metrics methodology for measuring quality allows an organization to:

• Identify and increase awareness of quality requirements and goals.
• Provide a quantitative basis for evaluating and making decisions about software quality in a timely manner.
• Increase customer satisfaction by predicting and then quantifying the quality of the software before it is delivered.
• Reduce software life cycle costs by improving process effectiveness and customer satisfaction.
• Provide feedback on the metrics programme itself and validate the set of metrics being tracked.

Defining the Metrics Programme Framework

The key to the effective software metrics within an organization is to prepare a plan describing how metrics will be used to meet strategic management goals.

The first component of a metrics programme is a framework that describes the metrics to be collected, how to collect the data, and how to apply the results of analysis.

• A software quality metrics framework hierarchy begins with the establishment of quality requirements and quality goals.
• Then, by the assignment of various quality factors, the project team outlines the definitions for each of the quality requirements. [Refer to 'The Living Creature - Testing Web Applications' on the QA Labs website for some examples of quality factors.]
• Next, direct metrics for each quality requirements are obtained by decomposing each quality factor into measurable attributes. The direct metrics are concrete attributes that provide more useful definitions than quality factors to analysts, designers, programmers, testers, and maintainers.

The decomposition of quality factors into direct metrics facilitates objective communication between management and technical personnel regarding the quality objectives.

Keep the following questions in mind when considering the direct metric for each quality factor and its quality requirement or goal:

• What is this metric supposed to tell us?
• What is the theoretical relationship between the characteristic or attribute to be measured and the measurements being taken?
• Are you taking these particular measurements because they're the right ones or because they're convenient?

Beware: Often there is a lack of relationship between the metrics and what we want to measure. This makes the metric gathering process difficult and drawing valid conclusions improbable.

Example Metric and Interpretation

A sample metric that should be easy to gather from the Defect Database would be the number of existing Defects versus their Status over a series of Builds.

The corresponding abbreviated information table for this metric would be as follows:

Item	Description	Example
Name	Name to be given to this metric.	Defects Vs Status per Build (Internal Release)
Quality Factors	Quality Factors that relate to this metric.	Stability, Correctness, Completeness
Target Value	Numerical value of the metric that is to be achieved in order to meet quality requirements.  Include the critical value and range of the metric.	Zero known defects un-addressed in the Defect database system - Ideal target value for this metric would be to see the trend towards zero defects for status "New" and "ReOpened" and to a lesser extent "Resolved".
Application / Impact	Description of how the metric is used and what its area of application is.  Indication of whether this metric can be used to alter or halt the project (as "Can the metric be used to indicate deficient software quality?").	This metric is used to keep track of the number of defects in each of the available states in the Defect database.  This can be used as one reflection of the level of quality/stability of the current application.  In the future these metric values can be used to calculate the defect open/close rates.
Data Items	Input values that are necessary for computing the metric values.	Values used to calculate the metrics: - Number of defects with a status "New" - Number of defects with a status "Closed" - Number of defects with a status "Postponed" - Number of defects with a status "Rejected" - Number of defects with a status "ReOpened" - Number of defects with a status "Resolved"
Computation	Explanation of the steps involved in the metric's computation.	Collect the Data Items for the range of Builds to be considered.  Plot each Data Item as a series with Builds along the x-axis and number of defects along the y-axis.
Interpretation	Interpretation of the results of the metric's computation.	The numbers of defects un-addressed (New, ReOpened, Resolved) will give an idea of the current state of the application, and of the amount of effort that will be required to meet the Target Value(s).
Considerations	Considerations of the appropriateness of the metric (eg: can data be collected for this metric?  Is the metric appropriate for this application?).	If a defect has been addressed it does not necessarily need to have been fixed (eg: Postponed, Rejected). Equivalent Minimum Time to perform similar test coverage for testers and defect fixes for programmers on each Build must be available or the data collected will be skewed and interpretations flawed.
Tools	Software or hardware tools that are used to gather and store data, compute the metric and analyze the results.	Tools Necessary: - Export of the Defect database to an Excel spreadsheet - The Defect database
Example	An example of applying the metric.	A sample graph of this metric is shown below. [You would insert a graph that displays the number of defects that are in each status used in the Defect Database across a range of Builds.] From this graph, observations of the trends of each status can allow conclusions to be drawn about the readiness of the software for external release, and how many more Builds are required and how much development and test effort is required to reach that goal.

- adapted from IEEE Standard for a Software Quality Metrics Methodology

Conclusion

With a number of well-defined metrics measured and recorded over time, the subjectivity of future estimates and software evaluations is greatly reduced. The metrics provide a firm quantitative basis for decision-making.

As just one example, if you knew that in past projects of certain size and duration that the testing effort consisted of X hours with Y test cases, this information would provide a starting point for estimates in future projects. Of course metrics do not eliminate the need for human judgment in software evaluations; they only provide a starting point for such an estimate. - Refer to Benchmarking and Your Product - An Outline on the QA Labs website for related information.

- adapted from QA Labs Metrics Programme Process Template

About QA Labs Inc...

QA Labs is a powerful player on your team supplying the critical competitive advantage you need today. Our mission is to help you make your software products succeed in the marketplace, whatever the climate. We work with you to make wise choices that reflect project constraints, industry trends, and business considerations. We are the largest independent software quality assurance and testing service provider in Canada. For more information, please visit www.qalabs.com.