Task- Based software testing

Once in a Software Testing conference held in Banglore, India, the topic of discussion was "How to test software’s impact on a system’s mission effectiveness?"

Mostly cutomers want systems that are:

- On-time
- Within budget
- That satisfy user requirements
- Reliable

Latter two concerns (out of above four) can be refined into two broad objectives for operational testing:

1. That a system’s performance satisfies its requirements as specified in the Operational Requirement Document and related documents.

2. To identify any serious deficiencies in the system design that need correction before full rate production.

Following the path from the system level to software two generic reasons for testing software are:

- Test for defects so they can be fixed - Debug Testing
- Test for confidence in the software - Operational Testing

Debug testing is usually conducted using a combination of functional test techniques and structural test techniques. The goal is to locate defects in the most cost-effective manner and correct the defects, ensuring the performance satisfies the user requirements.

Operational testing is based on the expected usage profile for a system. The goal is to estimate the confidence in a system, ensuring the system is reliable for its intended use.

Task-Based Testing is a variation on operational testing. The particular techniques are not new, rather it leverages commonly accepted techniques by placing them within the context of current operational and acquisition strategies.

Task-based testing, as the name implies, uses task analysis. This begins with a comprehensive framework for all of the tasks that the system will perform. Through a series of hierarchical task analyses, each unit within the service creates a Mission Essential Task List (Mission of System).

These lists only describe "what" needs to be done, not "how" or "who." Further task decomposition identifies the system and people required to carry out a mission essential task. Another level of decomposition results in the system tasks (i.e. functions) a system must provide. This is, naturally, the level in which developers and testers are most interested. From a tester’s perspective, this framework identifies the most important functions to test by correlating functions against the mission essential tasks a system is designed to support.

This is distinctly different from the typical functional testing or "test-to-spec" approach where each function or specification carries equal importance. Ideally, there should be no function or specification which does not contribute to a task, but in reality there are often requirements, specifications, and capabilities which do not or minimally support a mission essential task. Using task analysis, one identifies those functions impacting the successful completion of mission essential tasks and highlights them for testing.


Operational Profiles: The process of task analysis has great benefit in identifying what functions are the most important to test. However, the task analysis only identifies the mission essential tasks and functions, not their frequency of use. Greater utility can be gained by combining the mission essential tasks with an operational profile an estimate of the relative frequency of inputs that represent field use. This has several benefits:

1. Offers a basis for reliability assessment, so that the developer can have not only the assurance of having tried to improve the software, but also has an estimate of the reliability actually achieved.

2. Provides a common base for communicating with the developers about the intended use of the system and how it will be evaluated.

3. When software testing schedules and budgets are tightly constrained, this design yields the highest practical reliability because if failures are seen they would be the high frequency failures.


The first benefit has the advantage of applying statistical techniques in:

- The design of tests
- The analysis of resulting data

Software reliability estimation methods such as Task Analysis are available to estimate both the expected field reliability and the rate of growth in reliability. This directly supports an answer to the question about software’s impact on a system’s mission effectiveness.

Operational profiles are criticized as being difficult to develop. However, as part of its current operations and acquisition strategy, some organizations inherently develops an operational profile. At higher levels, this is reflected in the following documents:

- Analysis of Alternatives
- Operational Requirements Document (ORD)
- Operations Plans
- Concept of Operations (CONOPS) etc.

Closer to the tester’s realm is the interaction between the user and the developer which the current acquisition strategy encourages. The tester can act as a facilitator in helping the user refine his or her needs while providing insight to the developer on expected use. This highlights the second benefit above the communication between the user, developer, and tester.

Despite years of improvement in the software development process, one still sees systems which have gone through intensive debug testing (statement coverage, branch coverage, etc.) and "test-to-spec," but still fail to satisfy the customer’s concerns (that I mentioned above). By involving a customer early in the process to develop an operational profile, the most needed functions to support a task will be developed and tested first, increasing the likelihood of satisfying the customer’s four concerns. This third benefit is certainly of interest in today’s environment of shrinking budgets and manpower, shorter schedules (spiral acquisition), and greater demands on a system.


Task-Based Software Testing

Thus, Task-based software testing is the combination of a task analysis and an operational profile. The task analysis helps partition the input domain into mission essential tasks and the system functions which support them. Operational profiles, based on these tasks, are developed to further focus the testing effort.

Debug Testing

Debug testing is directed at finding as many bugs as possible, by either sampling all situations likely to produce failures using methods like code coverage & specification criteria etc, or concentrating on those that are considered most likely to produce failures like stress testing and boundary testing methods.
Survey of unit testing methods are examples of debug testing methods. These include such techniques as statement testing, branch testing, basis path testing, etc. Typically associated with these methods are some criteria based on coverage, thus they are sometimes referred to as coverage methods. Debug testing is based on a tester’s hypothesis of the likely types and locations of bugs.

Consequently, the effectiveness of this method depends heavily on whether the tester’s assumptions are correct.

If a developer and/or tester has a process in place to correctly identify the potential types and locations of bugs, then debug testing may be very effective at finding bugs. If a "standard" or "blind" approach is used, such as statement testing for its own sake, the testing effort may be ineffectual and wasted. A subtle hazard of debug testing is that it may uncover many failures, but in the process wastes test and repair effort without notably improving the software because the failures occur at a negligible rate during field use.

Integration of Test Methods

Historically, a system’s developer relied on debug testing (which includes functional or "test-to-spec" testing). Testing with the perspective of how the system would by employed was not seen until an operational test agency (OTA) became involved. Even on the occasions when developmental test took on an operational flavor, this is viewed as too late in the process. This historical approach to testing amplifies the weaknesses of both operational and debug testing. I propose that task-based software testing be accelerated to a much earlier point in the acquisition process. This has the potential of countering each respective method’s weaknesses with the other’s strengths.

Conclusion: Task-based Software Testing evaluation is a combination of demonstrated, existing methods (task analysis and operational testing). Its strength lies in matching well with the current operational strategy of mission essential tasks and the acquisition community’s goal to deliver operational capability quickly. By integrating task-based software testing with existing debug testing, the risk of meeting the customer’s four concerns (on-time, within budget, satisfies requirements, and is reliable) can be reduced.