Chapter 4.Test design techniques
Test Documentation Standard [IEEE829].
Test conditions are documented in a Test Design Specification.
Test cases are documented in a Test Case Specification.
Test procedures are documented in a Test Procedure Specification (also known as a test script (or) a manual test script).
Formality of test documentation
Very formal testing would have extensive documentation which is well controlled, and would expect the documented detail of the tests to include the exact and specific input and expected outcome of the test.
Very informal testing may have no documentation at all, or only notes kept by individual testers, but we'd still expect the testers to have in their minds and notes some idea of what they intended to test and what they expected the outcome to be.
Test analysis: identifying test conditions
Test analysis is the process of looking at something that can be used to derive test information. This basis for the tests is called the 'test basis'. It could be a system requirement, a technical specification, the code itself (for structural testing), or a business process.
A test condition is simply something that we could test. If we are looking to measure coverage of code decisions (branches), then the test basis would be the code itself, and the list of test conditions would be the decision outcomes (True and False). If we have a requirements specification, the table of contents can be our initial list of test conditions.
Exhaustive testing (defined as exercising every combination of inputs and preconditions)
Test conditions should be able to be linked back to their sources in the test basis - this is called traceability.
IEEE 829 STANDARDS:
TEST DESIGN SPECIFICATION TEMPLATE
Test design specification identifier
Features to be tested Approach
Refinements Test identification Feature
Pass/fail criteria
Test design: specifying test cases
IEEE 829 Standards:
Test Case Specification Template
Test case specification identifier Output specifications
Test items Environmental needs
Input specifications Special procedural requirements
Intercase dependencies
Test cases can now be prioritized so that the most important test cases are executed first, and low priority test cases are executed later, or even not executed at all.
In order to know what the system should do, we need to have a source of information about the correct behavior of the system - this is called an 'oracle' or a test oracle. This has nothing to do with databases or companies that make them. It comes from the ancient Greek Oracle at Delphi, who supposedly could predict the future with unerring accuracy.
Test implementation: specifying test procedures or scripts
The document that describes the steps to be taken in running a set of tests (and specifies the executable order of the tests) is called a test procedure in IEEE 829, and is often also referred to as a test script.
The test schedule would say when a given script should be run and by whom. The schedule could vary depending on newly perceived risks affecting the priority of a script that addresses that risk, for example. The logical and technical dependencies between the scripts would also be taken into account when scheduling the scripts. For example, a regression script may always be the first to be run when a new release of the software arrives, as a smoke test or sanity check.
IEEE 829 STANDARDS:
TEST PROCEDURE SPECIFICATION TEMPLATE
Test procedure specification identifier
Purpose
Special requirements
Procedure steps
CATEGORIES OF TEST DESIGN TECHNIQUES
Testing performed at different stages in the software development life cycle will find different types of defects
Component testing is more likely to find coding logic defects than system design defects.
Two main categories of testing are static and dynamic. Dynamic techniques are subdivided into three more categories: specification-based (black-box, also known as behavioral Techniques),Structure-based (white-box or structural techniques) and experience-based. Specification-based techniques include both functional and non-functional techniques (i.e. quality characteristics).
Static testing techniques
Static testing techniques do not execute the code being examined and are generally used before any tests are executed on the software. They could be called non-execution techniques.
Specification-based (black-box) testing techniques
These are also known as 'black-box' or input/output-driven testing techniques because they view the software as a black-box with inputs and outputs, but they have no knowledge of how the system or Component is structured inside the box. In essence, the tester is concentrating on what the software does, not how it does it.
Functional testing is concerned with what the system does its features or functions. Non-functional testing is concerned with examining how well the system does something, rather than what it does. Non-functional aspects (also known as quality characteristics or quality attributes) include performance, usability, portability, maintainability, etc.
Structure-based (white-box) testing techniques
Structure-based testing techniques use the internal structure of the software to derive test cases. They are commonly called 'white-box' or 'glass-box' techniques
Experience-based testing techniques
In experience-based techniques, people's knowledge, skills and background are a prime contributor to the test conditions and test cases. The experience of both technical and business people is important.
Where to apply the different categories of techniques
Specification-based techniques are appropriate at all levels of testing (component testing through to acceptance testing) where a specification exists.
Structure-based techniques can also be used at all levels of testing. Developers use structure-based techniques in component testing and component integration testing. Structure-based techniques are also used in system and acceptance testing, but the structures are different.
Experience-based techniques are used to complement specification-based and structure-based techniques, and are also used when there is no specification, or if the specification is inadequate or out of date.
This may be the only type of technique used for low-risk systems, but this approach may be particularly useful under extreme time pressure - in fact this is one of the factors leading to exploratory testing.
SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES
The four specification-based techniques:
• Equivalence partitioning;
• Boundary value analysis;
• Decision tables;
• State transition testing.
Equivalence partitioning and boundary value analysis
Equivalence partitioning
The idea behind the technique is to divide (i.e. to partition) a set of test conditions into groups or sets that can be considered the same (i.e. the system should handle them equivalently), hence 'equivalence partitioning'. Equivalence partitions are also known as equivalence classes - the two terms mean exactly the same thing.
The equivalence-partitioning technique then requires that we need test only one condition from each partition.
Boundary value analysis
Boundary value analysis (BVA) is based on testing at the boundaries between partitions. If you have ever done 'range checking', you were probably using the boundary value analysis technique, even if you weren't aware of it. Note that we have both valid boundaries (in the valid partitions) and invalid boundaries (in the invalid partitions).
As an example, consider a printer that has an input option of the number of copies to be made, from 1 to 99.
To apply boundary value analysis, we will take the minimum and maximum (boundary) values from the valid partition (1 and 99 in this case) together with copies to be made, from 1 to 99.
Designing test cases
The more test conditions that can be covered in a single test case, the fewer test cases will be needed in order to cover all the conditions. This is usually the best approach to take for positive tests and for tests that you are reasonably confident will pass.
British Standard 7925-2 Standard for Software Component Testing
Why do both equivalence partitioning and boundary value analysis?
Technically, because every boundary is in some partition, if you did only boundary value analysis you would also have tested every equivalence partition.
Equivalence partitioning and boundary value analysis are described in BS7925-2 including designing tests and measuring coverage.
Decision table testing
The techniques of equivalence partitioning and boundary value analysis are
Often applied to specific situations or inputs. However, if different combinations
Of inputs result in different actions being taken, this can be more difficult to show using equivalence partitioning and boundary value analysis, which tend to be more focused on the user interface. The other two specification-based techniques, decision tables and state transition testing are more focused on business logic or business rules.
A decision table is a good way to deal with combinations of things (e.g. inputs). This technique is sometimes also referred to as a 'cause-effect' table. The reason for this is that there is an associated logic diagramming technique called 'cause-effect graphing' which was sometimes used to help derive the decision table.
Decision tables provide a systematic way of stating complex business rules, which is useful for developers as well as for testers.
Decision tables can be used in test design whether or not they are used in specifications.
Decision tables aid the systematic selection of effective test cases and can have the beneficial side-effect of finding problems and ambiguities in the specification. It is a technique that works well in conjunction with equivalence partitioning. The combination of conditions explored may be combinations of equivalence partitions.
Using decision tables for test design
The first task is to identify a suitable function or subsystem that has a behavior which reacts according to a combination of inputs or events. The behavior of interest must not be too extensive (i.e. should not contain too many inputs) otherwise the number of combinations will become cumbersome and difficult to manage. It is better to deal with large numbers of conditions by dividing them into subsets and dealing with the subsets one at a time.
Once you have identified the aspects that need to be combined, then you put them into a table listing all the combinations of true and false for each of the aspects.
*Refer Foundation ISTQB Syllabus for brief understanding (Decision table concept)
Take an example of a loan application, where you can enter the amount of the monthly repayment or the number of years you want to take to pay it back (the term of the loan). If you enter both, the system will make a compromise between the two if they conflict. The two conditions are the loan amount and the term, so we put them in a table (see Table 4.2).
TABLE 4.2 Empty decision table
Conditions Rule 1 Rule 2 Rule 3 Rule 4
Repayment amount has been entered
Term of loan has been entered
TABLE 4.7 Decision table with outcomes in one row
Conditions Rule 1 Rule 2 Rule 3 Rule 4
Repayment amount has T T F F
been entered
Term of loan has been T F T F
entered
Actions/Outcomes
Result Error Process loan Process Error
Message amount term message
The final step of this technique is to write test cases to exercise each of the four rules in our table. In this example we started by identifying the input conditions and then identifying the outcomes. However in practice it might work the other way around - we can see that there are a number of different outcomes, and have to work back to understand what combination of input conditions actually drive those outcomes. The technique works just as well doing it in this way and May well is an iterative approach as you discover more about the rules that drive the system.
State transition testing
State transition testing is used that the system can be in a (finite) number of different states, and the transitions from one state to another are determined by the rules of the 'machine'. This is the model on which the system and the tests are based. Any system where you get a different output for the same input, depending on what has happened before, is a finite state system.
*State transition testing process is explained by using ATM card process and opened word doc. For detailed understanding refer ISTQB syllabus.
A state transition model has four basic parts:
• The states that the software may occupy (open/closed or funded/insufficient funds);
• The transitions from one state to another (not all transitions are allowed);
• The events that cause a transition (closing a file or withdrawing money);
• The actions that result from a transition (an error message or being given your cash).
Note that in any given state, one event can cause only one action, but that the same event - from a different state - may cause a different action and a different end state.
One of the advantages of the state transition technique is that the model can be as detailed or as abstract as you need it to be. Where a part of the system is more important a greater depth of detail can be modeled. Where the system is less important the model can use a single state to signify what would otherwise be a series of different states.
Testing for invalid transitions
Deriving tests only from a state graph (also known as a state chart) is very good for seeing the valid transitions, but we may not easily see the negative tests, where we try to generate invalid transitions. In order to see the total number of combinations of states and transitions, both valid and invalid, a state table is useful.
The state table lists all the states down one side of the table and all the events that cause transitions along the top (or vice versa). Each cell then represents a state-event pair. The content of each cell indicates which state the system will move to, when the corresponding event occurs while in the associated state. This will include possible erroneous events - events that are not expected to happen in certain states. These are negative test conditions.
State transition testing is described in BS7925-2, including designing tests and coverage measures.
Use case testing
Use case testing is a technique that helps us identify test cases that exercise the whole system on a transaction by transaction basis from start to finish.
A use case is a description of a particular use of the system by an actor (a user of the system). Each use case describes the interactions the actor has with the system in order to achieve a specific task. Actors are generally people but they may also be other systems. Use cases are a sequence of steps that describe the interactions between the actor and the system.
Use cases are defined in terms of the actor, not the system, describing what the actor does and what the actor sees rather than what inputs the system expects and what the system’ outputs. They often use the language and terms of the business rather than technical terms, especially when the actor is a business User. They serve as the foundation for developing test cases mostly at the system and acceptance testing levels.
Use cases can uncover integration defects, that is, defects caused by the incorrect interaction between different components. Used in this way, the actor may be something that the system interfaces to such as a communication link or sub-system.
* For detailed understanding of use case testing refer ISTQB syllabus
Use cases describe the process flows through a system based on its most likely use. This makes the test cases derived from use cases particularly good for finding defects in the real-world use of the system (i.e. the defects that the users are most likely to come across when first using the system). Each use case usually has a mainstream (or most likely) scenario and sometimes additional alternative branches (covering, for example, special cases or exceptional conditions). Each use case must specify any preconditions that need to be met for the use case to work. Use cases must also specify post conditions that are observable results and a description of the final state of the system after the use case has been executed successfully.
STRUCTURE-BASED OR WHITE-BOX TECHNIQUES
1. Using structure-based techniques to measure coverage and design tests
Structure-based techniques serve two purposes: test coverage measurement and structural test case design. They are often used first to assess the amount of testing performed by tests derived from specification-based techniques, i.e. to assess coverage. They are then used to design additional tests with the aim of increasing the test coverage.
Structure-based test design techniques are a good way of generating additional test cases that are different from existing tests. They can help ensure more breadth of testing, in the sense that test cases that achieve 100% coverage in any measure will be exercising all parts of the software from the point of view of the items being covered.
What is test coverage?
Test coverage measures in some specific way the amount of testing performed by a set of tests
Wherever we can count things and can tell whether or not each of those things has been tested by some test, then we can measure coverage. The basic coverage measure is
Number of coverage items exercised
Coverage = --------------------------------------------------- x 100%
Total number of coverage items
Where the 'coverage item' is whatever we have been able to count and see whether a test has exercised or used this item.
100% coverage does not mean 100% tested! Coverage techniques measure only one dimension of a multidimensional concept. Two different test cases may achieve exactly the same coverage but the input data of one may find an error that the input data of the other doesn't.
One drawback of code coverage measurement is that it measures coverage of what has been written, i.e. the code itself.
If a specified function has not been implemented, specification-based testing techniques will reveal (display) this. If a function was omitted from the specification, then experience-based techniques may find it. But structure-based techniques can only look at a structure which is already there.
Types of coverage
Test coverage can be measured based on a number of different structural elements in a system or component. Coverage can be measured at component-testing level, integration-testing level or at system- or acceptance-testing levels.
For example, at system or acceptance level, the coverage items may be requirements, menu options, screens, or typical business transactions. Other coverage measures include things such as database structural elements (records, fields and sub-fields) and files. It is worth checking for any new tools, as the test tool market develops quite rapidly.
At integration level, we could measure coverage of interfaces or specific interactions that have been tested. The call coverage of module, object or procedure calls can also be measured (and is supported by tools to some extent).
We can measure coverage for each of the specification-based techniques as well:
• EP: percentage of equivalence partitions exercised (we could measure valid and invalid partition coverage separately);
• BVA: percentage of boundaries exercised (we could also separate valid and invalid boundaries)
• Decision tables: percentage of business rules or decision table columns tested;
• State transition testing: there are a number of possible coverage measures:
- Percentage of states visited
- Percentage of (valid) transitions exercised (this is known as Chow's 0- switch coverage)
- Percentage of pairs of valid transitions exercised ('transition pairs' or Chow's 1-switch coverage) - and
longer series of transitions, such as transition triples, quadruples, etc.
- Percentage of invalid transitions exercised (from the state table)
When coverage is discussed by business analysts, system testers or users, it most likely refers to the percentage of requirements that have been tested by a set of tests. This may be measured by a tool such as a requirements management tool or a test management tool.
However, when coverage is discussed by programmers, it most likely refers to the coverage of code, where the structural elements can be identified using a tool, since there is good tool support for measuring code coverage. We will cover statement and decision coverage shortly.
Statements and decision outcomes are both structures that can be measured in code and there is good tool support for these coverage measures. Code coverage is normally done in component and component integration testing - if it is done at all. If someone claims to have achieved code coverage, it is important to establish exactly what elements of the code have been covered, as statement coverage (often what is meant) is significantly weaker than decision coverage or some of the other code-coverage measures.
How to measure coverage
An intrusive coverage measurement tool that alters the code by inserting instrumentation:
1 Decide on the structural element to be used, i.e. the coverage items to be counted.
2 Count the structural elements or items.
3 Instrument the code.
4 Run the tests for which coverage measurement is required.
5 Using the output from the instrumentation determine the percentage of elements or items exercised.
Instrumenting the code (step 3) implies inserting code alongside each structural element in order to record when that structural element has been exercised. Determining the actual coverage measure (step 5) is then a matter of analyzing the recorded information.
Coverage measurement of code is best done using tools. They increase quality by ensuring that more structural aspects are tested, so defects on those structural paths can be found. They increase productivity and efficiency by highlighting tests that may be redundant, i.e. testing the same structure as other tests.
In common with all structure-based testing techniques, code coverage techniques are best used on areas of software code where more thorough testing is required. Safety-critical code; code that is vital to the correct operation of a system, and complex pieces of code are all examples of where structure-based Techniques are particularly worth applying.
Structure-based test case design
If you are aiming for a given level of coverage (say 95%) but you have not Reached your target (e.g. you only have 87% so far), then additional test cases can be designed with the aim of exercising some or all of the structural elements not yet reached. This is structure-based test design. These new tests are then run through the instrumented code and a new coverage measure is calculated. This is repeated until the required coverage measure is achieved. Ideally all the tests ought to be run again on the un-instrumented code.
2. Statement coverage and statement testing
Number of statements exercised
Statement coverage = --------------------------------------------- x 100%
Total number of statements
Statement coverage is calculated by:
Studies and experience in the industry have indicated that what is considered reasonably thorough black-box testing may actually achieve only 60% to 75% statement coverage. Typical ad hoc testing is likely to be around 30% - this leaves 70% of the statements untested.
Different coverage tools may work in slightly different ways, so they may give different coverage figures for the same set of tests on the same code, although at 100% coverage they should be the same.
We will regard each line as a statement. A statement may be on a single line, or it may be spread over several lines. One line may contain more than one statement, just one statement, or only part of a statement. Some statements can contain other statements inside them.
* If detailed explanation of statement coverage and statement testing refer ISTQB Syllabus.
3. Decision coverage and decision testing (* refer ISTQB Syllabus for detailed understanding)
A decision is an IF statement, a loop control statement (e.g. DO-WHILE or REPEAT-UNTIL), or a CASE statement, where there are two or more possible exits or outcomes from the statement. With an IF statement, the exit can either be TRUE or FALSE, depending on the value of the logical condition that comes after IF. With a loop control statement, the outcome is either to perform the code within the loop or not - again a True or False exit. Decision coverage is calculated by:
Number of decision outcomes exercised
Decision coverage = -------------------------------------------------------- x 100%
Total number of decision outcomes
Reasonably thorough functional testing may achieve only 40% to 60% decision coverage. Typical ad hoc testing may cover only 20% of the decisions, leaving 80% of the possible outcomes untested. Even if your testing seems reasonably thorough from a functional or specification-based perspective, you may have only covered two-thirds or three-quarters of the decisions. Decision coverage is stronger than statement coverage. It 'subsumes' statement coverage - this means that 100% decision coverage always guarantees 100% statement coverage. Any stronger coverage measure may require more test cases to achieve 100% coverage.
We saw earlier that just one test case was required to achieve 100% statement coverage. However, decision coverage requires each decision to have had both a True and False outcome. Therefore, to achieve 100% decision coverage, a second test case is necessary
So one test is sufficient for 100% statement coverage, but two tests are needed for 100% decision coverage. Note that 100% decision coverage guarantees 100% statement coverage
4. Other structure-based techniques
Branch coverage is closely related to decision coverage and at 100% coverage they give exactly the same results. Decision coverage measures the coverage of conditional branches; branch coverage measures the coverage of both conditional and unconditional branches
Other control-flow code-coverage measures include linear code sequence and jump (LCSAJ) coverage, condition coverage, multiple condition coverage (also known as condition combination coverage) and condition determination coverage (also known as multiple condition decision coverage or modified condition decision coverage, MCDC). This technique requires the coverage of all conditions that can affect or determine the decision outcome.
Another popular, but often misunderstood, code-coverage measure is path coverage. Sometimes any structure-based technique is called 'path testing'.
Structure-based measures and related test design techniques are described in [BS7925-2].
5. Experience –Based Techniques
Error guessing
Error guessing is a technique that should always be used as a complement to other more formal techniques.
Error-guessing approach, used after more formal techniques have been applied to some extent, can be very effective.
There are no rules for error guessing.
A structured approach to the error-guessing technique is to list possible defects or failures and to design tests that attempt to produce them.
Exploratory testing
Exploratory testing is a hands-on approach in which testers are involved in minimum planning and maximum test execution. The planning involves the creation of a test charter, a short declaration of the scope of a short (1 to 2 hour) time-boxed test effort, the objectives and possible approaches to be used.
The test design and test execution activities are performed in parallel typically without formally documenting the test conditions, test cases or test scripts. This does not mean that other, more formal testing techniques will not be used.
Test logging is undertaken as test execution is performed, documenting the key aspects of what is tested, any defects found and any thoughts about possible further testing.
A key aspect of exploratory testing is learning: learning by the tester about the software, its use, its strengths and its weaknesses. As its name implies, exploratory testing is about exploring, finding out about the software, what it does, what it doesn't do, what works and what doesn't work. The tester is constantly making decisions about what to test next and where to spend the (limited) time.
This is an approach that is most useful when there are no or poor specifications and when time is severely limited. It can also serve to complement other, more formal testing, helping to establish greater confidence in the software.
- Choosing a Test Technique
One of the benefits of structure-based techniques is that they can find things in the code that aren't supposed to be there, such as 'Trojan horses' or other malicious code.
if there are parts of the specification that are missing from the code, only specification-based techniques will find that -structure-based techniques can only test what is there.
If there are things missing from the specification and from the code, then only experience-based techniques would find them.
State transition testing is unlikely to find boundary defects.
The internal factors that influence the decisions about which technique to use are:
• Models used • Tester knowledge I experience • Likely defects • Test objective
• Documentation • Life cycle model
The external factors that influence the decisions about which technique to use are:
• Risk • Customer I contractual requirements • Type of system
• Regulatory requirements • Time and budget
No comments:
Post a Comment