A guide on how to choose and use testing techniques

“When am I to use which testing technique?”. This is the question we will be exploring in this blog post. Our focus lies on unit tests, mostly done by developers.

One possible approach is described below, which is intended to serve as a guide for the reader. Following this guide, however, does not guarantee that the program will be error-free! It rather describes what we consider to be a reasonable approach. It is up to you to decide whether this approach is helpful to solve your issue! Over time and as you gain experience in testing, you will be able to assess and decide which testing techniques are best suited for which constellations. You will then no longer need this guide.

First of all, a few remarks: When naming the test technique, we rather mean the test execution because the specification of test cases are to be generated at an early stage of software development. Another factor that needs to be determined in advance is how detailed the tests are to be carried out, i.e. what degree of coverage is to be achieved and thus verified by the individual testing techniques.

The guide describes the approach as an If/When – Then sequence. As a rule, several If/When conditions will apply. Consequently, the applicable Then recommendations must be put into practice.

Prologue

When the programming has been completed, the compiler no longer issues messages and the test is to begin,
Then the code must be checked by a static analysis tool.

When all findings of the tool have been checked and eliminated,
Then a code review can be useful.

When all the results of the review have been clarified,
Then the test can be started.

When the test is due to start,
Then we use a tool to determine the code coverage during each test run and after the test runs to determine whether the predefined coverage has been achieved.

Off we go!

Note: It goes without saying that a found failure must be corrected and therefore the correction (and the testing of the correction) is not explicitly listed below.

If the test object has one or more main functions
Then these are to be checked by means of Happy-Path-Tests (or smoke test).

If the Happy-Path-Test fails,
Then it is not worth continuing with the selection of testing techniques. This would be a case of severe misunderstandings, which must first be resolved. Afterwards the source code must be adapted or corrected accordingly.

If the test object includes parameters which are largely independent and do not influence each other, and if these parameters have been defined for specific data ranges or values,
Then the equivalence partitioning must be carried out to generate the test cases. It makes sense to take the boundary value analysis into account as well, since the boundaries of the equivalence classes are the “interesting” test data.

If the test object has many parameters and these can be combined in one way or another,
Then the classification tree method is helpful to keep an overview and to use the tree to compile the test cases.

If the parameters of the test object can be combined randomly, and if this is a likely situation when running the program in practice,
Then combinatorial test design techniques must be carried out. Since testing all possible combinations is usually too complex, we recommend pair-wise testing.

If the parameters of the test object cannot be combined randomly, but depend on each other,
Then decision table testing is a good choice. The decision table shows which of the possible combinations are relevant for the test object and which ones must therefore be tested.

If the history (the run so far), the current status of the system or an test object is of relevance,
Then state transition testing must be carried out. Sequences of test cases to check the states and the state transitions must be implemented and tested. Whether the n-switch coverage (consideration of the length of the sequences) or a transition table or both are to be used is to be decided case by case.

If the inputs are subject to a syntax, i.e. they must comply with a given format or certain rules,
Then syntax testing is to be carried out to check whether the syntax rules have been complied with or violated.

If random inputs appear to be useful,
Then random testing can be used. In this context, it must be considered that the expected results and the results generated by the test run can usually not be compared because there are no expected results for randomly generated inputs.

If one or more testing techniques have been selected,
Then it must be checked whether additional testing techniques might be useful to check the test object.

Are you done now?

If the test object had been tested based on test cases that are usually generated by applying several testing techniques,
Then, it is to be determined which parts of the test object have not yet been executed, corresponding to the previously defined requirements for the coverage to be achieved (statement, condition, …).

If there are program parts that have not yet been executed,
Then supplementary test cases must be specified and carried out to execute these program parts.

If this does not work,
Then it is necessary to figure out the reasons why the program parts cannot be executed. The source code must then be corrected accordingly.

If the test object includes decisions which are based on several conditions linked by Boolean operators,
Then these must be tested separately with one of the three test techniques to prove conditions – depending on the required depth. (decision testing,  multiple condition testing, modified condition decision coverage testing (MCDC) recommendation: MCDC).

If loops play an important role in the test object
Then it must be examined whether the requirements for the loop test (e.g. bypass loop, if possible, run through loop only once, or several times) have been already fulfilled by the previous test cases or whether additional test cases must be generated.

But now we are done – aren’t we?

No, not yet. Now you, your experience, imagination, and wealth of ideas are in demand!

If you still have an inkling where it could go wrong – and that’s almost always the case -,
Then express your idea in a test charter and test it exploratively. The exploratory testing can be stopped, if you have the „feeling“ testing is sufficient.

If you can still think of something that cannot be expressed in a test charter,
Then free testing is the order of the day! When you run out of ideas, free testing is done.

If you remember errors made in previous projects or have a list of failures,
Then it makes sense to use these lists and the error guessing to complete the test activities.

Epilogue

When you have thought of everything and thus have tested your software sufficiently and adequately,
Then you can check-in the software with confidence and satisfaction and enjoy your evening off work. ⛱

Keep in mind

Further information and download

• “Lean Testing für den  C++ – Programmierer (in German)” by Andreas Spillner and Ulrich Breymann contains the described selection of testing techniques, which are described in the book in detail. In addition, each technique is supported with examples that always extend to the creation of test cases. C++ has been chosen as programming language. The program examples are kept in a way that they are understandable without problems and can be translated into other programming languages.
• Further information on the book including a free download of all programming examples
• Free download of the open book, with decision table testing, combinatorial test design techniques

Further readings

• Andreas Spillner, Tilo Linz: Software Testing Foundations: A Study Guide for the Certified Tester Exam, 5th edition, July 2021
• Andreas Spillner, Tilo Linz: Basiswissen Softwaretest (in German), 6. revised and updated edition, dpunkt.verlag, 2019
• ISO/IEC/ IEEE 29119-4:2015. Software and systems engineering — Software testing — Part 4: Test techniques