In this blog post, we pin down the reasons that make system testing a time-consuming task. But – we show you how you can overcome these pain points in system testing by defining your tests in a model-based way.
Ensuring a certain quality of your product is the #1 way to make sure that you live up to your customer’s expectations. Testing is usually the most efficient way to ensure a certain level of quality before shipping a product. However, it is usually also the most annoying and time-consuming task. Especially when it comes to testing an already integrated system, finding ways to test it, performing these tests, and locating errors contains a lot of time-consuming, repetitive tasks. Even though the test execution can be automated via so-called system tests, writing and maintaining the code for these tests is still cumbersome.
The pain in system testing
Testing an already integrated system via its graphical user interface or some hardware I/Os is usually a real pain for all people involved. If you dig a bit deeper, you can pin down this pain into the following three points:
- Maintaining system tests. Most effort in system testing usually lies in continuously updating created tests to still be executable on the current version of the system. One small change in the tested system usually requires a lot of effort for every considered test case to adapt it to the new system requirements. But before that, even finding out which test cases need to be adapted after a system change can be very challenging.
- Communication. The #1 reason that makes specifying and maintaining system tests difficult is that they require frequent communication between many different stakeholders. This usually involves at least
– a domain expert who knows the requirements of the system to be built
– a requirements engineer who aims to bring these requirements from the domain expert’s head into writing (e.g. into diagram-like visualizations on PowerPoint slides) a test expert to specify test cases based on this high-level requirements specification, and
– a test engineer to create test code based on the specified test cases
The communication between these roles even goes beyond specifying and adapting system tests though. For example, once a created test fails, the tester noticing this failure has to communicate with the requirements engineer to identify whether the test failure is due to (i) a defect in the system, (ii) a wrongly implemented system test, or (iii) a change in requirements. Afterwards, the result of this discussion needs to be communicated to (i) a test engineer to adapt the system test accordingly, or (ii) to the system engineer to fix a bug in the system. - Missing test integrity due to copy-pasting. Unfortunately, copy-paste is still a common practice in system testing. Well, it also saves quite some time in the short term (similar code does not need to be created twice). However, copy-pasting makes it almost impossible to maintain system tests in the long term. Why? Just imagine copying your base test case 20 times with some minor adaptations. Every time this base test case needs to be changed afterwards, you have to perform the same changes in its 20 other variants as well – manually! That is however not the biggest issue with this copy-pasting yet. Imagine only performing a change for 19 variants, because you missed one of them. Finding out about this issue again costs valuable time that you have to spend on maintaining system tests after even just a small change in the tested system – particularly when testing systems that involve hardware, where executing tests might take several days.
Optimizing the effort in system testing using model-based methods
Now that we know the main pain points in system testing, let’s have a look at how we can do things better. What this comes down to is defining system tests in a model-based way. Model-based system test definitions mean that we use some sort of high-level notation for describing several test cases. This high-level notation is optimized for efficiency, maintainability, and reusability, compared to common general-purpose languages like C, C#, or Python. Usually, this notation is also graphical, meaning that you can create system tests by drag- and dropping different elements. The result is a so-called test model as a representation of a list of system tests, together with the dependencies and shared commonalities of these tests. As this model is defined in a graphical high-level notation, it’s usually also easier for different stakeholders to interpret it, compared to reading plain test code. The test code which is run on the system is simply generated from the test model.
The benefits of model-based test definitions
In its essence, improving system testing with model-based methods optimizes your testing efforts in the following ways:
- Improved test integrity through encapsulation. Within a test model, you can define abstract building blocks that encapsulate behavior that is used by several test cases in this model. Besides avoiding defining the same behavior over and over again, these building blocks have the huge benefit of improving test integrity. Once you have to adapt the underlying requirements, the corresponding change is only performed on the common building block – and then automatically propagated to all test cases that make use of this building block.
This definitely sounds better than copy-pasting test code, doesn’t it? - Reduced communication effort through code generation: Once you have defined a test model, one of the biggest benefits is that you can automatically generate the code that is actually executed on the system. This means that the test can be defined and maintained solely through its high-level specification. As this specification is optimized for readability and reusability, it is easily understood and used by everyone. But what’s the big benefit of this? Well, it enables people with different backgrounds to talk to each other, using a shared document. Different stakeholders now have a test model as a common basis for discussion, without having to exchange emails, PowerPoint slides, and code at the same time. This creates a common understanding of test requirements and delivers a formal definition of system test cases that can be executed on the system without considerable overhead.
- Improved maintainability through improved adaptability. Ultimately, using model-based methods helps reduce the #1 pain in system testing: maintaining test cases. The used graphical high-level representation already makes it much easier to apply changes to a test model, compared to trying to understand and adapt test code. But the nature of this notation also leverages many further benefits that simplify adaptations to the test model in practice. By defining similar test cases in the same test model, similarities between these test cases become evident. Model-based notations usually provide dedicated elements to describe these commonalities, as well as their differences.
For example, these model elements can leverage defining tests for different system variants in the same file. Once you have to adapt or add one variant, you can simply check which points of the test model are affected by this change. Customizing predefined elements helps you to understand your test model even faster. For example, you could color the elements of the base test sequence in green, and error cases that diverge from this base sequence in red.
In this article, we have shown you how you can reduce the effort of defining and maintaining your system tests. We have investigated in detail how model-based methods can reduce the main pain points when it comes to system testing.
Start with model-based system testing now
Product information: Model-based system testing with vTESTstudio
