When it comes to system level testing, virtual environments offer several strong benefits for you. In this blog post, I want to highlight some of the most important ones:
Early system level tests
Perhaps the biggest benefit is that you can test much earlier on the system level. In a virtual environment, you simply don’t need the target device, and you can start testing right away. This reduces time to market when the actual hardware is simply not available yet. You can also run system level tests alongside with your unit tests. This gives you an early understanding of how your software behaves once it is integrated. And since everything is virtual, it is easy to make these system level part of your CI/CT pipeline, so you can merge changes with high confidence.
Scalable system level tests
When performing system level tests with real hardware, a major limiting factor is the scalability. This is due to the availability of target devices and test equipment. Often, the hardware is still under development and it is even harder to get your hands on a prototype. Thus, test engineers will compete for these resources, which limits the number of tests you can run in parallel. When using virtualization, on the other hand, all of these limitations go away. Every test engineer can create as many virtual environments as needed. You probably still need to run some tests on the target hardware. But you can be sure that the number of these tests is reduced to a minimum.
Simple test specification and execution
Traditional system level tests are hard to reproduce and are prone to errors. Often tests are only specified as a list of manual steps, e.g., in a text document. A test engineer follows these steps, sets up the test equipment and notes down the behavior. Not only is it easy to make a mistake here, but it is also almost impossible to reproduce tests exactly. This is due to the interaction with analog values, e.g., voltages, which always need to account for small variances.
In a virtual environment, you can specify your system level tests as a program. These test programs will apply the exact values every time. And you don’t need to account for variances of analog values because no hardware is involved. Test programs also fit well into DevOps workflows. The tests can be easily executed by your CI/CT pipeline, and you can clearly see how a test changed in a pull request.
Simple fault diagnostics
Hardware debuggers have come a long way and allow you to debug on the target device comfortably. But fully understanding a problem often requires many iterations of debugging. And for each iteration, you have to setup your environment again. When you do this with the target device, resetting the physical environment is a time-consuming task. And, if you are unlucky, it can be hard to hit the breakpoint at the right time. All of this goes away if you rely on virtualization. Not only is it easy to reset your environment. But you can also pause the execution of the environment, making debugging a much easier task.
Independence of back-end services availability
These days, many embedded devices are smart devices that require back-end services for proper operation. To setup a back-end service for testing can be a complex task. E.g., you have to clean and fill the underlying database for every test. This becomes easier if you virtualize the service with a model. The benefit of models is that they guarantee reproducible results. They also do not need a live internet connection, which might not be available otherwise.
System testing in critical or fatal scenarios
Last but not least a benefit that is not obvious but even more so important. Ensuring that your devices operate correctly in critical or fatal scenarios is often hard or costly. For example, consider that your device has to safely shut down if a motor breaks. To emulate such a motor break in the real world is at least a costly task. But in a virtual environment, it can simply be a feature of your model. There are many more situations that are hard to emulate in the real world but easy in the virtual one. E.g., testing input values out of the specified ranges and ensuring that your device does not do harm.
How to virtualize your system tests
Learn more in our video. In this video, we show how to virtualize all aspects of your system level tests. This includes:
- Your system under test
- The physical environment
- The software environment
