A test bed is an environment where you bring your IoT system under test (SUT) to run. Depending on the level of detail that the test bed provides, your SUT won’t note the difference whether it is executed in its natural environment or in a test bed environment. Of course, this is easier said than done because some SUTs need a very complex environment to run or start up at all. The goal in the end, however, is to test our SUT as thoroughly as possible – depending on the use case, of course. In IoT applications, we can take a look at two different parts of the system: The IoT device itself, as well as the backend application. Both parts of the IoT device require a slightly different approach or mindset if we want to create the test bed. The following four setups show a basic approach.
Setup 1: Fully virtualized test bed for your IoT device software
IoT test bed characteristics:
You already know about the function of your device. You have a pretty clear idea of what the software part should look like – or you already have an implementation of your function.
Challenges:
How to run and test your software in a test bed which is as realistic as possible although you don’t have the real hardware available yet?
Advantages:
Typically an IoT device is connected to sensors or actuators which interact with the environment on a physical level, e.g. measuring temperatures or air pressures and controlling some valves or switches. But what do you do if you only have a piece of software which implements your function?
Well, in this case, the goal is to provide an environment to the SW which includes all the inputs and outputs needed for your SW to operate correctly. But due to the fact that you don’t have your sensors and actuators available in real, you have to provide them in a simulated form. This simulation, however, should produce real (sensor) inputs for your SW under test and on the other hand, should take the real actuator output of the SW. This so-called remaining system simulation therefore has to contain a realistic model of the real environment in which the SW has to run in the end. If the sensor output wouldn’t match to the according inputs the overall simulation environment would be useless. Apart from the fact that you don’t need your real system in order to test your function, the full simulation has some more interesting advantages.
1.) Full control over the simulation
Full control means for instance that you are fully in charge of the simulation time in any part of your system. Since everything is simulated, also the simulation time is simulated, i.e. there’s no correlation to the wall clock in the first place. This means it gives you the possibility to debug your system in the same way as you would debug a desktop application. Typically in an embedded system, it is not so easy to debug because as soon you have real components, such as the microcontroller itself, you have timers and tasks which cannot be stopped that easily whenever you hit a breakpoint.
Another aspect is that you can decide about the level of detail in your environment models. For example, if you want to develop an automation for smart gardening you may have to deal with temperatures, humidity, pump switches and so on, which can be provided in an easy model or even table-based. Whereas if you develop an operation robot for a medical application you need to have a very detailed and sophisticated model of human tissue and the anatomy of the body.
2.) Fault injection
Since everything is simulated you can manipulate the environment without doing any harm. You can even reproduce tests that would very easily have caused serious damage in the real world. It won’t be necessary to go to your boss and tell him that you need another power supply because the last one was blown up in a shortage to ground test.
The simulated environment gives you the possibility to intensively test the corner cases of your system. Things you wouldn’t do too often in a real system due to the above mentioned reasons. However, testing these corner cases immensely increases the quality of your system.
3.) Easily deployable
For example, you want a colleague to also have a look at it after you detected some bug. Or you may want to parallelize your IoT test bed to do mass testing in order to increase test coverage and – of course – also to increase the quality of your product.
Setup 2: Fully virtualized test bed for your IoT backend software
IoT test bed characteristics:
You are developing the backend part of an IoT system. Since you’re focused on cloud technologies you don’t want to take care of any embedded system on the edge. You expect from your test bed that it provides everything you need to develop and test your application.
Challenges:
How to get your backend software connected to the “real world” without the real world?
Advantages:
Taking Setup 1 into account, it is just a question of how to cut your system into pieces. Then you have to decide which pieces should be simulated. In the case of Setup 2, we are concentrating on the backend part. So the IoT test bed should simulate the sensors and the connected devices. Again, since every part is simulated you can benefit from the same advantages as above:
- ) Full control over the simulation
- ) Fault injection
- ) Easily deployable
Setup 3: Test your real device software connected to a virtualized environment
IoT test bed characteristics:
At some point in your development workflow, you may want to leave the virtual world and connect some real device to your test bed.
Challenges:
How to connect a real device to a simulated test bed by utilizing the concrete protocols for serialization?
Advantages:
Again, as described in the setups above you are able to test your device in a realistic environment. Of course, if there’s real HW involved you may lose the advantage of fully controlling the simulation time of your system, i.e. the question of debugging becomes more complex. There’s still the possibility to do fault injection and the remaining system can also be easily deployed.
However, there’s one more advantage:
The reuse of tests. What do I mean by this?
Typically you don’t develop your system completely in one of the setups described here. Very often you develop it in a workflow process that brings you from a pure virtual setup (e.g. setup 1) into a setup that includes more real components (e.g. setup 3). But regardless of where you are in your process, the test objective is very similar or even the same. This means you can re-use tests you have developed at the very beginning of your development process also in later phases – often without modifying them. This saves a lot of development time and – again – increases your confidence in your product and the quality of the product itself.
Setup 4: Test your IoT backend software connected to a virtualized environment
IoT test bed characteristics:
The same applies as in setup 3. However, here you develop a backend application rather than an IoT device. So your backend application becomes the “real” part.
Challenges:
Again the challenge is to connect your backend application (SUT) to a remaining system that is able to provide a realistic environment for your SUT in order to work properly.
Advantages:
The advantages are obvious and the same as above. To formalize testing it is essential to provide an environment to your system which is able to stimulate your SUT in a plausible way and also to consume the output of your SUT and produce new outputs (closed-loop). Formalized tests are reproducible and – if designed correctly – provide a bigger test coverage compared to e.g. explorative testing which is more or less a “try and error” approach that is most likely not suitable to increase the quality of your product. Let alone the idea that you will probably be able to enjoy a good night’s sleep because you have more confidence in your product.
Conclusion – What’s important when setting up an IoT test bed?
Testing is an activity which often neglected or postponed to a later point in time in the development process although it is meanwhile common knowledge that the later you detect a bug the more expensive it becomes to fix it.
If you have the right tools at hand which help you accomplish this task in an easy way there’s no need to neglect or postpone anymore. So what should this tool be able to do?
- Support you in setting up a remaining system simulation regardless of whether you are focused on the device or the backend
- Support you in the technical details of how to connect your SUT in order to work properly. The appropriate protocol support for example (e.g. MQTT, HTTP, REST, …) should be available in the tool as well as the possibility to develop and implement tests. Ideally, the protocol details are handled implicitly by the tool so you can concentrate on the application level, e.g. if you want to stimulate a temperature sensor you may just want to set the temperature from your test script rather than dealing with the details in which topic of a MQTT message and with which QoS level this value is sent and how to set up the connection to the broker.
- Support you in the way how tests should be executed, either on an explicit trigger or within a CI (Continuous Integration)/ CD (Continous Deployment) workflow
