The software test automation tools market is full of solutions that all claim to do the same thing: automate software testing. Wikipedia lists over 100 test framework tools for C/C++ alone! Additionally, when you look at product literature or demos, many of these test tools look alike.
This blog post contains information for engineers to consider when you have to evaluate embedded software test automation tools, specifically dynamic test automation tools.
Compiler integration
The point of the compiler integration is two-fold. One point is to allow the test harness components to be compiled and linked automatically without the user having to figure out the compiler options needed. The other point is to allow the test tool to honor any language extensions that are unique to the compiler being used.
Key Points
- Does the tool automatically compile and link the test harness?
- Does the tool honor and implement compiler-specific language extensions?
Support for testing on an embedded target
We will use the term “toolchain” to refer to the total cross-development environment, including the cross-compiler, debug interface (emulator), target board, and Real-Time Operating System (RTOS). It’s important to consider if the potential tools have robust target integrations for your toolchain and to understand what in the tools needs to change if you migrate to a different toolchain.
Ideally, the tool that you select will allow for “push button” test execution. All of the complexity of downloading the target and capturing the test results back to the host is abstracted into the “test execution” feature. That way, no special user actions are required.
When testing hardware isn’t available, a key feature is the ability to start testing in a native environment and later transition to the actual hardware. Ideally, the tool artifacts are hardware independent.
Key points
- Does the vendor support my toolchain?
- Can I build tests on a host system and later use them for target testing?
Integration with other tools
All tools need to operate in a multi-vendor environment. The most useful integrations for test tools are static analysis, configuration management, and requirements management tools.
Key Points
- Which tools does your candidate tool integrate with out of the box?
- Can the end user add integrations?
Tool qualification
When operating in a regulated environment such as commercial aviation or Class III medical devices, you should “qualify” the development tools used to build and test your application.
The qualification involves documenting what the tool is supposed to do (the tool operational requirements) and tests that prove that the tool operates in accordance with those requirements. Ideally, a vendor will have these materials off-the-shelf and a history of customers that have used the qualification data for your industry.
Key Points
- Does the tool vendor offer qualification materials that are produced for your exact target environment and toolchain?
- What projects have successfully used these materials?
Conclusion
Hopefully, this blog post provides useful information to navigate the offerings of test tool vendors. The relative importance of each of the items raised will be different for different projects.
Final suggestions are:
- Evaluate candidate tools using code that represents the complexity of the code in your application
- Evaluate the candidate tools with the same toolchain you are using for your project
- Talk to long-term customers of the vendor and ask them some of the questions raised in this blog post
- Ask about the tool’s technical support team; submit some questions directly to the support team to see how they respond
Remember that many embedded software test tools can support the items mentioned above. Be sure to evaluate how automated, easy to use, and complete the support is.
Don’t forget to download the whitepaper below. It contains a comprehensive list of criteria to consider the next time you evaluate embedded software test tools.
Further reading
- Whitepaper download: How to evaluate embedded software test tools
