Hi! In my blog post, I will introduce the six keys to reaching a higher level of quality in software, whether you are doing certified safety-critical software or another form of mission-critical or business-critical software.

Why is software so different from hardware?

Our interest of course is software, because it is something relatively new in human history. And it’s one of the most complex intellectual activities: any app that you have on your smart phone is more complex than Dante’s Divina Commedia, Michelangelo’s Cappella Sistina or any other historical human intellectual masterpiece in terms of person-years. This kind of complexity cannot be handled with standard techniques and approaches, like traditional engineering or civil engineering is applied to buildings and bridges, for example. Software also cannot be approached with statistical and quantitative analysis.

Software can only be handled by methodological approaches, where the development process is the critical component. So you should not concentrate on statistics, on tools, on these kinds of things, but you should be more aware of the process: having a robust process is the main objective, and then the tools will help you to reach this objective. You should not start from the end, from language, from tools, from technology: vice-versa, you should start from the process.

Key 1: Follow a good software development process

There are mainly two kinds of software. The first is the business- or mission-critical software, which is the regular software you would write where no life is in danger, but the software is critical for the success of the business.

Normally, there is no mandatory certification in this case, so there are some quality standards that you can decide to follow (for example the ISO-900x standard), but this is for some minor and organizational aspects. Another example is A-SPICE, the Automotive SPICE which is a pretty good process, or M.E.D.S. (Method for Efficient Development of Software).

Then there is safety-critical software, where human life could be in danger with a software malfunction. Here you have a mandatory certification, for example automotive ISO-26262, avionics DO-178C, medical IEC-62304, industrial IEC-61508 and railway EN-50128.

Learn more about implementing a professional software development process: The safe & secure software factory

Key 2: Write “perfect requirements” for higher software quality

What is a “Perfect Requirement”?

Safety-critical software needs are different than business-critical software, so you might have a different degree of rigor with different types of projects. SMEO (Small-Medium Enterprises) are also different from big companies, and working in small local teams is not like working on a big international project. Nevertheless, there are some common factors and properties that are mandatory for everybody, in any kind of company or project, because the requirements themselves, in various forms, are mandatory for everybody.

Requirements are always mandatory, but the rigor, the completeness, the rules could change according to the project’s criticality.

You should write very good requirements, even more “Perfect Requirements”. What does it mean? It means that you must follow specific Guidelines during the process of writing requirements, to help you to avoid common errors like using ambiguous words, taking assumptions, leaving gaps and so on. And also, you must adopt a good Checklist during the Requirement Review phase, that allows you to remove any inconsistencies, ambiguities, gaps, and concepts that are expressed in a complex way or with too many assumptions, in order to have requirements that are formal, clear and especially Testable. This is the key to the next point, Requirement-Based Testing. You can write your own Guidelines and Checklists thanks to your experience, or you can follow some Requirement Standards and finally you could eventually adopt commercially available ones.

Missing this important aspect and using another form of specification could cause problems later in the project. And in order to reach good software quality, you should make the effort to invest your time in requirement analysis, and to write a very good requirement specification. Following Software Safety-Critical Standards or Business-Critical Standards, is a very good starting point in order to have a perfect set of requirements, for later testing phases.

To go deeper into requirements: https://www.coderskitchen.com/requirements-in-agile-development-process/

Key 3: Adopt a requirements-based testing approach

Once you have good requirements, then you unlock the third step: adopt a good requirement-based testing (RBT) approach.

Let’s write for example the definition of the SIN()  function:

We could describe it as the projection of a vector of radius 1 and angle X on the Y axis. Then it is now much much easier to understand it, because you could test the more significant values, like 0, PI, PI/2 and so on. You could also add some robustness definition of the requirement, for example you could tell that the angle is between minus PI and plus PI plus/minus 2PI. And you can tell that the SIN of plus or minus infinity is NAN (Not A Number) and the SIN of NAN is again NAN.

So we have now a complete requirement.

Definition (requirement) of SIN(x) function:

The projection of a vector of radius 1 and angle X, on the Y axis

For Robustness:

If you want to test this function it’s very easy because you can use some significant angles which you may remember from school. For instance, the SIN of 0 is 0, the SIN of PI/2 is 1, the SIN of PI/ 4 is a square root of 2 divided by 2, and so on.

This is requirement-based Unit Testing, exactly what you need to do.

Do you really care about the implementation, about the code?

NO!

This is the way to do real black-box testing based on the requirements, which is need to achieve a requirements-based testing approach. And of course, you can be supported by tools for traceability between your requirements and test cases.

Key 4: Perform software testing in all phases

Starting from the implementation phase, you can do software unit testing, static analysis, white-box testing on host or on target.

Then you can do software integration testing, where you start to integrate some of your software components and test how they work together, eventually calculating code coverage. You can do it on your host PC with an emulator/simulator. Code coverage will indicate how thoroughly you have tested these software components, and where additional tests need to be added.

Next, you can do hardware integration testing, or system validation and generate code coverage on the real target or ECU. Adding tests to achieve 100% code coverage will ensure a thorough test suite, from a code perspective.

And you can start to do change-based testing to reduce the number of test cases you run at every cycle, as we will see in our final 6th key.

You can do HIL (Hardware-In-the-Loop) testing, running your tests in a real physical I/O loop, and collecting code coverage.

This approach supports software testing throughout the full software development lifecycle, from unit testing to system validation.

More information: How to test like Google using test automation

Key 5: Perform continuous integration with Jenkins

The objective here is to achieve faster turnaround cycles while also finding as many errors as soon as possible. One way to accomplish this is to do parallel and virtualized headless test execution, in order to use any possible slot of time (including nights, weekends, …) and computational resources.

You can use Jenkins as a Master and run the test cases on physical or virtual nodes. You can test in multiple configurations, supporting massive parallel testing with the continuous integration servers.

More information: Continuous integration: Building the ultimate CI pipeline

Key 6: Perform change-based testing (CBT)

Using a normal regression testing approach, you will need to re-run all the test cases (unit, integration, system test cases) and it takes a lot of time: it could take days or weeks, because some other tests are very long and the test results are very difficult to analyze.

What about if we collect the coverage and we keep the source code traceability to test cases?

The concept here is to identify the only test cases to be executed because of code changes, because of a previous test failure, or because of changes in the requirements.

The big advantage is that with CBT, thanks to the traceability between the source code and the test cases, an algorithm determines which test cases have been affected and will rerun just a very small subset of them, in a fraction of the time.

It could be 100 or even 1000 times faster than a full regression test run, because only the test cases that have been changed will be rerun. Your advantages: reduce the testing time, find errors faster and improve test efficiency.

More details in our video: Six keys to higher software quality

00:27 Software testing: Manual, human, automated?
04:37 Product quality assurance
06:22 Software quality: Process assurance
09:39 Definitions: Manual testing vs. automated testing
12:17 Key 1: Follow a good SW development process
13:35 Key 2: Write perfect requirements
15:07 Key 3: Adopt a requirements-based testing approach
17:38 Key 4: Perform software testing on all phases
19:22 Key 5: Perform CI with Jenkins
20:44 Key 6: Perform change-based testing

Get started with your projects

• Product information: Static Code Analysis for C and C++
• Product information: Automated testing with VectorCAST

I hope you enjoyed my blog post, please send me an email with your questions.