How to maintain software quality?

Andreas Horn

In my previous blog post “What is software quality and how to measure it?” I have explained what quality is and how we can measure it using indicators. At the end of the article, we concluded that the occasional measurement of quality, e.g. only at milestones, is not enough. Quality is integral, so its measurement should be as well. It should be a natural part of the development process. And to minimize overhead (and to overcome laziness) it should be automated where possible.
At first, it sounds obvious that to maintain software quality we have to measure it regularly. And also, automation is a “no-brainer”. When we take a look at a generic toolchain and the work products of a project, we will better understand the underlying problem.

Measures aggregated into Code Quality Indicator in a generic toolchain

Bringing measures together

What we develop consists of many work products, that are edited and managed in a multitude of different tools, connected into a toolchain. The quality we want to measure is in our software. But our work products are distributed throughout the toolchain. And so are also the measures. To automate the measurement, all these measures have to be brought together.
In every modern development process, there is one location, where many tools and work products come together. And this is, where integration, testing and deployment are continuously performed: The CI/CD/CT Pipeline. That means that also all the measures can be gathered here. And integrating the measurement into this pipeline turns our software quality monitoring into something “continuous”: continuous quality monitoring (or CQM for short).

Let us take the source code as an example and see how the integration of quality monitoring into the pipeline can help to derive the code quality continuously.

Using static analysis tools, we can collect violations of MISRA Guidelines and HIS-metrics.
The source code management system can provide information, e.g. on code stability
Unit tests can give us test results and coverage data.
With the code linked to requirements, we can also integrate information from the requirements management tools, like stability, status, test coverage, etc.
The software quality monitoring can then aggregate the code quality indicator (status) using the data generated by the pipeline.

Software quality monitoring is not limited to source code, there are also other work products and tools that contribute to our quality. So, as a CI server can show us the health of our build process over time, for all jobs and pipelines running on one server, quality monitoring should be able to do the same for whole projects. And Continuous Quality Monitoring should show us not only the indicators, but it should also help the whole team, from developer to manager, to find the current pain points. I.e., if an indicator is not as expected, we can dive deeper, looking at the measures building up the indicator and down to the elements, to which the measure functions are applied. In short, CQM enables us to maintain software quality.

How to adapt software quality?

Monitoring can help us to find out where set targets have not been met and by that enables maintaining the software quality. But that is still not enough to reach our quality goals. With CQM we can measure the current quality continuously. But just setting 100% as a target for the overall quality will neither be helpful nor feasible. Hence, we have to adapt our quality expectations. If we always just ask: “Have we reached the goal?”, we will never know how far away we currently are (if you have kids you know this situation: you just left home with your kids in the car, and 5 minutes after departure they start asking „Are we there yet?“). We have to ask for the quality that is important for a project in its current state. If we ask the right questions we will not only know where we are, but also what to do next.

Step by step: Software quality gates …

This includes the planning of software quality over the development cycle using different quality gates, and the adaptation of what makes quality. Remember: Degree of fulfillment, set of inherent characteristics, requirements. But before we adapt the quality itself, let us first take a look at quality gates:
Quality gates are automated checks integrated into the development pipeline. They can focus on different aspects of our development, e.g. process, product, reliability, and can be activated selectively. A violation of a quality gate can stop the delivery pipeline. Hence, quality gates can be the tool to check and maintain the achieved software quality by enforcing our expectations, especially on the master release branch.
But the gates can be completely over-the-top for a single developer, who is trying to fix an issue or implement a new feature. Or insufficient for a formal release.

… integrated into the development process

Different quality expectations for different development phases

Development usually takes place in feature branches that are merged into the master branch (or trunk) once the development of the corresponding feature is completed. During this process it is sufficient to be more focused on correctness, omitting the completeness aspect, following the approach “Test less, fail faster.” Of course, as soon as merged into the master branch the full-scale quality monitoring is applied, with full (or incremental) test runs, automated checks, and quality gates.
For a formal release, additional tasks and checks like report creation, reviews, and alignment with management or customers may be required. What can be automated during this formal release process should be part of the quality gates on the master branch, what comes on top of this can be left out of the continuous monitoring as it is only needed once per formal release.

Plan which software quality to maintain …

When these releases are to take place is determined in advance in the project plan. And just like we plan the project, we also plan its quality along with it. Over the project duration, we should avoid degradation of quality and increasingly improve to reach the target (assuming that we want to deliver good quality to the customer). The rest is up to us. Do we first want to focus on software quality improvement and then maintain a high level until the final delivery? Or start low, do not get worse, and then shift the focus on quality improvement in the end?
And there are also different types of releases. Our first release will not be the best software quality, in most cases we will start with engineering drops, to give the customer an impression of the current progress, and then raise the quality expectations (e.g. alpha, beta) until we reach gold status in the end.

… and adapt expectations

There are still more possibilities to adapt: Until now, we have treated all measures with the same priority. It can be necessary to change this. Characteristics can not only influence each other, they can also be supporting or contradicting. A trade-off has to be made. So, based on expectations, characteristics may have different priorities.
One last possibility (at least in this article) for adaptation are what gives us in the end the degree of fulfillment, the target values. Depending on company strategy, product, experience, or customer requests target values can be adapted. But be warned: If target values deviate too much between projects this makes comparing projects extremely difficult.

Conclusion

Now we can put all the pieces from my last blog post (What is software quality and how to measure it?) and this one together. We have learned that achieving software quality means first of all meeting expectations. And this starts with the elicitation and identification of the expectations and needs of all stakeholders. Without these, we are not able to measure the degree of fulfillment. To help us map what we have measured to the characteristics that we have derived from the requirements, we can use indicators, which aggregate measures or other indicators to enable focusing on the relevant pain points and facilitate quality indication. Software quality is in the product and thus, measuring quality must be an integral part of the development. If we achieve the integration of the measurement into our CI/CD/CT Pipeline, we also get our quality status continuously (we end up with Continuous Quality Monitoring), which enables us to efficiently maintain software quality.
This is a very powerful instrument, if we have planned our quality and adapted our expectations. Because software quality is never the same. It changes between milestones, projects, customers and over time.

Any questions? Contact me