Quality Assurance (“QA”) is arguably the most critical aspect of safety-critical software. However, QA is rarely given the attention or credit befitting its crucial role. Consider the following statements and assess whether they are true or false; answers and explanations follow:
- Safety-critical QA’s most important role is assessing final product quality.
- Safety-critical QA personnel perform technical reviews.
- Safety-critical QA plays a key role in review of requirements, design, and code, and also conducts tests.
- Every safety-critical development project requires at least three different persons, and the most crucial of those for certification is the independent QA person.
If the above questions were truly easy, congratulate yourself on your genuine QA knowledge. If they simply seemed easy, then the information that follows is for you. In fact, answering the above without understanding the overall quality assurance role in safety-critical development is like understanding Fourier transforms without first understanding The Calculus: impossible for mere mortals …
In safety-critical software development, quality assurance has three primary responsibilities:
- Ensuring the project-specific engineering plans and standards comply with the applicable safety-critical standards
- Assessing, and then ensuring that the independent engineering organization has followed those plans, from project inception through completed delivery.
- Build and retain evidence of the above via recorded audits with resolution of all issues, defects, and process improvements encountered during.
In non-safety-critical development activities “Quality Assurance” often implies a more adjunct, and more passive, measurement role. Wikipedia, for example, aptly states QA “is the systematic measurement, comparison with a standard, monitoring of processes and an associated feedback loop that confers error prevention.” In most industries, QA reports to engineering, assists with documentation technical reviews, and executes various system/software tests near the end of the development lifecycle; however none of these are the roles of QA within safety-critical. Why doesn’t safety-critical QA perform these traditional industrial and commercial consumer industry QA roles? The weaknesses of such a traditional, yet common, QA interpretation for other industries if applied to safety-critical:
- Who ensures project plans and standards are correct and compliant to the safety-critical standards?
- Who ensures applicable processes are deterministic, repeatable, clearly defined, and compliant to those safety-critical standards?
- Who ensures the engineering process feedback loop has been defined correctly and then followed?
- Who is responsible for ensuring proof exists that errors were appropriately identified, dispositioned, and resolved?
In safety-critical development, the answer to the above is simple: “Quality Assurance.” QA is based upon what AFuzion Inc. terms the “QA Pyramid” as depicted below:
Safety-critical standards and guidelines are somewhat “flexible” regarding the manner in which QA processes are defined, scheduled, and performed. However, while flexible, safety-critical requires defined and measured processes be methodically applied to ensure product quality. Not merely to improve product quality: improving a weak product until it became a mediocre consumer product is clearly insufficient within safety-critical. Clearly, the quality of safety-critical development depends upon the quality of components comprising development.
Safety-critical QA: big-picture and small processes
Which components of the safety-critical development process affect quality? All of them. Are all those components equally important? No and yes: “no” they do not all have the same potential impact upon quality, but “yes” since each component can affect quality then each component has equivalent status as a potential failure point within development. Safety-critical quality assurance then is tasked with applying the industry guidelines while reviewing all aspects of the safety-critical development process to determine how the inputs and outputs to that process should be independently assessed. Therefore, QA must take both a big-picture view of that entire ecosystem, while also addressing the smallest engineering processes which can affect product quality. It starts with quality assurance planning …
Upon first glance, the above list of safety-critical QA objectives seems easy; almost obvious. For example, avionics software is obviously “safety-critical” and governed by the standard “DO-178C”; quality assurance is considered the most important role in development, truly. More important even than the actual software development and testing. And, in other, non-safety-critical development environments, “quality assurance” appears to embody similar objectives: assess product implementation to measure and improve quality. As a result, virtually every consumer electronics product manufactured today has some form of basic quality assurance performed upon it.
As an example of safety-critical QA for aviation, the QA plan addresses the key topics depicted in the following figure:
The quality assurance role
Many companies, and certainly traditional companies with a track record of reliable development, are adept at defining a quality system and following it. However, DO-standards place the emphasis, or burden of proof, upon the quality assurance department. The reason for this is both technical and politically pragmatic: where multiple people and departments bear equal responsibility for proof of quality, finger-pointing and a lack of true accountability may ensue. Where instead one department is responsible, the authority is clearly designated: independent quality assurance. Whereas QA in non-safety-critical industries participates in technical reviews, executes tests, and reports to engineering management, safety-critical QA is decidedly different; the following figure depicts the explicit differences between engineering and QA within safety-critical:
So, does QA bear sole responsibility for quality such that all the project engineering staff can relax and choose not to follow the plans and standards? Of course not; engineering must follow all applicable plans and standards as assessed by technical reviews and quality assurance audits. Does QA bear the responsibility to assess technical compliance? No; technical compliance is assessed via reviews, and reviews are done by engineering. QA does not perform technical reviews; instead, QA performs audits to assess engineering’s adherence to the defined process. This safety-critical QA process becomes easier with practice, or with training.
Wait, isn’t this really verbal nitpicking by thus differentiating between reviews and audits? Yes, QA “reviews” engineering process adherence by performing audits. But this is not called “review” since in DO-standards reviews are thorough and technical in nature and thus performed by engineering. Remember, engineering performs verification which includes reviews, tests, and analysis. Therefore, engineering is responsible for performing the reviews of engineering artifacts including requirements, design, code, and tests. QA performs audits to ensure engineering follows the review processes contained in the verification plan.
More information
- Free Whitepaper “Introduction to DO-178C” from AFuzion
- Safety-critical Quality Assurance training
