In the book How Google Tests Software – Help me test like Google, the authors describe how bugs resulting from incomplete testing are one of the biggest barriers to Google’s continuous improvement and success. After analyzing their testing environment, they found 3 key issues:
- For complete testing, developers must write 2-3 lines of “Test Code” for every line of code being tested
- This “Test Code” requires the same amount of maintenance as the application code and often has more bugs than the application code
- Most developers do not perform complete testing
Google’s test strategy
Google’s solution was to create a well-defined and team-focused approach to testing that supports continuous integration. The “hyper incremental” builds of the application serve as the norm and ensure that the developers are responsible for quality.
Use test automation
You may ask the question “Can my organization adopt this process?” With a massive amount of resources, it may be possible. However, this issue can be an obstacle for many development teams. But you can build something similar by utilizing test automation.
Comparison of testing processes at Google and the a corresponding Test Automation tool-set.
| Google Software Testing | Corresponding Test Automation |
| Software Engineers (SWE): Responsible for coding and test case development. Utilize the “code a little, test a little” method using an Agile Test-Driven Development approach. | Automatically Generated Tests: Provides built-in support for “Test-Driven Development” and allows you to build test cases as soon as the header files are created. |
| Software Engineer in Test (SET): Assist the SWE with the unit testing. The SET’s are responsible for building test harnesses and frameworks including stubs, mocks, and fakes for every version of the application. The SWE’s use the harnesses to create the tests necessary to fully test the new and modified code. | Test Harness Generation: Replaces the manual effort involved with creating the framework, mocks, stubs, and fakes. Additionally, the tool should automatically update the test artifacts as the code changes. |
| Unit Test Dashboard Google has created a dashboard to enable them to run every test for every project every day. | Test Manager/Report Generating Tool: In addition to the basic functionality of test control and reporting, it also allows the same tests to be run with different configurations of the source code, or with different compilers. |
| Code Coverage Analysis: Google analyzes Code Coverage for all levels of testing. | Code Coverage Metrics Report: Capture Statement, Branch, and MC/DC code coverage for any unit, integration, or system level test. It should also allow combining of coverage across all testing into a single metric. |
| Dependency Analysis: Google has built a dependency analysis of each project so each developer knows exactly what code and test cases are affected by any source change. This provides immediate feedback on how each source change affects the entire system. | Change-Based Testing Filter: Support for Change-Based Testing, which performs dependency analysis for each source change and computes the sub-set of tests that are directly or indirectly affected by that change. A single click should run only those affected tests, reducing incremental test time from days to minutes. |
Create a continuous integration environment
Developing an all-encompassing software testing environment may seem expensive and daunting at first. Ad hoc testing methods are not beneficial to applications in the long term. At least if they are not augmented to additional testing that is performed.
You should not ignore the benefits of creating a continuous integration environment. You can implement and scale it easily when using off-the-shelf test automation solutions.
Increase software quality
The increase in overall software quality as a result of improved testing enables developers to tackle more features and issues (saving time and cost). It also results in a better end product. The better the product, the better the consumer perception, and the better your business will perform long term.
2 thoughts on “How to test like Google using test automation”
My company is replacing VectorCAST with Googletest.
The claim is VectorCAST is slow, its learning curve is deep, and time consuming to maintain.
Googletest can automate the complete functionalities of a complex application.
Wonder if there is any trade off between VectorCAST and Googletest.
Thanks,
Dat
Thank you for your question. As the blog post highlights, organizations that choose to adopt open source technology and that want to have a high level of automation to go with it, are required to invest a huge amount of time and resources to make it all work. Google invested in code coverage analysis, homegrown dashboards, scripts that perform dependency analysis, and much more. Most organizations don’t have an infinite number of resources to deploy to developing homegrown tools, and so VectorCAST has an advantage as it comes with all of this automation built in. The power of VectorCAST automation does come with some trade-offs around ease of environment set-up. However, set-up is typically just done once so shouldn’t be a barrier to deployment. Other factors that impact ease of unit testing, regardless of the tool being used, include language (C vs C+), compiler, embedded target if applicable, code design, code complexity, and use of variants.
Finally, if you are in a safety-critical industry, using a certified tool for safety-related development, like VectorCAST, is either required by your industry certification authority, or removes a risk for your organization.