Test Lean & Ship Healthy: A Handbook on Delivering High Quality Software in the DevOps World

Shipping code fast is key to delivering a successful software product. First-mover advantage is critical in many domains, and with the easy availability of cloud infrastructure, open source components, and CI/CD practices and tools, it's never been more possible to go fast.

However, increasing speed at the cost of everything else ("move fast and break things") is untenable beyond a certain point. When shortcuts are taken, tech debt accumulates, slowing development over time. Furthermore, the occasional bug or incident must be tolerable -- a non-starter in domains where availability is paramount, e.g. cloud infrastructure, or application security products.

This seems at first glance like a fundamental tradeoff, but doesn't have to be: if processes and tools evolve to support rapid development, we can maintain quality while shipping on time. The key is being extremely deliberate with our testing efforts and continuously optimizing them for efficiency.

In this book, we introduce four Test Lean principles that can enable you to deliver high quality software:

1. Complete automation over manual testing
   • Deployments and infrastructure should be fully automated and transparent
   • Any engineer on the team should be able to deploy
   • Automated quality gates (static analysis, security scans, passing tests) should be kept relevant
   • Break builds to uphold standards
   • Constant reevaluation of pipeline speed and quality gate relevance
   • No manual testing
   • No manual gatekeeping (except for code review)
2. Early error detection over end-to-end testing
   • Quality cannot be "tested in"
   • Testing must be interleaved with development
   • An ensemble of testing techniques is more effective and efficient
   • All developers are involved in writing and maintaining tests
3. Monitoring and recovery over exhaustive testing
   • Testing only increases MTBF, which cannot increase availability beyond a certain point
   • Reducing MTTR via monitoring is required for high availability
   • Exhaustive testing isn't a cost efficient way to increase quality
4. Increasing confidence in delivery over bug finding
   • No amount of testing can show that a system is free of bugs. However, the purpose of testing in a commercial software environment is not to produce flawless software, but to increase confidence in delivery: to give us assurance that our software can not only evolve correctly, but work correctly in the hands of real users.
   • To that end, we employ methods and tools to build bug free software, but are willing to compromise rigor when that would impede delivery.