Book Review: How We Test Software At Microsoft

I really enjoyed this book. I don’t know if I learned a lot of new stuff but HWTSAM did remind me of a lot, and encouraged me to believe in a rosier future for all software testers.

Book Authors: Alan Page, Ken Johnston, Bj Rollison

HWTSAM makes it clear that because of Microsoft’s size many styles of testing go on in Microsoft and this book presents some stories about Microsoft and some of their techniques. No company I have worked for has faced, has not even close to, the same testing problems as Microsoft. Microsoft builds and tests for the long term. No company I have worked in has built for the long term - if a product takes a few years to write and then lives on in support for 10 years, and you have a company culture that acknowledges that from the outset, then the development process changes wildly - and you get the Microsoft approach.

( amazon.co.uk | amazon.com )

With a ratio of testers:developers of 1:1, those testers keep busy by testing the product as it currently stands and building a test approach for the long term - and that involves automation, a lot of automation.

Rather than write ‘one test book to rule them all’ HWTSAM tells some stories, outlines some important techniques and approaches, and recommends the mighty Binder, Testing Object-Oriented Systems, specifically chapter 3 “Testing: a Brief Introduction” (which i now have to go off and re-read).

The first few chapters describe Microsoft as a promised land where software developers and software testers live in harmony with each other, where everyone enjoys what they do with a high discretionary effort quotient. Joshing aside, the first 2 chapters present Microsoft as a great company to work for, one that really values the testing staff and reads as the best recruitment literature for any company I’ve ever read.

“Tester DNA has to include a natural ability to do systems level thinking, skills in problem decomposition, a passion for quality and a love of finding out how something works and then how to break it.”…“now that is what makes up a tester that makes them different from a developer” (page 25)

I might quibble with the ’natural ability’ part of the above but I really like the description as an attempt to say what makes a tester different from a developer.

Part 2 contains the Test Technique stuff. I like the unstated ‘obviously testers will have enough technical ability to do all this’ attitude that underpins this section. HWTSAM covers three big ‘functional’ techniques in depth: Equivalence Class partitioning, Boundary Value Analysis, and Combinatorial Analysis - with enough side stories and details to make the coverage of the technique interesting even if you have familiarity with them. And although not yet in the ’tools’ section - we learn about PICT ’the’ tool for combinatorial testing.

The chapter on structural testing treats the technique as a pure ‘white box’ technique. I wouldn’t limit structural techniques in this way since I use structural techniques on models I’ve generated without using the source code. Although hopefully this chapter helps dispels the notion that testers looking at source code is a baaaad act. Techniques covered: block testing, decision testing, condition testing, and basis path testing.

I really like the various ‘asides’. As an example, one from chapter 6 “In the five-year study at Microsoft…” I love that, mentioned in passing, not drawing extreme attention to it, as though all companies do internal five year studies. This book has many examples and stories drawn from long studies in an effort to improve. Not your typical ‘count this to improve’ metric evangelising tales. Microsoft seem to have a deep seated culture of improvement and practical “so how will we know?” investigation - I think I found much of my value in these stories, and I have not properly assimilated it yet.

But honestly, the Model Based Testing chapter, far too short - but it still contained some good tips. The full story of Model Based Testing at Microsoft still sits out there; elusive, mythical, awaiting a telling. And I want to hear it. SpecExplorer gets a mention but it seems like the ‘good’ version will live inside Visual Studio rather than sit outside for everyone to use.

I haven’t covered the contents in detail, when you read it you can learn all about:

Tools, automation, bug reporting, non-functional testing, built in customer feedback systems, services and the future - for yourself.

I expected more of a techniques book, instead I got a much more readable and enjoyable description of some of the testing life inside Microsoft. Plenty of stories to learn from and a description of a testing culture to aspire to.

Related Reading

• HWTSAM @ amazon.co.uk
• HWTSAM @ amazon.com
• Alan Page blog
• Bj Rollison blog
• Pairwise Testing
• Microsoft Tester Center (what no single rss feed!)
• Microsoft Tester Center Downloads
• PICT
• NModel
• UI Accessibility Checker
• MsaaVerify
• Testing Applications with AppVerifier
• Fiddler
• Robert V. Binder, Testing Object-Oriented Systems book review
• Acceptance Test Engineering guide
• Microsoft patterns and practices developer Center