Asking questions like “Do we need testing?” are important. But there are other questions we should probably ask first.
I was trimming out my email and found a pitch for an article that I had forgotten about. The article was never commissioned and I’ve used parts of this in conference talks, but I don’t think I’ve used these notes for a blog post till now…
Nothing is True
“rien n’est vrai, tout est permis” The Master of Assassins, Betty Bouthoul, 1936
The above quote could be translated to “Nothing is True, everything is permitted”. I’ve seen the phrase credited to both ‘The Master of the Assassins’ by Betty Bouthoul (written in 1936) and Alamut by Vladimir Bartol (published in 1938).
Consider the following questions (all of which I’ve seen mentioned online):
- Why do we need to test?
- Why do we need testers?
- Why do we still need to test in Agile?
Are these questions which plague Software Testing Professionals?
If so, I encourage you to step back before answering.
Those are not the first questions we ask.
They are perfectly valid questions to ask. They’re just not the first ones to ask.
Questions as distractions
If you are being asked those questions then perhaps you are being offered a smoke screen by someone who is concerned about the solidity of their own belief in their own discipline or process and tries to push all doubt onto the Testing effort as a distraction.
Consider the phrase “Nothing is True, everything is permitted”.
The supposed last words of Hassan-i Sabbah, head of the Hashshashin order of Assassins.
I struggle to remember if I first read this phrase in a book by Colin Wilson, or Robert Anton Wilson, or William S. Burroughs. I can’t vouch for its origin but most recently it has been popularised as the motto of the Assassins in the Assassin’s Creed series of games.
“Nothing is True” would falsify most absolute statements e.g “Testing is necessary”, “We must test”, “You need Software Testers” etc.
“Everything is permitted” would allow the development of software without testing, or without testers.
But ’testing’ is such a low place to start when applying the motto. This is a meta-motto designed to apply to the meaning of life and change behaviour in much more radical ways.
“Nothing is True”
Statements like “Nothing is True” and “All models are wrong” take us very close to Godel’s incompleteness theorem where we can make statements that cannot be proven.
“Nothing is True” falsifies itself.
Perhaps it means “Nothing”, as a thing, “is true”? Language is ambiguous so we can read it as we choose. I recommend reading it in the way that offers you the most options and flexibility.
“All models are wrong” is a statement, out of context, and is itself a model. Is it wrong?
These statements aren’t particularly useful at offering guidance on their own. I tend to take them as reminders that absolutes contain the roots of their own destruction and that we will find more value looking at the details of our current situation.
Defining our situation as appropriate for our context. If we approach our situation with a fresh mind where “Everything is permitted” then we can create something that avoids the dogma that we’ve heard of, that everyone knows is true, or is the “done thing”.
By approaching our situation with a fresh mind we can create something tailored to our specific needs and constraints.
Do we need programming?
What if we applied the motto to programming?
- Do we need to program at all?
Perhaps we can deliver a software system without programming.
Perhaps we can take an off the shelf system and by changing some configuration we could have a customised system.
Will GitHub Copilot reduce reliance on programming? Perhaps Copilot will force reliance on coded unit tests to check that the AI generated code copy and pasted into the system meets the end requirement.
We might still want to test in those scenarios.
We have found situations where we are not programming, but we require testing. We may indeed at times need testing after all.
Do we need to develop Software?
What if we applied the motto to Software Development?
“Software Development” as a concept encompassing both programming and testing.
- Do we need to develop software at all?
Perhaps we could simply buy a product off the shelf and use it? After all that is what most of us do when we live our lives. I didn’t configure, or program, or test, my accounting software. I simply used it. Everything is permitted.
But even Software Development is too low a level.
Do I need Software?
Do we need Software?
For accounting, with the need to file my accounts digitally. I have some regulatory compliance that I must adhere to.
Some form of software seems required.
But if “everything is permitted” I could do my bookkeeping in paper ledgers and using a simple text editor create the files required for regulatory compliance, and use simpler software for uploading.
Possibly doable, but a riskier proposition, and it sounds like a time-consuming one.
I can probably save money, time, and reduce risk by using accounting software.
Having thought ‘Do I need Software?’ I can now justify the accounting software.
I have a business need.
I can then explore the different options available for meeting that need and consider the risks involved with the different options.
What do I need?
Perhaps I need neither Software Development, Programming or Testing.
Work back through a chain of reasoning.
- “Do we need to do testing at all?”
- Well if we don’t we won’t know if the programming has been done correctly.
- “Do we need to do programming at all?”
- Well if we don’t we can’t build the new Software.
- “Do we need new Software?”
- If we don’t we can’t meet business requirement X.
- “Do we need to meet business requirement X?”
- Absolutely.
Something is viewed as true, in context.
When we reach a perceived truth we can then explore what it means.
- “How would we know if we met business requirement X?”
This provides us with a measurable, or verifiable, or checkable, or testable situation and we can evaluate options against it.
This is the point at which we start asking questions to figure out the most appropriate methodologies and processes we will use to implement our options.
Very often we don’t go this high in our reasoning. We default to an assumption that we will create software that we will develop ourselves.
Verify Objectives
- “As a user I want to be able to download my bank statement as a csv file”
Do projects which have stated objectives such as the above actually measure, or verify, or check that the user does actually download their bank statement as csv.
Not just that they can, which is what the programming and testing would help with.
But that they do, which is the value the function is supposed to add.
Do our features go unused? Do we know?
GDPR and privacy concerns often make this type of knowing harder to determine which is a problem to product teams world wide, but vastly reduced with SAAS systems where API calls could be measured on the server.
If the user doesn’t use the feature then it doesn’t matter if we programmed it, and by extension doesn’t matter if we tested it. That user story wasn’t “true”, anything at that point was permitted: it could have been a link to a blank file, it wouldn’t matter, it wasn’t used.
Low Level Questions
We often spend a lot of time at lower levels of questioning.
As ’the business’ we might be better off considering higher levels of questioning and identifying how we can check that our high level belief was true.
Most of us don’t work as ’the business’ and so we apply this thinking to our discipline.
Do the things we do, need to be done?
And again, we don’t start with “Why?” we consider “What if…?”
- What if we didn’t test?
- What if we didn’t automate?
- What if we didn’t report our coverage?
- What if we didn’t raise defects?
Elaborate on the above “what ifs”:
- What Risks would we be exposed to?
- How could we prevent those risks from manifesting?
- How could we detect them if they did?
- What impact on time would the alternatives have?
- What cost implications would the alternatives involve?
- etc.
At the very least you will know that alternatives to your approach exist.
You might choose to use these chains of reasoning when asked “Why?”.
“Simply considering the question, ‘do we really need to do any research at all?’ will have a positive effect on any research that is conducted, because it at least focuses attention on the key issues that need to be addressed and begins the process of figuring out the most appropriate methodologies.” Truth Lies & Advertising, by Jon Steel pg 66
I post content on a daily basis to Patreon. For only $1 a month you can receive my weekday blog posts, adfree access to my YouTube videos, and short online training courses. patreon.com/eviltester
