Evil Tester

I have trimmed out a lot of my Software Testing books. And by trimmed, I mean, dumped or traded in on Amazon. I still have a few left, some because I see real value in them, and others because my hoarding instinct overrides my Zen de-cluttering zeal.

Here are 3 books I’d like to read. But they haven’t been written yet.

Apologies to the author’s and artists associated with the books I’ve based these new books on, but my desire for humorous pastiche overrode any concern that the original creative forces might feel insulted.  I picked on you because of your mythic stature in the Software Testing book world, because of the value that I took from you in my early years, and you met my needs.

So my first 3 are:

• The Art of Software Jesting by Ben Fold Wired
• The Complete Guide to Software Besting by Phil Betzels
• “I Object!”, Re-oriented Software Testing: A heretical Approach by Bell Diesel

The Art of Software Jesting

I think we take software testing too seriously. Or rather, we take ourselves too seriously. The danger being that Software Testing becomes a subject pumped up full of its own self importance, filled with pompous pontification.

e.g. "Exploratory testing must not be used as your main process. You must only use it after you have a stable system and have achieved coverage from your scripted tests."
blah de blah de blah, yeah yeah yeah

I don’t see enough evidence that we take ourselves seriously enough to mock ourselves.

Kings and Queens of olden days of yore took themselves pretty seriously, as did everyone else because their life depended on it. And yet they had a Court Jester to keep them balanced and reduce their hyper inflated sense of self importance.

Humour can help effect change, by laughing at yourself for beliefs with small amounts of evidence.

Lighten up.

Projects are a ridiculous place to shouting matches.

I appreciate many people on projects don't appear to have a sense of humour, and don't take it kindly when you try and inject one into the project. But as a weapon to disarm and defuse a situation, it helps.

It helps me not take my approach too seriously. It helps me work with people rather than roles.

Evil Tester was born from a requirement to have a sense of humour and expose the ridiculousness in many of the approaches and attitudes that I had adopted in the past, and other people on projects had adopted.

So if you are really serious about testing - take it seriously enough to laugh at it. And start with yourself, and your processes, don't start with other peoples.

Have you mocked your testing today?

Complete Guide To Software Besting

I think we all know that “Testing is not Besting the Software produced” but if you want a persona to adopt as you test then periodically adopt the “Bester”.

This thinking hat will let you approach testing in a different way than the other hats you wear.

You have to decide your own limits in your software testing approach, so make sure your limits allow you to exceed the the requirements for ‘goodness’, you want to be better than that.

“I Object!”, Re-oriented Software Testing: A Heretical Approach

I think that we have to think heretically. We have to pursue the path that we consider true, treating all dogma as valid grounds for testing and evaluating alternatives.

Testers need to think and act differently, otherwise why would other Software Professionals want you on their project?

We need to make decisions contextually and based on knowledge, not from dogma.

You need to take responsibility for your test approach – and if that choice requires that you fly in the face of fashion and the masses, then I hope you do it.

The more heretics we have, the more we will advance in unexpected ways.

Other Reading Material?

This is based on the Test Bash talk I gave in March 2012. You can read Marcus Gartner’s summary of it here

Perhaps I’d like you to read “Selenium Simplified”, but maybe I’d rather you just bought it… and a few copies for your friends.

But what about you? What testing books would you like to read, that haven’t been written yet?

What is Testing?

Do you care?

Why?

• Do you want to know because you want to know where the limits of your job start and end?
• Do you want to know because someone asked you that question and you think you need to give them an answer?
• Do you want to know because you really like definitions of abstract concepts?
• Do you want to know because you need to pass an exam and give someone the right answer?

You might have a whole bunch of other questions - most likely you do since I only provided four alternatives to the "What is Testing?" question.

Questions I would rather believe people ask themselves include:

• What do I do?
• Why do I do what I do?
• What do I need to do now?
• What risks can I think of?
• How can we find out if this software can ...?
• I wonder if the software can still do that when ...?

But enough questions. Let me provide a simple answer to the first question "What is Testing?"

I don't think that such a 'thing' as "Testing" exists. Conversationally  a normal person might say something equivalent using the word sequence:

"There is no such thing as testing"

And while that might not help you pass your exam, and fails as a definition, and 'they' probably don't want to hear that answer; it might help you figure out where your job starts and ends.

Regardless, the above answer provides comfort and guidance to me during times of trouble.

If I ever find myself in a mess and think "What is testing?" I can politely answer myself "There is no such thing as testing, figure out a better question to ask". Or "Stop hallucinating, concentrate on what you see happening now, figure out what to do next."

If I had an actual answer, and if I knew what a thing called testing looked like then I fear that I:

• might never look at the testing thing differently.
• might think that as a "tester" I couldn't do anything that didn't look like the testing thing.

Over the years I've come to believe that my job involves looking at things and processes and concepts differently.

I want the freedom to make stuff up, the freedom pull in resourceful concepts as required, and the freedom to do what it takes to achieve the identified needs of the projects I work on.

I don't find the question "What Is Testing?" helpful.

• I care what preconceptions other people might have about my role and their expectations of me and people like me ("Testers").
• I care about the processes we use to identify, mitigate and make manifest risk.
• I care about making what we do efficient and effective.
• I care about agreeing actions and who will take those actions now.

I care about a whole bunch of stuff. And I test, because I care.

I just don't care about questions like "What is Testing?".

Do you?

Why?

PS. I think I managed to avoid all mention of reification and e-prime. Did you notice?

First I used XPather, then it was FireFinder, and since neither of those seems particularly compatible with the most recent versions of Firefox…

I now use FirePath.

FirePath operates as a Firebug extension and provides a handy “Inspect in FirePath” context menu entry.

FirePath handles XPath, CSS and JQuery selectors.

I don’t have any spare tools extensions in my tool box, so if Firepath dies I’m not sure what I’ll do.

Does anyone have any suggestions to add in the comments?

I was working out the kinks in my high level software testing model, and, through a process of speed reading and stichomancy I found that I have re-created an early Buddhist doctrine.

In “The Story of Chinese Zen” by Nan Huai-Chin, I find listed the five Skandhas:

• form
• sensation
• conception
• activity
• consciousness

I was boiling my model down to:

• model
• observe
• intent
• manipulate
• reflect

I’ve re-ordered my list to  tie in more closely to the Skandhas.

Quite a useful coincidence.

Below I list a simple set of my correspondence ‘tween the lists.

I have ‘model’ instead of ‘form’ because our world comes to us from our perception of it, not from it itself. Perception allows us to experience bias and hallucination, and provides the scope for us to change how we perceive.

As testers ‘sensation’ comes to us through our awareness, with our observation. We have to learn how to expand our range of observations and utilise tools to help us observe. Acts of observation can help us turn noise into data and subsequently into information which we can act upon.

When we test with intent, we bring purpose into our testing. We know what we set out to do/explore/check/exploit/etc. We often move off the beaten path and open ourselves to surreptitious happenstance, but only if we observe that happening can we utilise it.

Manipulate – the favourite ‘bad’ word of the hypnotist, although even as a hypnotist I felt happy using it, and as a tester I do it frequently. Shaping the system through my action.

Reflect, if all we did was progress from our initial intent then we would not learn. We reflect, to learn, move on, ever better, and more deeply.

And so, dear reader. Did you create your model yet? If so, see where else you can find it!

Follow on Reading:

• “The story of Chinese Zen” on amazon.com and on amazon.co.uk 
• http://en.wikipedia.org/wiki/Nan_Huai-Chin
• http://buddhism.about.com/od/whatistheself/a/skandhasexplan.htm

Thanks to James Lyndsay for the recent chat about correspondences between my model and his.

The notion of personas never really worked for me. “Bob is 35, single and likes kittens...” Blah Blah Blah.

Clearly Bob has all the characteristics of a fictional closet psychopath.

And that works better for me. “Bob is a closet psychopath”. I can use that sentence to inform my testing. I can attempt to  test like a closet psychopath.

Other personas you might want to adopt:

• Sociopath
• Psychopath
• Paranoiac

Of course, those are just the most obvious examples of personas we could use in testing.

“Alan, you’ve clearly misunderstood the point of persona’s”

Well, persona’s have a rock solid history that the UX community don’t seem to promote. So its time to remind ourselves of what we’re really doing.

By looking at the true history of the persona we remind ourselves of techniques that help us build personas more effectively and quickly.

But I warn you now. Persona testing ain’t no walk in the park, princess.

And duly warned, we proceed to the history lesson…

Everyone knows the word Avatar now. Since I haven’t seen the Cameron movie I don’t immediately think of a blue alien animated thing.

I think of an Avatar as a personification. So a persona and Avatar merge and become one and the same.

Which neatly reminds  us that the Hindu mythos has an Avatar as an incarnated god.

And that Gods don’t incarnate in just one mythos.

Other mythos offer similar approaches. From the Vodoun, we can take inspiration from the Loa and from the techniques of the Houngan. We could attempt the deliberate summoning of a Loa and the subsequent possession to help us fully manifest a persona as part of our software testing process.

Imagine, in your next morning standup, if you adopt persona techniques in your testing, you could truthfully say:

“This morning I am going to eat raw chillis and cover myself in chilli rum to summon forth Baron Samedi. Much merriment will ensue as I perform some usability testing on the latest release.”

Clearly some of you reading may imagine that I mean this metaphorically. You may imagine my hinting that you should examine the Nanchons of Loa to find inspiration for your personas. Good for you.

The true testers among you will have already begun constructing your Vodoun altar.

Some of you may find your sensibilities more in keeping with a Western Magical tradition. If so, then fear not.

The western magical tradition has a rich set of rituals for the invocation of Gods. Looking up invocation in pretty much any book of ceremonial magic can kick start your use of possessional persona processes.

For those of you without a bent towards the traditional. Modern magic has taken the use of invocation ever further by drawing down fictional Godheads. As exemplified in Phil Hine’s Pseudonomicon, harnessing the Lovecraftian realm.

And I could go on, and on. Clearly I’ve barely scratched the surface of inspirational sources for your persona testing.

It would be inconscionable of me to leave you with a warning from the Pseudonomicon itself, which because of the power of reverse psychology will probably do exactly nothing of the sort:

“working with the Cthulhu Mythos is dangerous due to the high risk of obsession, personality disintegration or infestation by parasitic shells”

Yup consider yourself warned.

Treat the use of Personas in Software Testing with care.

Have you tried to build your own definition of Software Testing? One that you can refine as you learn more stuff and the years go by?

That never worked for me. I don’t appear to align myself well with definitions and classifications.

Building my own models however, now that works better for me.

I have started work on a new model. I want to create a simpler meta model of my testing process.

I draw inspiration for this activity from various sources, some I have listed below:

"I must create a system or be enslaved by another man’s
I will not reason and compare: my business is to create."

William Blake, Jerusalem

When we try to convey thought by writing, we are bound to sit down solidly, and construct a holy Qabalah out of nothing.

Aleister Crowley, Magick Without Tears

Tao was always nameless.
When for the first time applied to function, it was named,
Inasmuch as names are given, one should also know where to stop.
Knowing where to stop one can become imperishable.

Tao Te Ching, as translated by Ch’u Ta-Kao

It is the part of the scientist – of the intelligent and honest man of letters and of the intelligent and honest clergyman as well – to entertain heretical and forbidden opinions experimentally, even if he is finally to reject them.

Norbert Wiener, ‘God & Golem, Inc.’

You keep using that word. I do not think it means what you think it means.

Inigo Montoya, as channelled by William Goldman for “The Princess Bride”

You’re on your own. And you know what you know.
And YOU are the guy who’ll decide where to go.

Dr. Seuss, “Oh, the places you’ll go!”

Do you have any quotes that inspire you that you would care to share?

More on the model… later.

There I am, figuring out how to debug my FitNesse automation from within eclipse. And up pops the Firefox ‘update failed dialog’ and interfering with my automation.

A bane and a pain when using Selenium RC. But with WebDriver there are easy ways round this.

Start firefox with a profile and set the "app.update.silent" firefox property to true.

The update error will still happen, but at least firefox won’t try and tell your automated processes about it.

profile.setPreference("app.update.silent", true);

For more details visit http://kb.mozillazine.org/Category:Preferences

Oh, and the FitNesse in unit tests is http://fitnesse.org/FitNesse.UserGuide.RunningFromJunit

e.g.

      @Test
      public void runAFitNesseTest(){

            JUnitHelper helper = new JUnitHelper("./fitnesse",
                        new File(System.getProperty("System.java.io.tmpdir"), "fitnesse").getAbsolutePath());

            // type in the name of the test you want to debug here
            String testName = "FitNesse.AcceptanceTestSuite.ATestCase";

          try {
                  helper.assertTestPasses(testName);
            } catch (Exception e) {
                  // TODO Auto-generated catch block
                  e.printStackTrace();
            }
      }

I generally test web apps. And Web apps generally use an email address as the unique identifier. So by test number 2, some of you may have run out of email addresses to test with.

If this happens to you, don’t panic! Because here are the Evil Tester hints and tips for getting more email addresses than you probably ever wanted, but as a tester, have always needed.

Read the rest of this entry »

I’ve been programming more JavaScript recently. This helps my testing in a number of ways:

• When testing web sites I can understand the client side code
• I can nudge the client side into different states by executing ad-hoc JavaScript through the console
• The DOM web developer displays make ever more sense

It also helps my automation;

• My ability to use the JavaScript calls has improved so I don’t have as much trouble with web sites that don’t play nice
• My CSS selector skills have improved

Clearly for most Selenium automation purposes, we don’t need a large grasp of JavaScript, we mainly do quick DOM access scripts, the kind of thing you would do through the console for debugging. I have found the JavaScript Pocket Reference a handy little book.

• JavaScript Pocket Reference [amazon.co.uk | amazon.com]

I have done more programming in JavaScript. Going deeper into JavaScript and building a few apps has helped me enormously. I have used “Test-Driven JavaScript Development” by Christian Johansen as a learning text, and found that useful.

• Test-Driven JavaScript Development [amazon.co.uk | amazon.com]

If you want to learn more about JavaScript I can recommend the following resources to start.

Free Online JavaScript books:

• Eloquent JavaScript has an earlier version than the print book, but still very good, with an excellent section on Objects.
• Essential JavaScript Design Patterns provides a concise introduction to various usage patterns.
• JQuery Fundamentals
• Building iPhone Apps with HTML, CSS, and JavaScript from O’Reilly Open Feedback Publishing System a good general intro to HTML, CSS and JavaScript

I found the following CSS Selector links useful

• http://net.tutsplus.com/tutorials/html-css-techniques/the-30-css-selectors-you-must-memorize/
• http://tools.css3.info/selectors-test/test.html

And remember to subscribe to the JavaScript Weekly news

I have been using WebStorm as my JavaScript IDE as out of all the IDEs I tried, the JavaScript debugging worked out of the box. WebStorm has now become my default HTML and XML editor.

Have you had anyone ask you a question about estimation? I get asked these types of questions and I suspect that the person really wants answers about how to communicate and justify their guesses.

I think they hope that some process exists which will accurately and objectively give them a set of numbers. And by using these numbers they can disavow responsibility for the production of them. And no-one will hold them responsible if the objectively produced ‘estimate’ does not meet reality.

Well in reality ‘Estimate’ really equals ‘Guess’.

So if you want some strategies:

• Answer the question you want them to ask
• Ask how long they want it to take
• Trust your gut
• Assumptions, Risks and Issues
• Track the passage of time

I could add an etc. in there, to alert you to the incompleteness of all of this.

But I won’t.

You should assume incompleteness in my presentation. And map it on to your model to identify the blanks.

What culture do you think you work in?

Do you think you work in a culture where ‘estimates’ and ‘actual time’ have become synonyms?

Do ‘they’ berate you for continuing to work, after the ‘estimated’ length of time has passed?

Well, you have learned that regardless of where the estimate came from ‘they’ will still hold you responsible for it.

So you need to develop some beliefs. You need to believe that “estimate equals guess”.

You need to believe that any supporting documentation for your guess does not provide a justification for it, it provides a partial model of the thinking that led to the guess.

‘They’ don’t need to believe any of this.

You need to.

Then your communication will change.

Estimates equals Guesses

All estimates equate to GUESSES.

We don’t know how long things will take.

I’m a pretty good psychic. But even I don’t know.

Sometimes we put a framework around our GUESSES. And that can reveal some of our ASSUMPTIONS. It can reveal what RISKS we have considered. It can reveal what ISSUES we perceive.

The framework reveals part of the model we think we used when creating the GUESS. It helps us explain how we created the GUESS.

But even when we do that, our estimates remain GUESSES.

So what could you do?

Do you try and ‘educate’ them?

I don’t.

I’m obnoxious like that.

• So if you get asked “How long will your testing take?”
  • Don’t say “I will provide you with an estimate for testing.”
  • Say “I don’t know.”
  • The conversation might end there.

You might respond with the question you want them to ask.

• So if you get asked “How long will your testing take?”
  • ask “Do you mean? If I had to guess, right now, how long I think it will take for you to have enough information that you can decide whether or not you want to release this? If so then…”
  • Yup. I say stuff like that.
  • Because then we start talking about what they really want to know.
  • I don’t think I engage in an education process. I try and communicate so we all work from the same assumptions and understandings.

The test managers among you might feel a little nervous with this. As a test manager I used to feel nervous saying that. So you might ask instead:

“How long do you want it to take?”

We live in a world of deadlines. So guesses often don’t help very much.

By asking “How long do you want it to take?” you can build a plan which tries to maximise the value that you can add as quickly as possible before the deadline. Work on the high value items, risks and issues.

Don’t worry about the guesses. Think about the constraints.

When you ask “How long do you want it to take?” then you can think about how much stuff you might fit into that timescale, does it ‘feel’ right?

Trust Your Gut

What you ‘feel’ can feed into your guessing process.

You can get the whole guessing process over really quickly if you just come up with number.

You won’t have stated any assumptions. You won’t have considered any risks. you won’t have any weighted factors. But you can come up with a number quickly.

And your gut will tell you how you feel about it.

Does it feel right?

More importantly – does it feel wrong? If so, you can:

• Ask yourself what feels wrong about it. Bam. Now you’re starting to come up with assumptions, risks and issues.
• Come up with a new ‘better’ guess.

When you immediately decide to come up with a new ‘better’ guess. Then you trusted your gut.

If you ask your questions. You distrusted your gut. So you ask those questions to help you regain your trust in your gut.

Assumptions, Risks, Issues

Why bother with the questions?

Why bother with Assumptions, Risks and Issues?

Well, they help you communicate.

Until you develop the force of personality where you can give a number and have it accepted as a best ‘guess’. You might want to engage in a communication process.

If you do want to communicate then you can explain to people the things you have assumed. e.g.

• I assumed we deliver a release candidate on day X
• I assumed we have a clean build with no failing unit tests
• etc.

You can do the same for risks and issues.

We track the passage of time

If your guess gets shot down and ‘they’ convince you into using a shorter guess. Then you can ‘manage’ the on-going process by monitoring for any:

• negated assumptions
• change in riskiness of the risks
• transformation of risks into issues

This may not help in the short term as you simply point out that the correctness of the guess becomes ever less likely.

But you might help build credibility long term by communicating based on your frame of reference.

We track over time, the ‘goodness’ of our progress, and how much distance we appear to have to travel before we reach our end point.

We validate our model over time, by comparing it with the reality that we perceive.

Have you covered everything?

No.

I just want to encourage you to keep it simple.

I think you could waste time on a complicated ‘estimation’ process using spreadsheets and lots of factors.

‘They’ might well think they need those spreadsheets. Their process may require those spreadsheets. If so, use the spreadsheets to communicate your model, don’t use them to create the guess.

Of you could:

• guess,
• start to do the work,
• based on how long it currently takes extrapolate forward and…
• guess again

If you do this then you need more courage.

Swapping a belief of “objective estimation”, for “guesses and underpinning models” takes time.

I will not try to convince you that you should.

I just want to make you aware that you could.