Evil Tester

• I may have some of the arrows the wrong way around in the above diagram. After all, who really knows what influences what.
• And as to the “?”, I hope that you decide what comes next.

There exist test ‘concepts’ which, while seemingly simple, have a tendency to melt my brain.

• Black Box/White Box Testing
• Functional/Non-Functional Testing
• Positive/Negative Testing

Years spent studying hypnosis and revelling in the ambiguities of communication have left me with an inability to parse language the way I did as a child.

I used to have the ability to read, and use, these terms without blinking. Now I throw a ParseException.

I realise the above concepts may not seem strange to the majority of people reading this blog post. So I will attempt to describe how I respond to reading these terms, in the hope of spreading a contagion against them.

First some generalities

First some generalities across all the ‘concepts’.

I think that I think, and test, in terms of Systems. More specifically in terms of models of systems.

Yes/No models jar my sensibilities. Testers lives in the grey areas and find treasures in the shadows. Absolutes do not help me explore the System. I do not want guidance by absolutes, I want guidance towards flexibility.

All of these concepts encourage me to think and communicate in ambiguities, while pretending specificity and authority. If I say White Box testing, you still have no idea what I mean. If I describe the models that I use to guide my testing then you have a better idea of the type of tests I might create, and the type of gaps I may encounter, regardless of the colour that I paint that model.

These concepts all encourage me, as a tester, to ignore the model and think in vague conceptual terms. They categorise techniques out of context and far removed from the model that the techniques apply to.

They encourage debate about irrelevancies. e.g. Should we do White Box, or Black Box testing? Developers do White Box Testing, testers do Black Box Testing? Where does Grey box testing come in?

The amount of time wasted on projects when people ‘debate’ if the requirement should form part of the ‘functional’ or ‘nun-functional’ specification, instead of getting on with communicating what they need shocks me. (I estimate this in terms of man-years for some projects)

etc.

Black Box / White Box Testing

Ah, the most pernicious of them all.

How comfortingly I viewed these terms in my early years. I could ignore the relationships between my test models. I remember times when I made no effort to learn the technology or architecture of the system. I could simply focus on inputs and ignore flow, because I did my testing all “Black Box” like.

Then over time I grew uneasy. I started to apply White Box techniques to ‘Black Box’ingly understood systems.

This concept started to seem like the devil on my shoulder whispering “Relax, don’t do the work to view it as a system, treat it as a ‘black box’, see how much simpler it all becomes”.

I started to feel as though the constant repetition of this phrase in the testing literature stopped me hearing the siren call of the technology.

I weaned myself away from these terms by using them as insults. I engaged in a form of aversion therapy.

I labelled all testing I did without understanding the technology, or without having made an effort to observe the architecture, or without building a structural model as ‘Black Box’ testing. I imbued the term with an association of laziness.

I view all my areas of superficial modelling, as having greater risk of defects unfound.

I associated my use of ‘White Box’ testing with over-confidence. I felt nervous if I only read the code, or only based my testing on a superficial understanding of the structure. For me, the phrase warns me about my complacency. I try not to plan for complacency.

This helped me move towards a more Systems focussed approach to my software testing. I no longer feel a pull towards exclusively the Darkness or the Light.

I try to concentrate on models and identify techniques to apply to models.

I ignore the categories and partitioning of techniques presented by the Canon.

Functional/Non-functional Testing

I periodically write non-functional automated test scripts… Sometimes my scripts simply don't work and they remain non-functional until I fix them to get them functional. <insert canned laughter here>

In the past, I have communicated my testing in terms of “I built tests to cover the ‘Non-Functional Requirements’”. Which in context could have read “Speed Monster Funky Requirements” and I would have just as happily used that phrase in my test communication.

When we mean requirement based testing, then we should say “requirement based”. Removing the word “requirement” suggested to me that “Herein lies magic”. That the “Functional” tester does something different from the “Non-functional” tester. That we can phase the testing differently and do “Functional” testing and then later do “Non-functional” testing.

I removed my unease towards this phrase by always appending/inserting the word requirement.

But … I don’t like the distinction between “Functional” and “Non-Functional” because of the ineffectual processes and discussion that I have seen result from the distinction. Sadly Software development has a history of considering functionality as separate from the other “ilities” that might relate to that functionality. A tradition or an old charter, or something.

And yet I don’t see why I should let this influence my testing. I’ll do whatever ‘it’ takes, and if that means merging the functional with the non-functional to get more advantages in my testing or tooling then I will jolly well do so.

Positive and Negative Testing

I assume “positive” and “negative” as testing terms, must have come from a more objective diagnostic context than I get involved in. From a context where a negative test, rules something out completely, and a positive test rules it in.

I classify much of the testing I do as exploratory or experiment based. I do things to try and rule out certain possibilities and I could possibly count that diagnostic. But I don’t think my tests demonstrate the certainty that “positive” and “negative” testing suggests to me. I find more value in thinking in terms of a general experimental process rather than a diagnostic process.

But even the above description doesn’t seem to map to how the testers I meet describe positive and negative testing.

Testers seem to describe negative testing in terms of exercising error validation and reporting. And positive as “stuff we think will work”. I tend to view the existence of error handling functionality in applications as a positive thing.

I find it hard to use the terms –ve and +ve testing. When I try to show that the system does not work, I talk in terms of trying to ‘break’ it – clearly I don’t break it, but it works as a term for me. I also view this as a positive thing for me to do.

Yes, I freely admit to misusing words, particularly if the misuse of the word frees my thinking. I try to stop using words that constrain me.

End Notes

For all of the above ‘concepts’ I can read the definitions and almost understand why those names fit those descriptions. But I try to find other, more exact, or more contextual, or more personal, ways of describing the testing that I do.

You may not need to go as far as I did to de-programme their effect.

Younger testers may even have an immunity to them.

I still feel their pull though. My brain still finds a ‘reasonableness’ in all of them while at the same time each of the ‘concepts’ strikes me with dread. Each of them seems to proudly state that we can easily view the System as two simple partitions and we can relax about modelling or understanding the complexity.

If you speak to me using these terms, I will have to ask you what you mean. Do not assume that my testing background will allow me to understand you.

I try to stand firmly in a graduated shading testing scheme. Sometimes I even imagine colour in my test approach. Someday I will attempt to employ synesthesia.

Did you enter? Probably not. And if not, you lost your chance to save the world.

The survey will remain open as I found many of the answers highly entertaining. So feel free to pop random entertaining snippets into the survey and I shall read them and chuckle.

Dare I draw conclusions from the survey, given some may have viewed it in jest?

Of course.

But first… three serious points.

1. I take software testing incredibly seriously. I poke fun at it. Because I take it seriously.
2. If it did not seem like a serious topic to me. I wouldn’t bother poking fun at it.
3. I consider software testing so serious a topic that I need to think about it, not just coast along with the Status Quo. I can’t help find humour in it if I think about it. We laugh to change. If I don’t find a way to mock it then I probably haven’t thought it through from enough angles.
4. Do you find testing serious enough to laugh at?

So, on with the statistics and numbers. The closest you will come to metrics on this web site.

We asked “Do you hold a Certificate in software testing?”

Our Survey said…

• Yes == 56%
• No == 44%

Proof positive then that the people who replied know what they replied about. I think.

We asked “Did your certification offer value for money?”

Our Survey Said…

• I'm not certified, didn't you read my answer to question 1? == 41%
• Yes == 13%
• No == 31%
• Definitely not, I got my money back == 0%
• Other == 16%

Contentious. Some people didn’t think it offered value for money, BUT DIDN’T ASK FOR THEIR MONEY BACK!

Meaning – this could become a profitable business to get into. No-one asks for their money back.

Certification providers take note, you can drop your standards and it won’t make any difference.

We asked “Would you like to self certify yourself in Software Testing?”

Our Survey Said…

• Yes, clearly that would be better == 50%
• No, I can't possibly know better, or as much, than the official certification experts. == 3%
• Never! What new certification scam are you trying to pull? == 47%

It seems as though some people saw through my thinly veiled attempt to see how much appetite exists for self-certification. And 50% think it sounds good. So before the certification crowd jump on this particular bandwagon. This one belongs to me.

And 3% thought they didn’t know as much as the certification experts. Which clearly (in an abuse of the survey process, but in keeping with poor statistical analysis) means that 97% of people doing the survey think they know better than the certification experts. And therefore they consider self-certification a good idea.

We asked “If a self-certification scheme did exist, how much would you pay for the privilege?”

Our Survey Said…

• Nothing, it should be free - like all good things in life == 63%
• < £10 == 3%
• >= £10 && <= £50 == 16%
• >£50 && <= £100 == 0%
• > £100 (I only value something if I pay a lot for it) == 3%
• Other == 16%

I don’t know what you see from these results, but I see:

1. a free sample to hook in the 63%
2. an immediate uptake of 35% of respondents
3. hopeful conversion of the 63% to a total of 50%

Self-certification as a potential goldmine.

We asked “If it did cost money, what would a self-certification scheme pack have to contain so that you felt you were getting value for money?”

At this point I’m giving away the secrets of my future success. Because you too could use the information here to create a money spinning self-certification pack.

Our Survey Said…

• Sample Certificates to fill in == 25%
• Sample Answers to help you show off your newly certified knowledge in interviews == 25%
• Empty promises about industry standard knowledge and future career prospects == 31%
• Hints and tips on how to fill in your certificates == 22%
• Affiliate links to lamination services so you could create laminated certificates == 22%
• A direct debit form so you could continue to pay for your certification every year (and thereby 'prove' that you were still worthy of the certification) == 13%
• A letter congratulating you on your decision == 28%
• A document explaining example topics you could certify yourself in == 28%
• Multiple levels of certification in one pack (e.g. novice, journeyman, master, expert, Nobel Prize Winner) == 25%
• A Membership card == 38%
• A Badge == 41%
• Money off vouchers for additional certification packs == 16%
• Other == 50%

Clearly people want to experience a sense of belonging – hence the popularity of the badge and membership card. This may explain why the certification sales pitch based on Social Proof (everyone else does it, why don’t you) does so well.

The direct debit, whilst looking good from a money angle, doesn’t seem that popular, so CAT’s compulsory re-certification scheme might not go down too well with the paying masses. The CAT folk might want to consider changing that aspect of the scheme. But since they didn’t get back to me with previous comments I sent them on their scheme, they might not take any notice of this one either.

So quick wins for all certification providers. Supply a badge and a membership card to boost sales.

We Asked “Do you wish this was a prize survey?”

Our Survey said…

• Yes, I only fill in surveys where I can win a prize == 28%
• No, I fill in surveys for the fun of clicking buttons and feel sure that the person on the end of the survey has read my 'other' answers carefully, so carefully in fact that I have undoubtedly succeeded in changing their mind about something or other == 50%
• Other == 22%

Clearly people are opinionated. And want to express themselves. And you don’t need a prize to get people to fill in a survey.

In fact. Surveys have shown that if you do give a prize, you can’t trust the results.

Meaning. You can trust the results of this survey all the more.

And lastly, We Asked “Is Software Testing Certification Evil?”

• Yes, Obviously == 22%
• No, Of course not == 22%
• Maybe, I'm not sure == 31%
• Undoubtedly, my Holy Book lists certification as one of the main causes of discord in the world == 19%
• Other (and no I won't explain why) == 6%

Pretty evenly split there. And maybe people just answered joshingly, but only 22% of our respondents claimed absolutely certainty that certification did not have the “Big E” connotation.

So time for a PR blitz from the certification providers me-thinks. And if any want to pay me for this marketing advice, please get in touch.

Favourite moments

Ah, the blooper reel. For as the credits roll, we can partake of the ‘other’ answers.

In terms of cost, there clearly exist people I could milk this scheme for all they had for they threw monetary amounts that no-one in the real world would pay for certification. Almost up to £1000 pounds or more for some people. Quite ridiculous and I’m sure none of the other schemes would charge anywhere close to that.

Some really good ‘stuff I want in the pack’ answers. Which certainly made me want to get one.

• Hosted online certificates
• A Propeller hat
• Doughnuts
• Test
• Discounts on framing services
• Cuddly Certification Toys

And some people saw parts of the page in Dutch which I thought splendid.

Conclusion

So there you have it. Self-certification clearly has legs.

And if you see any certification providers starting to give out Doughnuts on their training courses – you know where they got the idea.

Well I have learned a lot about the self-publishing process. Any testers thinking about going down this route feel free to email me with questions and I’ll help you avoid the mistakes I made. The delays in the process were primarily my misunderstanding of the process.

I have finally received a proof of Selenium Simplified and the 2nd printing should start appearing in stores again soon.

I have started work on some Selenium 2.0 writing, in preparation of a workshop in May http://skillsmatter.com/event/agile-testing/london-tester-gathering-2011

Workshop seems to have sold out, but a waiting list exists which you can sign up to.

I probably won’t post much about Selenium 2.0 here until I have the Sigist preparation and subsequent recovery out of the way.

I used to consider incorporating techniques from other disciplines into testing as something a little different. It felt right, but since the ‘industry’ didn’t do that, it seemed like a way of individually revealing our personal approach to testing.

But testing has a secret history. The building of the Traditional Testing Canon has remained shrouded in mystery until now. So for all testers following tradition, set yourself free, continue to follow Tradition, just follow the one true one. As I reveal here as “Thee True And Aythentic Historee of Software Testing - A tale of Action and Adventure”.

Read the rest of this entry »

Tony Bruce posted some quotes inspired by Taoist traditional writing from The Magic of Metaphor. I have a particular fondness for Taoist classics and I have a fair few translations of the Tao Te Ching on my bookshelf. I love the feeling of simplicity generated when reading.

I cherry picked the following quotes with a relevance to my testing.

If you have not before taken the following approaches in your testing then I encourage you to try them (failing this you could just read “The Berenstain Bears Inside, Outside, Upside Down” as initially recommended to me by Rob Sabourin):

On Subtle Wisdom…

• In order to contract a thing, one should surely expand it first.
• In order to weaken, one will surely strengthen first.
• In order to overthrow, one will surely exalt first.
• In order to take, one will surely give first.

What is planted by the best planter can never be removed;
What is embraced by the best embracer can never be loosened.

On Absolute Equality:

• Blunt all that is sharp;
• Cut all that is divisible;
• Blur all that is brilliant;
• Mix with all that is humble as dust;

Chapter LXIV:

• What is motionless is easy to hold;
• What is not yet foreshadowed is easy to form plans for;
• What is fragile is easy to break';
• What is minute is easy to disperse.
• Deal with a thing before it comes into existence;
• Regulate a thing before it gets into confusion.
• The common people in their business often fail on the verge of succeeding.
• Take care with the end as you do with the beginning,
• And you will have no failure.

All the above from the 1937 Translation by Ch’u Ta-Kao

You can find various translations around the web:

• http://www.feedbooks.com/book/215/tao-te-ching
• http://www.taoteching.org/
• http://www.duhtao.com/translations.html
• http://en.wikipedia.org/wiki/Tao_Te_Ching#Online_English_translations

Selenium Simplified was on Amazon, and now is awaiting copies – why? Because I took it out of print to make changes. I was embarrassed by the number of spelling errors and obvious grammar errors in the text so I made a revision.

I’m waiting for a proof copy to come through from the print on demand publisher so that I can get it back on the virtual Amazon shelves again.

Admittedly no-one who purchased the book had complained to me about the spelling errors, but I couldn’t in all conscience, knowing they were there, let it keep selling.

This will undoubtedly have hurt sales, but I felt I had to do it.

Sadly I didn’t expect it to take so long. I expected it to be a quick swap in and out of the PDF with the print on demand publisher but they have a longer cycle time than I anticipated. Hopefully it won’t be much longer.

I am also setting the book up to print on Lulu, so if you need an emergency source to buy it, before it becomes available on Amazon again, you can go direct to Lulu. I’m selling it with a 20% discount bringing the price to £19.99 because of all the trouble people faced. Hopefully Lulu postage doesn’t wipe out that saving for those of you not in the UK.

http://www.lulu.com/product/paperback/selenium-simplified/14939945

The e-book is also still available, and I have held off increasing the price until the 2nd revision is back on the Amazon shelves.

http://www.compendiumdev.co.uk/selenium

I hope it won’t be too long before it gets back in Amazon, but this world of printing is a little slower than I had anticipated. Next time I’ll release it as a new edition instead of a new printing and it should all be a lot faster.

Apologies to everyone wanting to buy it, and thanks for all the enquiries.

I doubt it, but what the heck. Let’s try. Inspired I was by Dot Graham’s explanation of the original reasons behind certification in “Certification is evil?”. Which in turn was inspired by the Software Testing Club “Tester is for life, not just for Christmas” e-book.

My views on certification in the e-book were “You’re better off becoming an ordained minister from the Church of the Subgenius for only $30.00 than putting any money, or time, towards the Testing Certification Scam.”

Read the rest of this entry »

Both my and David Burns’ books are now available on amazon:

• Mine: Selenium Simplified   [amazon.com] [amazon.co.uk] published by Compendium Developments
• David’s: Selenium 1.0 Testing Tools   [amazon.com] [amazon.co.uk] published by Packt

Read the rest of this entry »

When I use TestNG for my Selenium tests, I really like the BeforeSuite and AfterSuite annotations because then I can share a Selenium session amongst the tests. With nested JUnit suites I can achieve a similar effect, and I can also go some way to grouping my tests to make it easier to create browser specific suites.

Read the rest of this entry »