Evil Tester

I think people use ‘standard’ testing phrases too much. e.g.

• System Testing
• Testing
• Performance Testing
• Functional Testing
• Non-functional Testing
• Black box testing
• Unit Testing
• etc. etc.

The user of each phrase holding the assumption that the reader/listener of each phrase understands the phrase the same way as the user. They don’t, they have their own models that underpin that phrase.

Try using real words.

One recent example from the world in which I inhabit:

• I could have said “we need to build automated integration tests for these stubs”.
• Instead I said “we need write some automated scripts which check that our stub behaves the same way as the system we have stubbed”.
• I received the answer. “Ah, I call those , <xyz> tests” (I forget the actual words of the <xyz> label)
• I said “Fine, let’s write some <xyz> tests”

Now we have a label, for the concept. And more importantly we have agreement on what we intend to do.

I used real words to communicate.

You can do this for all aspects of your testing, resulting in contextual communication of your test planning process:

• performance  testing might become “check that the user can use the logged-in page in under X milliseconds after logging in”
• load testing might become “check that the server (web and app) don’t show any signs of stress under the estimate load of the first two hours after launch (we’ll monitor it on the server side”
• “any testing phrase” might become “some other set of words in your language of choice that communicates what you mean”

Feel free to inform your planning process, if you must, with concepts such as “performance testing” or “load testing” or <insert any other testing phrase here>. But do not use them to communicate.

I think that eventually you will wean yourself off those testing concepts and start to work from first principles and communicate in simpler, non-buzzword and less ambiguous terms.

I recommend you try it:

• to communicate your test planning.
• when asking questions on LinkedIn.
• when thinking.
• just try it.

As I worked on the code for my “migrating to selenium 2.0” workshop at Tony Bruce’s Skills Matter event recently. One of my automation challenges involved waiting for an element to have clickability. It existed in the DOM, visible, and all other rendered event style isCondition()s. But when clicked on, it could not respond to events.

For a reason of bias, I spent far too many minutes banging my head against the automation wall. My brain insisted on trying to fix the problem at the point my code encountered it, i.e. when trying to click on it in the Page Object.

private void clickSubMenuItem(String subMenuText) {

    // insert lots of banging of head with failed automation code here
    // because the element below is visible but isn't responding to clicks yet
    // can't figure out how to detect if the element is responding to clicks
    // before I click on it because click doesn't throw an exception
    // because it is there to click on

    String xpathExpression = "//div[@class=\"GALD-WOBF\" and .=\"" + subMenuText + "\"]";
    driver.findElement(By.xpath(xpathExpression)).click();
   
    wait.until(weCanSee.GWTTitleMatches(subMenuText));
}

I solved the automation issue by waiting, at the point my code puts the system into the state of unclickability, and waiting until it re-enters a state of clickability.

public void openClose() {
    WebDriverWait wait = new WebDriverWait(driver,20);
    ExpectedConditionFactory weCanSee = new ExpectedConditionFactory();

    By imageLocator = By.xpath("//div[.=\""+ this.menuHeading + "\"]/../div/img");
    driver.findElement(imageLocator).click();

    // This is the wait I needed in the code above, but it only made sense here
    // do not consider it open or closed until the menu stops growing,
    By parentDiv = By.xpath("//div[.=\""+ this.menuHeading + "\"]/../../..");
    wait.until(weCanSee.elementStopsGrowing(parentDiv, driver));
}

Both waiting styles offer valid automation choices.

I found it interesting that I defaulted to the first style, and I don’t think I’ve used the second style as much.

You can imagine more trade-offs associated with each style:

1st Style – Before Follow On Actions

• Tests might go faster because, follow on actions may not require earlier waits as guard conditions
• Page objects become more complicated because we have to remember to check for all guard conditions before taking action
• Guard checking happens in multiple places
• Risk of test intermittency because of emergent behaviour which we might find hard to debug

2nd Style – Immediately After triggering condition

• Exit condition checks in single place
• Less chance of exposing emergent behaviour as application keeps synchronised to a ‘good’ state
• Leads to doSomething and doSomethingAndWait style methods

For some reason my approach to automation defaulted to the first style.

This post should act as a reminder to consider both styles, and maybe I won’t get stuck too long on a synchronisation problem again.

The Software Testing Club have a competition running in the March 2011 issue of The Testing Planet to win a copy of Selenium Simplified. You can find the competition on page 9. Right after the review by Michael Larsen.

I’ve been waiting for the pdf version of The Testing Planet to be released so that I could read the review and can report with great pleasure that Michael found value in the book and gave it a positive review.

A similar competition run in Professional Tester Magazine ended with a winner very quickly so download the magazine and get your entries in to The Testing Planet quickly.

Best of luck.

This tutorial on Burp Suite covers the Repeater and Intruder functionality.

• Repeater allows you to play back a message to the server and amend it before it goes out.
• Intruder allows you to play back messages, with various elements of the message varying with each playback e.g. a different set of parameters

In the tutorial video I explain how I use each function in my testing.

Other proxy tools do similar things. I also use JBroFuzz as it provides very similar functionality, so experiment with that too.

Did you can spot the deliberate mistake in the video? Blogging helps you make mistakes in public very easily – which helps you learn quickly. Any comments on the video will help me improve the quality of future videos.

In March 2011 I gave a talk at the London Sigist on technical testing. I’ll make the slides for that available (…sometime soon). I didn’t want to give a ‘blaggers guide’ to technical testing. So I presented an overview of some of the thought processes and models I use.

At the end of the talk I provided a list of tools that I use. I use Burp Suite as one of my proxy servers.

I currently have a “No Excuses” hat on, so I currently try to provide as much information as I can in bite size chunks which people can pick up and move forward with. I wrote “Selenium Simplified” as a “Now you have no excuses for not learning how to program” book.

I recommend that if you want to go further with technical web testing you read the book “The Web Application Hacker’s Handbook” written by the people behind the Burp Suite tool. [amazon.com] [amazon.co.uk]

And in the same spirit I will now experiment with some videos. In the first of which I provide a simple overview of Burp Suite, in particular the Intercept and Site Map functionality. I don’t cover the nuances of usage, but I cover enough to get you started. So if you haven’t started using a proxy server as an essential part of your web testing… no excuses – start here:

I still have a lot to learn about creating video tutorials, so I appreciate all comments.

A while back, Rob Lambert started collecting Nursery Rhymes for the Software Testing Club.

Since I enjoy Humpty Dumpty I created a version for STC. It has just become available on the Software Testing Club website, along with Rob’s entertaining take on The Gingerbread Man.

They have posted the uncensored version of Humpty so those of a weak disposition take heed of this warning before visiting said site.

We all know by now the following things about test plan documents:

• a test plan document does not substitute for the process of test planning,
• a test plan document represents one way of communicating the results of the test planning process,
• the process of test planning stops only when you stop testing (even then, you might obsess so much that you carry on planning)

I didn’t always know those things.

One of the parts of Selenium 1.0 that I never enjoyed was debugging automation that didn’t work. I had to faff about creating custom Firefox profiles with Firebug installed and set to go through a proxy.

Selenium 2 makes all of that so much easier. With the code below, my test runs through a proxy server on port 8081 and has Firebug installed so that I can breakpoint my test and then go debugging around in the browser DOM, debug the JavaScript to make sure events get fired, etc. etc.

I found most of the information to do this on the Selenium Forums http://groups.google.com/group/selenium-users

The code should be easy enough to follow if you are already using Selenium 2

import java.io.File;
import java.io.IOException;
import org.junit.Test;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxProfile;

@Test
public void FirefoxUseExtensions() throws IOException{

  // setup the firefox proxy to point to Burpsuite
  // on port 8081

  Proxy localhostProxy = new Proxy();
  localhostProxy.setProxyType(Proxy.ProxyType.MANUAL);
  localhostProxy.setHttpProxy("localhost:8081");

  // Prior to running the test
  // Download the firebug extension file
  // to a local folder

  String extensionPath = System.getProperty("user.dir")
      + "/"
      + "firefoxExtensions/firebug-1.6.2.xpi";

  // create a custom profile

  FirefoxProfile profile = new FirefoxProfile();

  // set the profile to use the proxy settings

  profile.setProxyPreferences(localhostProxy);

  // stop firebug showing the first run
  // screen by setting the 'last version'
  // to the current one downloaded
  // try without this and see what happens!

  profile.setPreference("extensions.firebug.currentVersion", "1.6.2");

  // add the extension to firefox

  profile.addExtension(new File(extensionPath));

  // start firefox with the custom profile

  WebDriver driver = new FirefoxDriver(profile);

  // go to your favourite testing web site

  driver.get("http://www.eviltester.com");

}

Honestly who doesn’t enjoy reading conspiracy theories? Who doesn’t enjoy putting on the “Helm of Paranoia”.

I use paranoia as a learning strategy and as a testing strategy.

You can too…

For learning:

1. Assume that the author of the book lied to you
• This forces you to do more research
• Read additional books
• Read differing opinions
• Check their facts
2. Assume the author didn’t want you to know the truth
• They deliberately obfuscated their writing so you have to simplify them
• They deliberately made it overly specific so you would not learn the different ways to apply the technique, so you have to generalise it
• They deliberately didn’t teach you the principles, so that they would retain the edge and you could never compete. So you need to take their approach apart to identify the principles that it codifies
3. Assume the author didn’t know the truth and others misled them
• Only you can draw out the patterns in the text to identify the underlying earth shattering secret truth that has remained hidden in the text.

For testing:

1. Assume that the system has lied to you
• Yes it ‘says’ it has saved the record – how do you know?
• Yes it ‘says’ you can’t use numbers in the password – how do you know?
2. Assume the system didn’t want you to know the truth
• Look for the hidden buttons that others have overlain and obscured
• Look for the hidden windows
• Look for the hidden div elements (how can you make them appear)
• Look for the hidden form fields – what secrets do they hold
• Look for the secret cookies – what do they mean? What information have they tracked about you?
• What other secrets has it hidden from you?
3. Assume the system doesn’t know the truth
• Can you mislead it?

The truly paranoid among you will now rightly assume that I have lied to you, that I don’t want you to know the truth, and that others have so misled me that what you have here is a pale reflection of the truth that only you can uncover.

And for your ongoing research pleasure, I heartily recommend Paranoia Magazine. I haven’t read it for a long time as I haven’t found a shop that will stock it – no doubt due to the machinations of one cabal or another. But that won’t stop you getting a good old historical dose of paranoia with the free back issues at http://www.paranoiamagazine.com/freedownloads.html (Just don’t tell them I sent you, they don’t need to know.)

My testing style, attitude and approach has had many influences. I only recently realised that one of them has become very mainstream.

"nothing is true, everything is permitted"

Words attributed to Hassan-i Sabbah on his death bed. Specifically ambiguous and open to misuse. Perfect for testing.

• http://old.disinfo.com/archive/pages/article/id1562/pg1/

This went all mainstream with the “Assassin’s Creed” game series

• http://assassinscreed.wikia.com/wiki/The_Creed

Fortunately when dealing with software systems, many of the moral dimensions of the Assassin’s phrase disappear and we no longer have to engage in the philosophical debate about boundaries of human actions. We just get on with the testing.

Objections such as “No user would ever do that” have no stopping power on the Assassin influenced tester.

If the system allows me do it, and I observe that the system allows me do it, and I figure out how to manipulate the system into doing it, I do it.

This phrase helped me through some difficult and turbulent waterfall testing times.

Try it on and see how it fits your testing style.

Just remember to treat software systems and people systems differently.