I spoke at the Online BrowserStack Breakpoint conference.
Much of the automating we do to support testing involves detecting change. Once our tests pass, they fail when the system changes and the automated execution alerts us to the change. There are other ways that automating can help us.
Links
A live recording is included in Evil Tester Talks which has been edited for audio and easier replay. The headshot video in Evil Tester Talks recording is larger. higher resolution because it was recorded locally through a camera rather than the conference system.
The Evil Tester Talks release also has a full transcript for both live, Q&A and and practice sessions.
Automating to Augment Testing
Published blurb:
Automation isn’t just to enable testing—it can also be wielded to support testing. Much of this involves detecting change. Once tests pass, they fail when the system changes, but we can automate the change detection work. We can also automate tactically to achieve quicker results. And if we write abstraction layers well, we can reuse them to create ad-hoc tools or ad-hoc scenario execution. Alan Richardson will talk about the other ways in which automation can help us, using case studies and his own experience.
Blurb
Much of the automating we do to support testing involves detecting change. Once our tests pass, they fail when the system changes and the automated execution alerts us to the change. There are other ways that automating can help us. We can automate to augment and support our testing, as well as automating away some of the regular change detection work. We can automate such that tools observe the system or traffic as we test, and alert us to conditions we might need to be aware of. We can automate the system when it is unstable to alert us to the fact that it might now be stable enough to spend time testing. We can automate tactically to quickly achieve results. And if we write abstraction layers well, we can reuse them to support us creating ad-hoc tools or ad-hoc scenario execution. This talk is going to focus on some of ’the other ways’ automating can help us, and using some case studies, the topics and skills I had to learn to use them.
There are four main case studies I will explore in more detail in the talk.
- Automating a registration form, which was unstable and had a lot of errors. We automated it to detect obvious bugs, prior to spending time testing it.
- Ad-hoc tooling experiments to support testing - I’ll show an example tool using Chrome Debug Protocol to scan http traffic for preconfigured words that I used to test for email leakage in messages, and how to use Proxy tools for similar functionality.
- Micro Bots to support testing - JavaScript bots that run from the console to create data and control systems to support testing
- Automation Bots - re-using Automation abstraction libraries to create bots in parallel that ‘use’ the system to support multi-user exploratory testing with a single human testing, and check for memory leakage.
Key Take Away Points:
- Abstractions that we use for automating can be used to support exploratory testing
- Automating can be tactical as well as strategic
- Test Automation is automation in support of testing
Slides
Live Recording
Related Projects
This talk uses code examples from:
- Model Based Example of Form Filling
- Chrome Debug Test Helper Prototype
- JavaScript Bots
- JavaScript Courses - My TestAutomationU course and Patreon “Protect The Square” course.
- RESTMud Autonomous Automation Bots
Interview
I was asked some questions prior to the talk.
An edited form of these were uploaded to the browser stack blog.
1. What does it mean to be an ‘Evil Tester’?
I try to tempt people into pushing the boundaries of what is acceptable, and do things to the system that no-one else would do. Things that no-one else would even imagine doing. Push the features to their limit to see how much they can cope with.
But it also means. Think the things that no-one else is thinking. Projects tend to have a lot of people on them, and that can lead to group think. We need to have a balance that causes people to stop, revisit their assumptions, and challenge the norm.
There are plenty of experts that will tell you that you “must” do certain things, and that you “can’t” do some other things. Think differently. Do whatever it takes in your environment to make a difference.
That’s what I mean by “Evil Tester”.
2. Can you tell us about the books you’ve authored?
There are three ‘current’ books: Java For Testers, Automating and Testing a REST API and Dear Evil Tester.
And the are all based on experience and trying to fill gaps in the market. I wrote Java For Testers because Testers tend to use Java differently. We don’t tend to create apps, we tend to create tests, and use a lot of libraries. So Java For Testers, teaches the basic Java needed with everything written Test First and covering libraries separate from core Java. Because once we know the basics of Java, everything else is just adding in libraries. To often people learn ’libraries’ rather than coding. And I saw too many people using ‘main’ methods to drive their automated execution because none of the YouTube videos they had watched used Test Frameworks.
Automating and Testing a REST API was designed to provide an introduction to the subject of REST APIs built around a Case Study and so it quickly covers the theory of APIs and HTTP, driving APIs interactively from the command line, GUI tools and Proxies. And covers automating with REST and HTTP libraries with various abstraction layers.
And Dear Evil Tester is a book encouraging the reader to think differently about testing, to challenge their assumptions, to think for themselves, and to develop the attitude and mind-set of a Tester.
I basically write the books that I needed when I started learning the subjects.
Then I have a bunch of older books for free on my web site since they are slightly out of date, still useful information, but I don’t promote them any more.
3. Can you give us a sneak-peek into your session for Breakpoint 2020? (Just a teaser with some takeaways)
I guess the core is about thinking differently about automating, with some real world examples of using automated execution differently.
I see a lot of people saying don’t automate when the system is unstable, so I have a case study of why that seemed like the most appropriate way of approaching the testing of one of the applications I worked on.
Examples of automating to support the tester, rather than automate acceptance criteria. Examples of re-using abstraction layers to support more different types of automating than linear paths.
Again, in the theme of “Evil Tester”. Thinking about what we ‘can’ and ‘could’ and how it ‘might’ help us, rather than what we ‘must’ and ‘should’ do, because everyone says that is what we do.
4. Have you worked on any side-projects recently?
I am always working on side-projects. Some are secret, and works in progress that might never happen.
But more are public and on my Github account.
Listed on my profile are a lot of test apps, I’ve written, for people to practice their testing and automating against.
I’m currently working on a Model Based REST API generator, called “The Thingifier” which currently can spin up a full API from a simple model of Entities and Relationships, complete with a GUI and Documentation. I will expand that to do more in the future. But at its most simple level it is a way of creating ‘practice APIs’ for learning how to test and automate APIs.
I also write applications to support my Digital Marketing, so I have a Twitter client called Chatterscan.com which is the only way that I read Twitter now. And I have some support tools for creating social media images and formatting TweetStorms. They are all listed on my marketing site:
And I have a lot of other tools that I don’t make public because I use them in my personal work.
5. What are you reading/learning right now? What made you interested in this?
Currently, in terms of study. I’m mainly studying Direct Marketing, Advertising and Copy Writing. Which includes people such as: Drayton Bird, Bob Stone, David Ogilvy, Rosser Reeves, John Caples. And a tonne of other people.
This is because I have to do all the marketing for my own books and courses, and effective sales and benefit communication is one of the keys to successful bug reporting and change management.
6. What advice would you give to people who’re new to testing?
Get hands on as fast as possible. Test as many applications and different types of applications.
Make notes of everything that you do, as you do it, in enough detail that you can come back to it months later and understand what you were doing and thinking.
Keep a list of all the things you don’t understand.
Work through that list of ‘stuff you don’t understand’; search for it online, watch videos, and read blog posts, until you do understand it enough that you can use it as part of your testing.
Never assume that an application is so simple that you can’t learn anything from it. I keep going back to old practice applications I’ve written, to see how can I view them differently now, and how differently I can test them now, based on all the things I’ve learned since I last used it.
If you want to go beyond ’test apps’ to practice on, then sign up for Bug Bounty programmes like ‘Hacker One’, you might not find any security issues, but these are sites and environments that are giving you permission to test them.
Keep learning. Keep practicing. Make notes.
Develop your own models of Testing so you can explain your approach, and the risks you identify, in your own words.
