Skip to main content
blog title image

4 minute read - Testing

Input -> Process -> Output - I test Systems

Apr 18, 2017

TLDR; I test systems, not processes.

Input -> Process -> Output

Input -> Process -> Output

Has always been a useful model for my testing

  • I have to understand the Input and find ways to craft it and control it.
  • I have to know how the Input will be Process(ed)
  • I have to know how Input maps to Output and how I can control the Process
  • I have to identify when I can build an expectation of the Output
  • I have to identify how I can observe the Output
  • I have to know what oracles I can use to evaluate my observation
  • etc.

I can build up a set of expectations and then find ways to observe those expectations and either confirm my model or expand my model with observed output that doesn’t meet my expectations.

Perhaps my model of expectations was not complete enough.

Perhaps I didn’t understand the process.

Perhaps something unintended happened and I found a problem?

Is useful but overly simplistic

It is a rich model but it is overly simplistic and leads to Test Scripts where “Given X, we do Y and expect Z” and this is all we have to look for.

It ignores all the things I don’t know that I don’t know in the process:

  • other inputs - Does it use the clock? Does it monitor the Internet connection? what else?
  • the side-effects - the outputs that I don’t know about and haven’t observed.
  • the state - the existing inputs that I don’t know about, and haven’t observed, and don’t know what effect they have on my process.
  • other processes in the environment which might impact mine - take up processing power, use up memory, deny access to a resource, consume my inputs, consume other inputs, etc.

Because I’m really dealing with a system

In other words, it makes me forget that I’m really dealing with a System.

Input(s) -> System -> Output(s)

A system where all the labels are rich in meaning

Where Inputs include:

  • state
  • existing memory
  • configuration
  • the stuff I enter

System includes:

  • memory
  • version
  • libraries
  • other applications
  • other running systems
  • more than a mere process, this is a system made from other systems, all processing and communicating in ways I have not yet observed or manipulated or interrogated.

Outputs include:

  • side-effects
  • files written
  • temporary files
  • log files
  • database updates
  • system.out messaging
  • memory reserved and released
  • garbage collected variables
  • the stuff I expected and observed
  • stuff that was output that I expect but didn’t observe
  • stuff I know happened but don’t have the technical knowledge or tools to observe

Input -> Process -> Output

Makes me think of something that starts work because I input data and it processes until it has done its work of outputting.

Input(s) -> System(s) -> Output(s)

Systems makes me think of something running and consuming or ignoring inputs all the time and I only jump in to concentrate on the inputs I care about and the outputs I’m observing.

The applications I test don’t fit into an Input -> Process -> Output model very well, but its very tempting to generalise quickly to that model.

And I can force myself to think in terms of Systems

So I have to ask myself:

  • what am I not observing?
  • what is used that I am unaware of?
  • what state exists?

On desktop I used tools such as:

  • file monitors to see what files were written and read as the application ran
  • memory monitors to see how the memory changed as I used the application
  • resource tools to see ‘into’ the application and find out what I missed
  • etc.

On the Web I use tools such as…

Browser Dev Tools to:

  • see what cookies are maintained and changed
  • see what local storage and session storage is used
  • what network traffic is sent
  • what source files are brought in from outside and which are brought in from cache
  • what files are from CDN, what are from servers?
  • what is a POST message and what is a GET?
  • what is in the DOM that I am unaware of?
  • how does the DOM change over time?
  • etc.

Proxy tools to:

  • review information in review
  • to more easily see when cookies are set in headers
  • to better understand the communication with the system(s) in play
  • etc.

Systems I want to Observe, Interrogate and Manipulate

I continually want to increase my ability to interact with the system: to observe its behaviour and data usage, to interrogate its running memory and resources in more detail, to manipulate its state and data and consumed data and ability to output.

I don’t test processes, I test systems.

And while I find it useful to think in terms of:

Input -> Process -> Output

I find it more useful to think in terms of:

Input(s) -> System(s) -> Output(s)

All the way down. And then back up to include myself and others.