Software Testing Podcast - The Test Cases and Test Scripts Episode - The Evil Tester Show Episode 019

The most common Testing Entities that I’ve encountered over the years are: Test Condition, Test Scenario, Test Case, Test Script. In my Testing I pretty much now have: Test Idea, Test Log (Execution Log). Is there any value in the original Testing Entities? There might be if we view them as logical concepts, but not really as physical artifacts.

This was released to Patreon supporters early, with full transcript and no ads.

Test Entities

The closest to “Test Script” is that I would map that on to an @Test annotated method for automating the execution and assertion of specific conditions.

Legacy Start Point

When I started in testing I took it for granted that professional testing involved using all the entities, and this was usually enforced by the tooling.

There would be a Requirements Tracking style hierarchy that the Test Conditions and Test Scenarios would be part of, and these would have cross references back to requirement documentation.

Some tools had plugins so that if you were reading a requirement document in MS Word, highlight text and then mark it as a Test Condition or Test Requirement and it would be populated in your tool.

This tight level of coupling between tools and concepts made changes incredibly hard to manage and very time consuming.

Test Cases might be organised into hierarchies and they would be cross referenced to Test Conditions to justify their existence.

Test Cases would then be organised into Test Suites to plan the testing.

Each Test Case would have to have a Test Script in order to be ’executed’ and tracking of testing was all done by marking Test Scripts as executed or not.

I don’t know which tools still enforce this type of behaviour and I see online people talking about Test Cases and showing what I thought of as Test Scripts.

Logical Expansion

In one sense the entities:

Test Condition
Test Scenario
Test Case
Test Script

These are:

Derivational (Derived From)
Expansion of Previous

As Logical Concepts they are fine. As Physical Implementations they will impact effective testing

They can be viewed as basically the same thing, but at each level they become more concrete or physical.

Test Condition

Something I’m going to use as a basis for my testing
Input
Process
Output
Derived from External Model

Test Scenario

A collection of Test Conditions
A Test Condition

Test Case

A specific set of inputs
A specific process
A set of expected result
Some people combine this with Test Script

Each of the above from either Test Conditions or Test Scenarios

A Test Case is a collation of Test Conditions as they map on to the system rather than the requirements.

Test Script

A set of steps to follow
A set of interrogations to make

A Test Script is the physical representation of the Logical ’thing we want to achieve’ (Test Case). It is the ‘how we want to achieve it’ (Test Script).

I learned that Test Scripts were too hard to maintain and really if I knew how to use the system I didn’t need a detailed Test Script, I could work from a Test Case because I knew how to operate the system I didn’t need a list of detailed instructions for each Test Case.

Sometimes a list of instructions can be useful, but it would be a template, it wouldn’t dictate specific data.

And the more I knew the system the less detailed the Test Cases needed to be because I just needed a prompt about what I needed to test, what specific idea or condition I was trying to test.

Work from ideas

Eventually that turned into a list of ‘ideas’ and it was more important to document:

what I think I’m going to do and why
what I actually did and how
what I think I still have to

As Physical Concepts:

This is where Testing falls apart, becomes:

highly inefficient,
time consuming,
hard to maintain,
coverage is based on scripts or cases

Formalised as:

spreadsheets,
templates,
tools

Because of where you work you may ‘have’ to use these.

Testing does not require the use of these as physical implementations.

Keep Them Logical

Very detailed representations of these entities makes the process of Software Testing incredibly time consuming.

Imagine… you get a new list of requirements which are documented. You now have to update all the existing conditions, create new conditions, cross reference them to existing or new Test Cases. If the docs list changes to the physical implementation then that means updating the Test Scripts… and there was no ‘find and replace’ mechanism.

Then when you actually started following the Test Scripts, you would get to step 2, discover the system didn’t match the script, then have to go off and update all the test scripts so you didn’t make much progress.

I think the general concepts behind the entiies are fine:

we have to know why we are doing something
we want to know what we are trying to achieve and what distinguishes it from other tests we’ve done
we need to know how to use the software and spot issues

Codifying these concepts into specific entities is good for Tool Vendors, and for expensive outsourced projects, it isn’t good for the cost conscious project or for testing.

Concepts

If we know the concepts behind the entities then we can more creatively work in environments that force us to use tools that implement the concepts.

Test Scripts become execution logs where the ‘actual results’ are where we write our logs. The ’expected results’ are often ‘-’ because they probably made the field mandatory.

But it is still very painful to split your list of ideas into separate Test Cases.

I tend to review the testing I’ve done as “Tests” rather than “Test Cases”.