TLDR; the API From Hell has endpoints for stressful HTTP conditions to expose the observation and interrogation issues in REST clients. Testers need to understand the limitations of their tools and infrastructure.
Who knew that creating an Evil API would be difficult? Not me, but the creation of the API From Hell was an interesting experience. I learned that HTTP Infrastructure tries to prevent some errors from being seen as well as a healthy skepticism around the results shown in tools when I haven’t seen the raw HTTP request.
Overview
What is it?
The API is a stress test for REST Clients.
I think people trust their REST Clients too much.
- When we see well formed XML body in the response view we trust that was what as received.
- When we look at the headers in the request, we trust that was sent.
I’ve learned to feed all my REST HTTP calls through a proxy, when I test with a REST Client, so I can double check the actual request and responses.
I created the API From Hell as way of helping other people learn to distrust their REST Client.
This doesn’t mean we should not use the REST client, but it means we need to understand the limitations of the client we are using, to avoid being misled during our testing.
What does it do?
It is an API with 70+ end points. Each of which potentially put some stress on the REST Client.
There are 7 or 8 ‘good’ requests when demonstrate how your client handles different content types e.g. HTML, CSV, YAML. But all the rest are there to help you check for limits.
e.g. something as simple as the /version returns a JSON body with no content-type header - how does the client handle it?
Then there are malformed responses with bad XML and bad JSON.
- Does the REST Client ‘fix’ the XML?
- Can you see the raw body?
- Does the client point out that the body is malformed?
From a tester perspective this is important.
- Does the client draw your attention to the error or is it possible to miss the problem?
If the client does not set of a siren and flash lights or highlight errors obviously in red then you might easily miss a problem because you might be biased towards looking at the headers you are testing instead of the body, or you focus on the body and miss the header issues.
Most REST Clients are not TEST tools
Unfortunately for testers. The REST Client Tools are not actually TEST tools. They can support your testing by making it easy to send requests. They can augment your error spotting by adding assertions to requests to check for known problems.
But do any automatically validate the responses against HTTP semantics? e.g. content-type matches the body, stated length matches the length in the body, headers are present when they should be e.g. for a 205 reset, there should not be a body - does the REST client tell you that?
Some do throw errors when there are body content issues. Sometimes this is accidental.
The tools are oriented towards ‘making requests’ and ‘asserting on defined conditions’, not ‘pointing out possible problems’.
HTTP Infrastructure Hides Problems
One additional factor is that often the tools never see problems. Because the HTTP infrastructure removes them.
A good example of this is the /status-code/204-with-body end point.
The returns a Status Code 204 (No Content) and adds content in the body.
Simple.
This was the hardest endpoint to create and test.
The Client tools would see the 204 and automatically return "" as the .body() response, even if there was content.
The HTTP server libraries would strip out the content, when they see a 204 status.
The most reliable way of using the API From Hell for this specific problem is to run it locally either from the .jar or the docker image because I went through some hoops to have Javalin return the content and there is no intermediate HTTP layer stripping it out.
In the current production environment hosted on Railway… the content is stripped out. But… the content-length header is still set.
Running locally it took me some time to be sure that I was actually returning a body because none of the tools would show it.
Even cURL wouldn’t show it - but using a -v to have curl run in verbose mode showed me:
* Excess found writing body: excess = 64, size = 0, maxdownload = 0, bytecount = 0
The only reliable way I found to know that I was actually sending the body was to use an HTTP proxy.
Prototype
My first prototype for the From Hell API was created using Mockoon
A simple to use API Mock.
I’ve never really found a reason to use an API Mock for production testing, but I thought this was a good usecase because it was easy to create malformed responses and I did find a few bugs in some REST Clients that I reported to the tool vendors.
I then expanded it into Java to create a ‘proper’ API with an Open API spec, and Swagger UI.
I ‘had’ to create a proper API because Mockoon was the first place I noticed that the content was stripped out of the 204 I don’t know if Mockoon was doing it, or the http server library that Mockoon was using.
The HTTP Infrastructure seem to try to protect itself.
What Clients Fail on Which Requests?
I don’t fully know yet.
Hopefully the REST Client tool vendors will run the API locally and try against their tooling and fix any issues.
If they do this then it would be nice if they could credit the API for helping.
I have started using the full 70+ endpoints, running locally to reduce HTTP Infrastructure interference on a few tools and I have noticed issues with every single one:
- tools showing strange error messages,
- tools hiding errors in one view which are visible in another,
- tools swallowing the error messages so you really have to know there is a problem in order to spot it.
Every. Single. Tool.
Big names. Small names. Commercial and open-source.
Every. Single. One.
This certainly helps me understand the limitations of the REST API tools I’m using.
I recommend running your favourite REST Tool, through an HTTP proxy against the API running locally - that way you are most likely to have all the error conditions, and you can check your REST Client against the Proxy results.
For a proxy I use:
My favourite is the 204 with body.
Where is The API?
The production version of the API From Hell can be found here:
And can be run from source here:
What’s next
I found the whole “HTTP Infrastructure protects itself” concept very interesting so I’m doing to convert the API From Hell to work on multiple tools e.g. different mocking technologies and with multiple languages and frameworks.
My API From Hell is driven from a JSON configuration so it will be fairly easy to create multiple implementation wrappers. Particularly with AI doing the implementing.
I’m interested to see what technologies ‘hide’ errors, which ones make it much harder to make mistakes.
It might be a little odd to try and find out which technology let’s you make mistakes.
But it is very important to know which technologies prevent you from finding mistakes.