Hurl 5.0.0, the Parallel Edition

Aug. 29, 2024

Hurl 5.0.0, the parallel edition

The Hurl team is thrilled to announce Hurl 5.0.0 Partying face emoji !

Hurl is a command line tool powered by curl, that runs HTTP requests defined in a simple plain text format:

GET https://example.org/api/tests/4567
HTTP 200
[Asserts]
header "x-foo" contains "bar"
certificate "Expire-Date" daysAfterNow > 15
jsonpath "$.status" == "RUNNING"    # Check the status code
jsonpath "$.tests" count == 25      # Check the number of items
jsonpath "$.id" matches /\d{4}/     # Check the format of the id

What’s New in This Release

Run Tests in Parallel

By default, Hurl runs requests as an HTTP client, outputting response bodies. For instance, this file:

POST https://example.com/login
[FormParams]
user: Bob
password: secret
HTTP 302

GET https://example.com/protected
Content-Type: text/plain; charset=UTF-8
HTTP 200

When executed, Hurl outputs the response body behind protected path:

$ hurl protected.hurl
Hello World!

Another usage of Hurl is to run tests against APIs, HTML content etc... With --test, Hurl displays a nice test oriented report:

$ hurl --test *.hurl
e.hurl: Success (1 request(s) in 191 ms)
a.hurl: Success (1 request(s) in 190 ms)
d.hurl: Success (1 request(s) in 196 ms)
b.hurl: Success (1 request(s) in 197 ms)
error: Assert status code
  --> c.hurl:2:6
   |
   | GET https://hurl.dev
 2 | HTTP 301
   |      ^^^ actual value is <200>
   |

c.hurl: Failure (1 request(s) in 198 ms)
--------------------------------------------------------------------------------
Executed files:    5
Executed requests: 5 (25.0/s)
Succeeded files:   4 (80.0%)
Failed files:      1 (20.0%)
Duration:          200 ms

Before 5.0.0, each Hurl file was executed sequentially, one-by-one. Starting with 5.0.0, tests are now, by default, run in parallel, allowing blazingly fast execution!

To develop this feature, we take a lot of inspiration from the venerable GNU Parallel.

In parallel mode, each Hurl file is executed on its own thread, sharing nothing with other jobs. There is a thread pool which size is roughly the current amount of CPUs and that can be configured with --jobs option. During parallel execution, standard output and error are buffered for each file and only displayed on screen when a job is finished. This way, debug logs and messages are never interleaved between execution. Order of execution is not guaranteed in --parallel mode but reports (HTML, TAP, JSON and JUnit) keep the input files order.

The parallelism used is multithread sync: the thread pool is instantiated for the whole run, each Hurl file is run in its own thread, synchronously. We’ve not gone through the full multithreaded async route for implementation simplicity. Moreover, there is no additional dependency, only the standard Rust lib with “classic” threads and multiple producers / single consumer channels to communicate between threads.

What’s also exciting is that we have also introduced in 5.0.0 a --repeat option, that repeats the input file a given number of times. Let’s say we’ve a stress test stress.hurl. Now we can play it 1000 times, with a pool of 16 jobs with a single command:

$ hurl --test --jobs 16 --repeat 1000 stress.hurl

For the anecdote, in our day job, we’ve found a regression in a Java getRandom method (part of Apache Common library) using these options. With a very simple Hurl file and a single command, we were able to reliably reproduce performances issues with concurrent requests. How convenient!

A last word about this new exciting feature: tests may need to be executed one-by-one for various reason. In this case, just limit the number of jobs to one (--jobs 1), and the tests will be run sequentially, as before 5.0.0

Hurl is quite famous for being fast, wait to see how incredible it is now!

Better Error Display

Starting with 5.0.0, we’ve begun to improve error rendering. We started to work on multiline body assertions to set the foundation for better error diagnostics with all type of asserts.

Let’s say we’ve an endpoint which returns a CSV file. Using multiline body assertions, our Hurl test is:

GET http://localhost:8000/deniro.csv
HTTP 200
```
"Year", "Score", "Title"
1968,  86, "Greetings"
1970,  17, "Bloody Mama"
1970,  73, "Hi, Mom!"
1971,  40, "Born to Win"
1973,  98, "Mean Streets"
1973,  88, "Bang the Drum Slowly"
1974,  97, "The Godfather, Part II"
1976,  41, "The Last Tycoon"
1976,  99, "Taxi Driver"
1977,  47, "1900"
1978,  67, "New York,New York"
1978,  93, "The Deer Hunter"
1980,  97, "Raging Bull"
1981,  75, "True Confessions"
1983,  90, "The King of Comedy"
1984,  90, "Once Upon a Time"
1984,  60, "Falling in Love"
1985,  98, "Brazil"
1986,  65, "The Mission"
```

Before 5.0.0, if we have differences between the expected body response and the real response, Hurl output was:

$ hurl deniro.hurl
error: Assert body value
  --> test.hurl:4:1
   |
   | GET http://localhost:8000/deniro.csv
   | ...
 4 | "Year", "Score", "Title"
   | ^ actual value is <"Year", "Score", "Title"
   |   1968,  86, "Greetings"
   |   1970,  17, "Bloody Mama"
   |   1970,  73, "Hi, Mom!"
   |   1971,  40, "Born to Win"
   |   1973,  98, "Mean Streets"
   |   1973,  88, "Bang the Drum Slowly"
   |   1974,  97, "The Godfather, Part II"
   |   1976,  41, "The Last Tycoon"
   |   1976,  99, "Taxi Driver"
   |   1977,  47, "1900"
   |   1977,  67, "New York, New York"
   |   1978,  93, "The Deer Hunter"
   |   1980,  97, "Raging Bull"
   |   1981,  75, "True Confessions"
   |   1983,  90, "The King of Comedy"
   |   1984,  89, "Once Upon a Time in America"
   |   1984,  60, "Falling in Love"
   |   1985,  98, "Brazil"
   |   1986,  65, "The Mission"
   |
   |   >
   |

The diagnostic was rather basic: we indicated the start of the expected body line, and dumped the whole actual body response.

With Hurl 5.0.0, we’ve begun to use diffing algorithms to improve the error rendering:

error: Assert body value
  --> test.hurl:15:1
   |
   | GET http://localhost:8000/deniro.csv
   | ...
15 | 1978,  67, "New York,New York"
   |   -1978,  67, "New York,New York"
   |   +1977,  67, "New York, New York"
   |

Now, we point to the first line where a difference occurs. It’s more focused, with less noise and we now have a base to improve (display all the differences instead of just the first for instance).

Under the hood, we’re using the excellent zero-dependency Similar Rust crate from Armin Ronacher. We’re very reluctant to add new dependencies to Hurl, and we were very happy to find such a high quality library without additional costs.

JSON Report

Along side HTML, TAP, JUnit, Hurl 5.0.0 introduces a brand-new type of report: the JSON report! With --report-json, you can export a whole Hurl test session in a structured data-oriented report. Every HTTP response headers, bodies is saved and accessible for analyse.

$ hurl --test --report-json build/report integration/*.hurl

Very convenient, and also different reports can be combined:

$ hurl --test \
  --report-json build/report/json \
  --report-html build/report/html \
  integration/*.hurl

Directories as Input

A little quality of life improvement: Hurl can use directories as input and recursively looks for .hurl files.

Instead of:

$ hurl --test integration/*.hurl

You can write:

$ hurl --test integration/

Or even

$ hurl --test .

And we take care of running your Hurl files!

Time Units

Another small but nifty improvements: time units!

When it makes sens, you can specify time units, either in [Options] section or in command line options. For instance, if we want to retry this request on error, with 2 seconds spaced retried, we can write:

GET https://foo.com/api
[Options]
retry: 10               # maximum number of retries
retry-interval: 2s      # retries are 2 seconds spaced
HTTP 200
[Asserts]
jsonpath "$.state" == "ok"

Using the new [--repeat] option, we can adjust our benchmarks by introducing a slight delay between requests with --delay 15ms:

$ hurl --test --repeat 500 --delay 15ms benchmark.hurl

Or specify a 30 seconds timeout:

$ hurl --max-time 30s foo.hurl

Others

There are a lot other improvements with Hurl 5.0.0 and also a lot of bug fixes, you can check the complete list of enhancements and bug fixes in our release note.

If you like Hurl, don’t hesitate to give us a star on GitHub or share it on Twitter!

We’ll be happy to hear from you, either for enhancement requests or for sharing your success story using Hurl!

 RSS feed