# hurl-orchestra

> AI first API automation testing: write each step of an API test as a small text file, and run them in the right order.

---

Source: https://www.estacouveflor.com/hurl-orchestra/guide/getting-started

# Install and run

This tutorial makes a suite of two Hurl files. The first file sends a login request and captures a token. The second file uses the token. At the end, you run the suite and read the report.

## Before you start

You need:

- Python 3.11 or a later version.
- [Hurl](https://hurl.dev/docs/installation.html) on your `PATH`.

## 1. Install hurl-orchestra

```bash
pip install hurl-orchestra
hurl --version
```

The second command must show the Hurl version. If it shows an error, install Hurl first.

## 2. Write the first file

Make a directory with the name `tests`. Put this file in it with the name `auth.hurl`:

```hurl
---
id: auth
outputs: [token]
---
POST {{base_url}}/login
{
  "user": "ada",
  "password": "{{password}}"
}
HTTP 200
[Captures]
token: jsonpath "$.token"
```

The frontmatter between the two `---` lines is YAML. The `id` is the name of the node. The `outputs` list tells hurl-orchestra which captures to share.

## 3. Write the second file

Put this file in the same directory with the name `profile.hurl`:

```hurl
---
id: profile
deps: [auth]
---
GET {{base_url}}/profile
Authorization: Bearer {{auth_token}}
HTTP 200
[Asserts]
jsonpath "$.user" == "ada"
```

The `deps` list tells hurl-orchestra to run `auth` first. The file gets the `token` capture of `auth` with the name `auth_token`.

## 4. Give the global variables

Put a `.env` file in the `tests` directory. When you run the directory, hurl-orchestra gives this file to each Hurl call:

```properties
base_url=https://staging.example.com
password=use-a-secret-store
```

::: warning
Do not commit a `.env` file with real secrets. In CI, write the file from your secret store before the run.
:::

## 5. Examine the plan

```bash
hurl-orchestra --dry-run ./tests
```

```text
Plan: 2 node(s)
  wave 1: auth
  wave 2: profile
```

The dry run does not call Hurl. Use it after each change to the frontmatter.

## 6. Run the suite

```bash
hurl-orchestra ./tests
```

```text
Variables file: tests/.env
SUCCESS: auth [captured: token]
SUCCESS: profile [injected: auth_token]
Report saved to tests/report.zip
```

The command stops with exit code 0 when all nodes pass. It stops with exit code 1 when a node fails.

## Next steps

- Learn how the waves operate in [The graph](/concepts/graph).
- Let a coding agent write the next files. Read [Work with AI agents](/how-to/work-with-ai-agents).
- Show the results in a pull request. Read [Report in GitHub Actions](/how-to/report-in-github-actions).

---

Source: https://www.estacouveflor.com/hurl-orchestra/how-to/work-with-ai-agents

# Work with AI agents

hurl-orchestra is AI first API automation testing. It has no model inside. Its files, its commands and its results are easy for a coding agent to write, check and read. Use this procedure to let an agent such as Claude Code or Codex write and repair your API tests.

## Why agents write good suites with it

| Property | What the agent gets |
|---|---|
| Plain-text Hurl files | A request and its assertions in one small file, with no code to compile. |
| Frontmatter with a few fields | The agent declares the order in `deps`. It does not write a script for it. |
| `--dry-run --json` | A plan that the agent can parse: the waves, the files and the variable names of each node. |
| One-line errors | Each error names the node and the field, for example `ERROR: deps for 'foo' must be a list`. |
| CTRF results | A JSON file with one test for each request and the reason for each failure. |
| `known_failures` with `reason` and `until` | A tolerance that a human reviews. An agent cannot hide a failure without a trace. |
| `llms.txt` | This documentation as plain text for the context of the agent. |

## Procedure

1. Copy these rules into the `AGENTS.md` or `CLAUDE.md` file of your repository:

   ```md
   ## API tests

   The API tests are Hurl files in `tests/`. hurl-orchestra runs them as a dependency graph.
   Documentation: https://www.estacouveflor.com/hurl-orchestra/llms-full.txt

   Rules:

   - Write one scenario in each `.hurl` file. Put the frontmatter between two `---` lines.
   - Declare each dependency in `deps`. Do not copy a login request into a file.
   - Declare each shared capture in `outputs`. Use it as `{producer id}_{output name}`.
   - Put URLs and secrets in variables. Do not write a secret in a file.
   - Do not add `known_failures` to hide a failure. Ask a human first.

   Procedure after each change:

   1. Run `hurl-orchestra --dry-run --json tests`. If `ok` is `false`, correct the error.
   2. Run `hurl-orchestra tests --report-ctrf results.json`.
   3. Read `tests/results.json`. Each failed test has the assertion and the actual value in `message`.
   4. Correct the file or report the defect in the API. Then do step 1 again.
   ```

2. Give the agent a task, for example: "Add a test that a customer cannot read the order of a different customer."
3. Let the agent run the dry run. The JSON plan shows the variables that each file can use:

   ```bash
   hurl-orchestra --dry-run --json tests
   ```

4. Let the agent run the suite and read `tests/results.json`.
5. Review the new files and the plan in the pull request, as you review code.

## Give the documentation to the agent

The site publishes two plain-text files:

- [llms.txt](/llms.txt) is a list of all pages with a short description.
- [llms-full.txt](/llms-full.txt) is the full documentation in one file, with the samples.

Give the URL of `llms-full.txt` to the agent, or copy the file into the context.

## Result

The agent writes each scenario as a separate file and declares its dependencies. It checks the graph before it sends a request. When a test fails, the CTRF message tells the agent the assertion and the actual value.

---

Source: https://www.estacouveflor.com/hurl-orchestra/concepts/graph

# The graph

hurl-orchestra reads the frontmatter of each `.hurl` file. It then makes a directed acyclic graph (DAG). Each file is a node, and each entry in `deps` is an edge.

## Nodes

The ID of a node is the `id` field. If a file has no `id`, the ID is the file name without `.hurl`. An ID must not contain spaces, because hurl-orchestra uses it in variable names.

A file without frontmatter is also a node. It has no dependencies and no outputs.

## Waves

A wave is a set of nodes that have all of their dependencies done. hurl-orchestra runs the nodes of one wave in parallel. It starts the next wave when the current wave is complete.

| Wave | Nodes | Why |
|---|---|---|
| 1 | `auth`, `catalog` | These nodes have no `deps`. |
| 2 | `create_cart` | It needs `auth`. |
| 3 | `add_item` | It needs `auth` and `create_cart`. |
| 4 | `checkout` | It needs `add_item`. |

The maximum number of parallel Hurl processes is the number of CPUs plus 4, with a limit of 32.

## Priority in a wave

The `priority` field sorts the nodes in a wave. hurl-orchestra runs each priority group in sequence, from the highest value to the lowest value. The nodes in one group run in parallel. For a procedure, read [Order the nodes in a wave](/how-to/order-a-wave).

Priority never overrides `deps`. A node with a high priority still waits for its dependencies.

## Validation

hurl-orchestra validates the full graph before it runs a node. The run stops with an error if:

- A dependency has no node with that ID.
- The `deps` make a cycle, for example `a` needs `b` and `b` needs `a`.
- A frontmatter field has an incorrect type.

Thus a broken graph never runs half of a suite.

## Discovery

When you give a directory, hurl-orchestra loads all `.hurl` files in that directory. It does not go into subdirectories.

When you give one or more files, hurl-orchestra loads all `.hurl` files in the same directories. It then runs the files that you gave and their dependencies. For a procedure, read [Run specific files](/how-to/run-specific-files).

---

Source: https://www.estacouveflor.com/hurl-orchestra/concepts/variables

# Variables

A node can give values to the nodes that need it. You declare the values in `outputs`, and you capture them in the `[Captures]` section of the Hurl file.

## Names

A dependent node gets each output with the ID of the producer as a prefix:

```text
{producer id}_{output name}
```

For example, the output `token` of the node `auth` becomes `auth_token`. Two nodes can capture a `token`, and the names do not collide.

If an ID or an output name contains a character other than a letter, a digit or `_`, hurl-orchestra changes the character to `_xx_`. The `xx` is the hexadecimal code of the character. For example, `auth-v2` becomes `auth_2d_v2`.

## Only direct dependencies

A node gets the outputs of the nodes in its own `deps` list. It does not get the outputs of the dependencies of those nodes. Add each producer that you need to `deps`. This rule keeps the source of each variable clear.

## An output without a capture

If a node declares an output and the Hurl report does not contain that capture, the node fails:

```text
FAILED: auth
Missing expected outputs: token; reported outputs: none
```

## How the values go to Hurl

hurl-orchestra writes the values to a private variables file for each call. Then it gives the file to Hurl with `--variables-file`. Thus a token does not show in the process list.

## Global variables

A variables file gives values to all nodes, for example `base_url`. hurl-orchestra gives the file to each Hurl call with `--variables-file`. It selects one file:

| Command | Variables file |
|---|---|
| `hurl-orchestra ./tests` | `tests/.env` |
| `hurl-orchestra` | `.env` in the current directory |
| `hurl-orchestra tests/profile.hurl` | `.env` in the current directory |
| `hurl-orchestra --env-file ci.env ./tests` | `ci.env` |

The run shows the file that it uses on the first line:

```text
Variables file: tests/.env
```

When you give files, hurl-orchestra does not read a `.env` file next to the files. The run tells you, and it shows the fix:

```text
NOTE: hurl-orchestra does not read tests/.env when you give files. It reads .env from the current directory. Add --env-file tests/.env to read it.
```

If the file of `--env-file` does not exist, the run stops before it calls Hurl.

## Variables from the command line

Give Hurl flags after `--`. hurl-orchestra gives them to each Hurl call:

```bash
hurl-orchestra ./tests -- --variable region=eu
```

For more information, read [Give flags to Hurl](/how-to/pass-hurl-flags).

---

Source: https://www.estacouveflor.com/hurl-orchestra/concepts/aliases

# Aliases

An alias runs one Hurl file again as a different node. Each alias has its own ID, so each alias has its own outputs.

## Syntax

In `deps`, write the template ID, a colon and the new ID:

```hurl
---
id: isolation
deps:
  - sign_up: alice
  - sign_up: bob
---
GET {{base_url}}/users/{{alice_user_id}}
Authorization: Bearer {{bob_token}}
HTTP 403
```

The template is the file with the ID `sign_up`:

```hurl
---
id: sign_up
outputs: [token, user_id]
---
POST {{base_url}}/users
{ "email": "{{newUuid}}@example.com" }
HTTP 201
[Captures]
token: jsonpath "$.token"
user_id: jsonpath "$.id"
```

hurl-orchestra runs `sign_up.hurl` two times, as `alice` and as `bob`. The `isolation` node gets `alice_token`, `alice_user_id`, `bob_token` and `bob_user_id`.

## Inputs

Each alias runs the same file with the same variables. To get two different results, use a value that changes on each call, for example the Hurl function `{{newUuid}}`. In the example, each alias makes a new user.

## The template node

The template file is also a node with its own ID. When you run a directory, the template runs one time as itself. The aliases also run.

```bash
hurl-orchestra --dry-run ./tests
```

```text
Plan: 4 node(s)
  wave 1: alice, bob, sign_up
  wave 2: isolation
```

When you run specific files, hurl-orchestra adds only the aliases that the files need.

## Dependencies of a template

If the template has `deps`, each alias gets the same `deps`. If the template `deps` contain aliases, hurl-orchestra makes those aliases too.

---

Source: https://www.estacouveflor.com/hurl-orchestra/concepts/failures

# Failures and skips

A node has one result. The result controls the exit code of the run and the nodes that come after it.

| Result | Console | Run fails | Dependents |
|---|---|---|---|
| Pass | `SUCCESS` | No | Run. |
| Fail | `FAILED` | Yes | Skip with `failed dependency`. |
| Known failure | `KNOWN FAILURE` | No | Skip with `known failure upstream`. |
| Skip | `SKIPPED` | Same as the upstream node | Skip. |

## A failure stops only its branch

When a node fails, hurl-orchestra skips all nodes that need it. The other nodes continue. Thus one run shows all independent failures, not only the first failure.

Try it on the score below. Select `create_cart`, then select it again for a known failure.

## Known failures

A known failure is a failure that you expect and that you documented. You declare its signature in `known_failures`. When a failure matches the signature, the run stays green. The report still shows the failure.

Each entry has a `name`, a `reason` and an optional `until` date. After the `until` date, the entry stops to match, and the failure fails the run again. For the procedure, read [Tolerate a known failure](/how-to/tolerate-a-known-failure).

The `--strict` flag makes each known failure fail the run.

## Retries

A `retry` policy runs a failed node again before hurl-orchestra decides the result. The node gets its result from the last attempt. For the procedure, read [Retry on back pressure](/how-to/retry-on-back-pressure).

hurl-orchestra applies the retries first. Then it compares the last failure with `known_failures`.

## Time limit

Each Hurl call has a limit of 300 seconds. After the limit, hurl-orchestra stops the call, and the node fails with `Hurl timed out after 300 seconds`.

---

Source: https://www.estacouveflor.com/hurl-orchestra/concepts/reports

# Reports

Each run writes a zip file. You can also write a CTRF report for CI.

## The zip file

The zip file contains the Hurl JSON report of each node that ran. Each node has a directory with its ID as the name:

```text
report.zip
├── auth/
│   ├── report.json
│   └── store/
└── create_user/
    ├── report.json
    └── store/
```

hurl-orchestra writes the zip file after each run, also when a node fails. Thus you keep the evidence of the failure.

| Command | Report |
|---|---|
| `hurl-orchestra ./tests` | `tests/report.zip` |
| `hurl-orchestra` | `report.zip` in the current directory |
| `hurl-orchestra ./tests --report-zip ci.zip` | `tests/ci.zip` |

A relative path goes into the directory that you gave. An absolute path stays the same.

## The CTRF report

[CTRF](https://ctrf.io) is a JSON format for test results. Add `--report-ctrf` to write it:

```bash
hurl-orchestra ./tests --report-ctrf results.json
```

Each Hurl entry becomes one test. Skipped nodes and known failures also show in the report. For the full rules, read the [CTRF reference](/reference/ctrf).

## The execution plan as JSON

A dry run with `--json` writes the plan as JSON. Scripts and AI agents can read the waves, the dependencies and the variables of each node:

```bash
hurl-orchestra --dry-run --json ./tests
```

For the fields, read [Command line](/reference/cli#json-plan).

---

Source: https://www.estacouveflor.com/hurl-orchestra/how-to/run-specific-files

# Run specific files

Use this procedure to run one file and the files that it needs. The other files in the directory do not run.

## Procedure

1. Give the path of the file to hurl-orchestra:

   ```bash
   hurl-orchestra tests/checkout.hurl
   ```

2. Read the first line of the output. It shows the dependencies that hurl-orchestra added:

   ```text
   Resolved dependencies: add_item, auth, create_cart
   ```

3. To run more than one file, give all paths:

   ```bash
   hurl-orchestra tests/checkout.hurl tests/catalog.hurl
   ```

## Result

hurl-orchestra loads all `.hurl` files in the directory of each file that you gave. It runs the files that you gave and all of their dependencies. It does not run the other files.

## Run only the listed files

Add `--no-deps` to stop the resolution. Then give each dependency on the command line:

```bash
hurl-orchestra --no-deps tests/auth.hurl tests/profile.hurl
```

If a dependency is not on the command line, the run stops before it calls Hurl.

## Read the .env file next to the files

When you give files, hurl-orchestra reads `.env` from the current directory, not from the directory of the files. To read a different file, add `--env-file`:

```bash
hurl-orchestra --env-file tests/.env tests/checkout.hurl
```

If you forget it, the run shows a `NOTE` line with this command. hurl-orchestra also writes `report.zip` to the current directory.

---

Source: https://www.estacouveflor.com/hurl-orchestra/how-to/preview-the-plan

# Preview the plan

Use this procedure to examine the order of the nodes before a run. A dry run does not call Hurl, so you do not need Hurl on the `PATH`.

## Procedure

1. Add `--dry-run` to the command:

   ```bash
   hurl-orchestra --dry-run tests/checkout.hurl
   ```

2. Read the waves:

   ```text
   Resolved dependencies: add_item, auth, create_cart
   Plan: 4 node(s)
     wave 1: auth
     wave 2: create_cart
     wave 3: add_item
     wave 4: checkout
   ```

3. If the command shows an error, correct the frontmatter. Then do step 1 again.

## Get the plan as JSON

Add `--json` for a result that a script or an AI agent can read:

```bash
hurl-orchestra --dry-run --json tests/checkout.hurl
```

The JSON contains the waves and, for each node, the file, the dependencies and the variables that the node gets. If the graph has an error, the JSON contains `"ok": false` and the message. For the fields, read [Command line](/reference/cli#json-plan).

## Use the dry run in CI

A dry run is fast and needs no network. Add it as the first step of a CI job. Then a broken graph fails the job before a request goes to the server.

---

Source: https://www.estacouveflor.com/hurl-orchestra/how-to/pass-hurl-flags

# Give flags to Hurl

Use this procedure to give Hurl options to all nodes or to one node.

## Give flags to all nodes

1. Put the flags after a `--` separator:

   ```bash
   hurl-orchestra ./tests -- --variable host=localhost --retry 3
   ```

2. Put the hurl-orchestra options before the `--` separator:

   ```bash
   hurl-orchestra --report-zip ci.zip ./tests -- --connect-timeout 10
   ```

hurl-orchestra gives all words after `--` to each Hurl call, with no change.

A flag without a value can also go before `--`, for example `--verbose`. A flag with a value must go after `--`.

## Give flags to one node

1. Add an `args` list to the frontmatter of the file:

   ```hurl
   ---
   id: slow_report
   args:
     - verbose
     - connect-timeout: 30
     - variable: region=eu
   ---
   GET {{base_url}}/reports/{{region}}
   HTTP 200
   ```

2. Write each flag without the dashes. hurl-orchestra adds `-` to a name with one letter and `--` to a longer name.
3. For a flag with a value, write a map with one key.

## Result

hurl-orchestra adds the `args` of the node after the flags from the command line. If the two places set the same flag, Hurl uses the last value. Thus the value of the node wins.

---

Source: https://www.estacouveflor.com/hurl-orchestra/how-to/order-a-wave

# Order the nodes in a wave

Use this procedure when independent nodes must run in a specified sequence. For example, a `delete` must not run before a `search` that needs the data.

## Procedure

1. Give a positive `priority` to the node that must run first:

   ```hurl
   ---
   id: create
   priority: 1
   ---
   POST {{base_url}}/notes
   { "title": "score" }
   HTTP 201
   ```

2. Give no `priority` to the node in the middle. The default value is `0`.

   ```hurl
   ---
   id: search
   ---
   GET {{base_url}}/notes?q=score
   HTTP 200
   [Asserts]
   jsonpath "$.items" count > 0
   ```

3. Give a negative `priority` to the node that must run last:

   ```hurl
   ---
   id: delete
   priority: -1
   ---
   DELETE {{base_url}}/notes/score
   HTTP 204
   ```

4. Examine the plan:

   ```bash
   hurl-orchestra --dry-run ./tests
   ```

## Result

The three nodes are in the same wave. hurl-orchestra runs the priority groups in sequence: `create`, then `search`, then `delete`.

::: tip
Use `priority` only for nodes that do not share data. If a node needs the output of a different node, use `deps`.
:::

---

Source: https://www.estacouveflor.com/hurl-orchestra/how-to/retry-on-back-pressure

# Retry on back pressure

Use this procedure when a server can refuse requests under load, for example with `429 Too Many Requests` or `503 Service Unavailable`.

## Procedure

1. Add a `retry` policy to the frontmatter of the node:

   ```hurl
   ---
   id: create_order
   retry:
     attempts: 4
     backoff_ms: 500
     max_backoff_ms: 20000
     multiplier: 2
     jitter: full
     when:
       - status: 429
       - status: 503
       - stderr: 'Connection refused'
   ---
   POST {{base_url}}/orders
   HTTP 201
   ```

2. In `when`, list only the failures that show back pressure. Without `when`, hurl-orchestra retries each failure, also a failed assertion.
3. Use `stderr` to match a connection error. It is the only field that can match when no response arrived.
4. Set `max_backoff_ms` to the longest wait that your CI accepts.

## Result

After a failure that matches `when`, hurl-orchestra waits and runs the full node again. The wait after attempt *n* is:

```text
min(max_backoff_ms, backoff_ms × multiplier^(n − 1))
```

With `jitter: full`, the wait is a random value from 0 to that limit. Thus nodes that fail together do not retry together.

If the response has a `Retry-After` header, hurl-orchestra uses it. It accepts seconds and HTTP dates. `max_backoff_ms` still limits the value.

The console shows each attempt, and the result line shows the number of attempts:

```text
RETRY: create_order attempt 1/4 failed (status 503), waiting 412ms
SUCCESS: create_order [2 attempts]
```

::: warning
The Hurl option `[Options] retry` also retries a request. If you use the two options together, the numbers of attempts multiply.
:::

For all fields, read [Frontmatter](/reference/frontmatter#retry).

---

Source: https://www.estacouveflor.com/hurl-orchestra/how-to/tolerate-a-known-failure

# Tolerate a known failure

Use this procedure when a node fails for a cause that you know and cannot correct now. For example, a shared test database can reach its connection limit.

## Procedure

1. Find the signature of the failure in the Hurl report: the status, a header, the body or the error text.
2. Add an entry to `known_failures`:

   ```hurl
   ---
   id: create_order
   outputs: [order_id]
   known_failures:
     - name: db_pool_exhausted
       reason: The shared test database is at its connection limit
       until: 2026-12-31
       link: https://example.com/runbooks/test-db-pool
       status: 500
       header:
         name: X-Error-Code
         pattern: '^ERR-DB-\d+$'
   ---
   POST {{base_url}}/orders
   HTTP 200
   [Captures]
   order_id: jsonpath "$.id"
   ```

3. Write a `name`. The console, the summary and the CTRF `tags` show it.
4. Write a `reason` that a different engineer can understand.
5. Set `until` to the date of the next review.
6. Add `link` to the ticket or the runbook.
7. Run the suite with `--dry-run` to validate the entry.

## Result

When the node fails and each signature field that you set matches, the console shows:

```text
KNOWN FAILURE: create_order [db_pool_exhausted] The shared test database is at its connection limit
```

The run stays green. The nodes that need `create_order` get no outputs, so hurl-orchestra skips them. The run ends with a summary:

```text
Known failures tolerated: 1 (db_pool_exhausted: create_order)
```

After the `until` date, the entry stops to match, and the failure fails the run with `known failure 'db_pool_exhausted' expired on 2026-12-31`.

## Make the known failures fail

Run with `--strict` to see if the cause still exists:

```bash
hurl-orchestra --strict ./tests
```

For all fields, read [Frontmatter](/reference/frontmatter#known-failures).

---

Source: https://www.estacouveflor.com/hurl-orchestra/how-to/report-in-github-actions

# Report in GitHub Actions

Use this procedure to show each Hurl request as a test in the summary of a GitHub Actions job.

## Procedure

1. Add a workflow file, for example `.github/workflows/api-tests.yml`:

   ```yaml
   name: API tests

   on: [push, pull_request]

   jobs:
     api:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v5
         - uses: gacts/install-hurl@v1
         - uses: actions/setup-python@v6
           with:
             python-version: "3.12"
         - run: pip install hurl-orchestra
         - name: Run the tests
           run: hurl-orchestra ./tests --report-ctrf results.json
         - name: Publish the test report
           if: always()
           uses: ctrf-io/github-test-reporter@v1
           with:
             report-path: tests/results.json
           env:
             GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
   ```

2. Keep `if: always()` on the report step. Without it, the step does not run after a failure.
3. Set `report-path` to the directory that you gave to hurl-orchestra and the file name. A relative `--report-ctrf` path goes into that directory.
4. Push the change and open the summary of the workflow run.

## Result

The [GitHub test reporter](https://github.com/ctrf-io/github-test-reporter) shows the passed, failed and skipped tests. A known failure shows as a flaky test with its name as a tag.

## Keep the Hurl reports

To keep the zip file, add a step after the tests:

```yaml
- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: hurl-reports
    path: tests/report.zip
```

---

Source: https://www.estacouveflor.com/hurl-orchestra/how-to/draw-the-graph

# Draw the graph

Use this procedure to make a Mermaid diagram of the dependency graph. GitHub shows Mermaid diagrams in Markdown files.

## Procedure

1. Add `--diagram` to the command:

   ```bash
   hurl-orchestra --diagram ./tests
   ```

2. Open `diagram.md`. The diagram shows each node with its number of outputs, and its priority if the priority is not 0.
3. To write to a different file, add `--diagram-output`:

   ```bash
   hurl-orchestra --diagram ./tests --diagram-output docs/api-graph.md
   ```

4. To replace a file, add `--diagram-overwrite`. Without it, hurl-orchestra does not change a file that exists.

## Draw the graph of one file

Give the file instead of the directory. The diagram contains the file and its dependencies:

```bash
hurl-orchestra --diagram tests/checkout.hurl --diagram-output -
```

The `-` value writes the diagram to stdout.

## Result

```mermaid
flowchart LR
    node_1["auth [out:1]"]
    node_4["create_cart [out:1]"]
    node_1 --> node_4
```

---

Source: https://www.estacouveflor.com/hurl-orchestra/reference/

# Reference

The reference gives the names, types and default values of each option and field.

## Parts of the reference

- [Command line](/reference/cli) shows each option of `hurl-orchestra` and the JSON plan.
- [Frontmatter](/reference/frontmatter) shows each field that a `.hurl` file can declare.
- [Console output](/reference/console-output) shows each line that a run can write.
- [CTRF report](/reference/ctrf) shows how a run becomes CTRF tests.

## Requirements

| Item | Version |
|---|---|
| Python | 3.11 or later |
| Hurl | A version with `--report-json` and `--variables-file` on the `PATH` |
| System | Linux, macOS or Windows |

## Exit codes

| Code | Description |
|---|---|
| `0` | All nodes passed, or the failures matched `known_failures`. |
| `1` | A node failed, the graph is not valid, or Hurl is not on the `PATH`. |
| `2` | The command line is not valid. |

---

Source: https://www.estacouveflor.com/hurl-orchestra/reference/cli

# Command line

```text
hurl-orchestra [options] [paths ...] [-- hurl options ...]
```

## Options

| Option | Default | Description |
|---|---|---|
| `paths` | `.` | A directory with .hurl files, or one or more .hurl files. The default is the current directory. |
| `--report-zip` `FILE` | `report.zip` | Write the hurl reports of all nodes to this zip file. |
| `--report-ctrf` `FILE` | - | Also write a CTRF JSON report to this file. |
| `--env-file` `FILE` | - | Give this variables file to each hurl call. The default is .env in the given directory, or in the current directory for files. |
| `--no-deps` | - | Run only the listed .hurl files. Do not add their declared deps from the same directory. |
| `--dry-run` | - | Show the execution plan. Do not run hurl. |
| `--json` | - | With --dry-run, write the plan as JSON for scripts and AI agents. |
| `--strict` | - | Make a failure that matches known_failures fail the run. |
| `--diagram` | - | Write a Mermaid diagram of the graph. Do not run the tests. |
| `--diagram-output` `FILE` | `diagram.md` | Write the diagram to this file. Use '-' for stdout. |
| `--diagram-overwrite` | - | Replace the diagram file if it exists. |

A script makes this table from the parser of the command. Thus the table and `hurl-orchestra --help` always agree.

## Paths

| Paths | Mode |
|---|---|
| None | Directory mode on the current directory. |
| One path that does not end in `.hurl` | Directory mode on that directory. |
| One or more `.hurl` files | File mode. hurl-orchestra adds the dependencies from the same directories. |

## Hurl options

Put Hurl options after `--`. hurl-orchestra gives them to each Hurl call. For more information, read [Give flags to Hurl](/how-to/pass-hurl-flags).

hurl-orchestra always gives these options to Hurl: `--test`, `--report-json` and, if a variables file applies, `--variables-file`.

## JSON plan

`hurl-orchestra --dry-run --json` writes one JSON document to stdout:

```json
{
  "ok": true,
  "env_file": "tests/.env",
  "resolved": ["add_item", "auth", "create_cart"],
  "waves": [["auth"], ["create_cart"], ["add_item"], ["checkout"]],
  "nodes": [
    {
      "id": "add_item",
      "file": "tests/add_item.hurl",
      "deps": ["auth", "create_cart"],
      "outputs": [],
      "variables": ["auth_token", "create_cart_cart_id"],
      "priority": 0,
      "known_failures": [],
      "retry_attempts": 1
    }
  ]
}
```

| Field | Description |
|---|---|
| `ok` | `true` if the graph is valid. |
| `error` | The error message. It shows only when `ok` is `false`. |
| `env_file` | The variables file of the run, or `null`. |
| `resolved` | The nodes that file mode added as dependencies. |
| `waves` | The node IDs of each wave, in the sequence of execution. |
| `nodes[].file` | The file of the node. For an alias, it is the file of the template. |
| `nodes[].variables` | The variables that the node gets from its dependencies. |
| `nodes[].retry_attempts` | The value of `retry.attempts`. `1` means no retry. |

---

Source: https://www.estacouveflor.com/hurl-orchestra/reference/frontmatter

# Frontmatter

The frontmatter is a YAML block between two `---` lines at the start of a `.hurl` file. All fields are optional.

## Fields

| Field | Type | Default | Description |
|---|---|---|---|
| `id` | string | The file name without `.hurl` | The ID of the node. It must not contain spaces. |
| `deps` | list | `[]` | The IDs of the nodes that must pass first. An item `template: alias` makes an [alias](/concepts/aliases). |
| `outputs` | list of strings | `[]` | The captures that the node gives to its dependents. |
| `priority` | integer | `0` | The sequence in a wave. A higher value runs first. |
| `args` | list | `[]` | Hurl flags for this node. |
| `known_failures` | list | `[]` | The failures that the node can tolerate. |
| `retry` | map | No retry | The retry policy of the node. |

## `args` {#args}

| Item | Hurl flag |
|---|---|
| `verbose` | `--verbose` |
| `v` | `-v` |
| `connect-timeout: 30` | `--connect-timeout 30` |
| `--insecure` | `--insecure`, with no change |

## `known_failures` {#known-failures}

Each entry needs `name`, `reason` and one or more signature fields. All signature fields that you set must match.

| Field | Type | Description |
|---|---|---|
| `name` | string | The name in the console, the summary and the CTRF `tags`. It must be unique in the node. |
| `reason` | string | Why the failure is acceptable. |
| `until` | date | After this date, the entry does not match. Optional. |
| `link` | string | Where the failure has documentation. Optional. |
| `status` | integer | The status code of the last response. |
| `header` | map | `name` and `pattern`. The name is not case-sensitive. The pattern is a regular expression. |
| `body` | string | A regular expression for the body of the last response. |
| `stderr` | string | A regular expression for the error output of Hurl. It can match when no response arrived. |

The last response is the last call that Hurl recorded for the node. With the Hurl option `[Options] retry`, it is the last attempt.

## `retry` {#retry}

| Field | Type | Default | Description |
|---|---|---|---|
| `attempts` | integer | `1` | The total number of attempts, with the first attempt. `1` disables the retry. |
| `backoff_ms` | integer | `1000` | The base wait in milliseconds. |
| `max_backoff_ms` | integer | `30000` | The maximum wait, also for `Retry-After`. |
| `multiplier` | number | `2` | The growth of the wait after each attempt. It must be 1 or more. |
| `jitter` | `full` or `none` | `full` | `full` selects a random wait from 0 to the limit. `none` waits the full limit. |
| `respect_retry_after` | boolean | `true` | Use the `Retry-After` header of the response. |
| `when` | map or list | Each failure | The signatures that start a retry. Each item takes `status`, `header`, `body` and `stderr`. |

hurl-orchestra validates all fields before the run. An unknown key in `retry` stops the run.

---

Source: https://www.estacouveflor.com/hurl-orchestra/reference/console-output

# Console output

Each node writes one result line when it is complete. A failure also writes the error output of Hurl.

## Result lines

| Line | Description |
|---|---|
| `SUCCESS: <id> [injected: … \| captured: … \| N attempts]` | The node passed. The brackets show the variables that it got and gave. |
| `FAILED: <id>` | The node failed. The error output of Hurl follows. |
| `KNOWN FAILURE: <id> [<name>] <reason>` | The failure matched a `known_failures` entry. The run stays green. |
| `SKIPPED: <id> (failed dependency: <ids>)` | A dependency failed. The node did not run. |
| `SKIPPED: <id> (known failure upstream: <ids>)` | A dependency was a known failure. The node did not run. |
| `RETRY: <id> attempt N/M failed (<status>), waiting <ms>ms` | A retry policy runs the node again. |

## Other lines

| Line | When |
|---|---|
| `Variables file: <path>` | A variables file applies to the run. |
| `NOTE: hurl-orchestra does not read <path> …` | In file mode, a `.env` file next to the files is not the variables file. |
| `Resolved dependencies: <ids>` | File mode added dependencies. |
| `Plan: N node(s)` and `wave N: …` | A dry run. |
| `Known failures tolerated: N (<name>: <ids>)` | The last line when a known failure occurred. |
| `CTRF report saved to <path>` | `--report-ctrf` wrote the report. |
| `Report saved to <path>` | The zip file is complete. |

## Error lines

A line that starts with `ERROR:` stops the run before a node runs. For the causes, read [Troubleshooting](/troubleshooting).

---

Source: https://www.estacouveflor.com/hurl-orchestra/reference/ctrf

# CTRF report

`--report-ctrf` writes a [CTRF](https://ctrf.io) JSON report. The [GitHub test reporter](https://github.com/ctrf-io/github-test-reporter) and other CTRF tools can read it.

## Tests

Each Hurl entry of a node becomes one test. An entry is one request with its assertions.

| Case | Test status | Message |
|---|---|---|
| All assertions of the entry pass | `passed` | None |
| An assertion fails | `failed` | The Hurl assertion on one line, for example `Assert status code: HTTP 200 (actual value is <500>)` |
| Hurl marks the file failed, and no assertion failed | `failed` | One test for the node |
| The Hurl process failed | `failed` | The error |
| A dependency failed | `skipped` | `failed dependency: <id>` |
| A known failure | `other`, with `flaky: true` | The matched signature. The entry name is in `tags`. |
| A dependency was a known failure | `skipped` | `known failure upstream: <id>` |

## Retries

With the Hurl option `[Options] retry`, Hurl writes one entry for each attempt, with the same index. The report keeps only the last attempt. Thus a request that failed two times and then passed is one passed test.

With the `retry` policy of hurl-orchestra, the report contains only the last attempt of the node.

## Summary

`summary.other` counts the known failures. They do not increase `summary.failed`.

---

Source: https://www.estacouveflor.com/hurl-orchestra/troubleshooting

# Troubleshooting

Find the error in the list. Then do the steps of the solution in sequence.

## `ERROR: 'hurl' not found on PATH`

hurl-orchestra calls the `hurl` command.

Solution:

1. Install [Hurl](https://hurl.dev/docs/installation.html).
2. Run `hurl --version` in the same shell.

A dry run does not need Hurl.

## `ERROR: deps for 'foo' must be a list`

The frontmatter of the file has an incorrect type. Similar errors are for `id`, `outputs`, `priority` and `args`.

Solution:

1. Open the file that the error names.
2. Correct the field. For the types, read [Frontmatter](/reference/frontmatter).
3. Run `hurl-orchestra --dry-run` to validate the change.

## `ERROR: 'foo' depends on 'bar' but no .hurl file or alias defines id: bar`

No node has the ID in `deps`.

Solution:

1. Make sure that a file has `id: bar`, or the name `bar.hurl`.
2. Put that file in the same directory as `foo`.
3. If you use `--no-deps`, give the file on the command line.

## `ERROR: alias template 'X' not found (used as 'Y')`

An alias names a template that the run did not load.

Solution:

1. Make sure that the template file is in the same directory.
2. Make sure that the template name is the ID of that file.

## `Circular dependency detected`

The `deps` make a cycle.

Solution:

1. Read the node IDs in the message.
2. Remove one of the `deps` in the cycle.

## `ERROR: env file not found`

The file of `--env-file` does not exist.

Solution:

1. Examine the path. A relative path starts at the current directory.

## Hurl cannot find a variable

Hurl shows an error for a variable without a value, for example `base_url`.

Solution:

1. Read the `Variables file` line at the start of the run. If it is not there, no variables file applies.
2. If you gave files, read the `NOTE` line. Add `--env-file` with the path that it shows.
3. For a variable from a dependency, make sure that the producer is in `deps` and the output is in its `outputs`.

## `FAILED: foo, Missing expected outputs`

The node declared an output, and the Hurl report has no capture with that name.

Solution:

1. Make sure that the `[Captures]` section defines each name in `outputs`.
2. Make sure that the response contains the value at that path.

## `SKIPPED: foo (failed dependency: bar)`

The node did not run because `bar` failed.

Solution:

1. Correct the failure of `bar`. The node runs again when its inputs exist.

## `FAILED: foo, Hurl timed out after 300 seconds`

One Hurl call took more than 5 minutes.

Solution:

1. Run the file with Hurl alone to find the slow request.
2. Add a timeout to the request, for example with `args` and `max-time`.

## `FAILED: foo, known failure 'name' expired on date`

The failure matched an entry, but its `until` date is in the past.

Solution:

1. Examine if the cause still exists.
2. If it exists, set a new `until` date and update the `reason`.
3. If it does not exist, remove the entry.

## `ERROR: known_failures[0] for 'foo' …`

An entry in `known_failures` is not valid.

Solution:

1. Make sure that the entry has a `name` and a `reason`.
2. Make sure that the entry has one or more signature fields.
3. Make sure that each regular expression is valid and each name is unique.

## `Diagram output already exists`

`--diagram` does not replace a file.

Solution:

1. Add `--diagram-overwrite`, or give a different `--diagram-output`.