# Concepts

Queues, tasks and attempts, what happens on each delivery, and how your endpoint checks that a request came from Workly. [Schedules](https://workly.run/docs/schedules/) and [task groups](https://workly.run/docs/groups/) build on the same pieces.

## Queues

A **queue** holds tasks and the policy for delivering them. It's created with the defaults below the first time you enqueue into it. Change its policy with `PUT /v1/queues/{queue}`, `workly queues put` or a [`workly.yaml`](https://workly.run/docs/deploy/).

| Setting | Default | |
|---|---|---|
| `max_attempts` | 5 | Total attempts before a task fails. |
| `min_backoff_seconds`, `max_backoff_seconds`, `backoff_multiplier` | 2, 3600, 2 | The wait between attempts (see [Retries](#retries)). |
| `timeout_seconds` | 30 | How long one attempt may take. |
| `max_concurrency` | 3 | Deliveries in flight at once, up to 10,000. |
| `rate_per_second` | none | Deliveries started per second. |
| `retry_on_4xx` | false | Retry 4xx responses other than 408 and 429, which are always retried. |
| `auto_pause` | none | `{after_failures: n}`: pause the queue by itself when its endpoint looks down (see [Auto-pause](#auto-pause)). |

The full list is in the [API reference](https://workly.run/docs/api/). Queue names are lowercase letters, digits, `-` and `_`; names starting with `_` are Workly's own.

A **paused** queue keeps accepting tasks but doesn't start deliveries; those already in flight finish. Workly shares deliveries fairly between queues, so a busy queue doesn't hold up a quiet one.

### Auto-pause

When an endpoint goes down, retries keep using up each task's attempts, and an outage longer than the retry window leaves every task `failed`, to retry by hand afterwards. A queue with `auto_pause` stops instead:

```yaml
queues:
  billing:
    auto_pause:
      after_failures: 20
```

- **What pauses it:** `after_failures` attempts in a row that end with a 5xx, a timeout or a network error. These say the endpoint itself is unwell. Other 4xx responses (usually about one task), 429 (the endpoint is pacing you, and `Retry-After` handles it) and lost attempts don't count, and only a success resets the count.
- **While it's paused:** no deliveries start, and tasks wait as `pending` without using attempts. One task at a time goes as a probe: 30 seconds after the pause, then after twice as long each time a probe fails, up to every 10 minutes. A probe is a normal attempt of the queue's oldest due task.
- **Resuming:** the first success resumes the queue on its own, and the waiting tasks are delivered within its rate and concurrency limits. Resuming it by hand (`POST /v1/queues/{queue}/resume`, `workly queues resume`, or **Resume now** in the dashboard) does it at once. Pausing it by hand stops the probes too.
- **Seeing it:** the queue's `auto_paused` says since when, the error, how many probes failed and when the next goes; it's absent when the queue isn't auto-paused. [Alerts](https://workly.run/docs/alerts/) go out when a queue pauses itself and when it resumes, and the `workly_queue_auto_paused` metric is 1 while it's paused.

Use it for a queue whose tasks go to one service. In a queue that delivers to many endpoints, such as webhooks to your customers' URLs, one of them being down would hold up the rest. Workly's own queues (`_workflows`, `_callbacks`) can't have it. A workflow step in an auto-paused queue waits like any task, and its run still fails after its `max_duration_seconds`.

## Tasks

A **task** is one HTTP request Workly owes your endpoint: a URL, a method (`POST` by default), headers, and a body, either `payload` sent as JSON or `body` with its own `content_type`. `delay_seconds` or `run_at` sets when it's first due. It can override its queue's `max_attempts` and `timeout_seconds`, and with `on_complete` call you back each time it finishes (see [Callbacks](https://workly.run/docs/callbacks/)).

A task is in one of these states:

| State | |
|---|---|
| `pending` | Waiting to be delivered: due now, waiting for its `run_at`, or waiting to be retried. |
| `running` | An attempt is in flight. |
| `succeeded` | An attempt got a 2xx response. |
| `failed` | It used up its attempts, or got a response that isn't retried. |
| `cancelled` | Cancelled before it succeeded or failed. A running task's current attempt still finishes and is recorded, but isn't retried. |

`failed` isn't the end: once the endpoint is fixed, retry one task (`POST /v1/tasks/{id}/retry`) or every failure in a queue (`workly queues retry-failed`), and it gets a fresh set of attempts. Finished tasks are kept, with their attempts, for `WORKLY_RETENTION` (7 days by default), then deleted.

### Idempotency keys

Enqueue a task with an `idempotency_key` that's already in the queue and you get the existing task back (`200` instead of `201`). A key holds as long as its task is kept, so retrying an enqueue after a timeout or a crash is safe. This deduplicates enqueues; deduplicating deliveries is your handler's job (see [Delivery semantics](#delivery-semantics)).

## Attempts

Each delivery of a task is an **attempt**, recorded with its start and end, duration, status code, response headers, the first 4 KB of the response body, and how it ended:

| Outcome | |
|---|---|
| `success` | A 2xx response. |
| `http_error` | Any other response. |
| `timeout` | No response within `timeout_seconds`. |
| `network_error` | The connection failed: DNS, refused, reset, TLS. |
| `lost` | Workly stopped mid-delivery, so the result is unknown. The endpoint may have done the work. |

This history is what the dashboard, the CLI and [MCP](https://workly.run/docs/mcp/) show when you ask why a task failed.

### Retries

- **Retried:** 5xx, 408 and 429 responses, timeouts, network errors and lost attempts.
- **Not retried:** other 4xx responses fail the task at once, since sending the same request again won't change the answer. Set `retry_on_4xx` on the queue if your endpoint uses them for temporary problems. Connections the server refuses to make, such as to a private address on a server that only delivers to the public internet, aren't retried either.
- **Redirects:** Workly doesn't follow them. A 3xx response (or a 1xx) fails the task at once, since it usually means the URL is wrong: point the task at where the redirect goes.
- **Backoff:** retry *n* waits a random time between `min_backoff_seconds` and `min_backoff_seconds × backoff_multiplier^(n−1)`, capped at `max_backoff_seconds`. The randomness (jitter) spreads out a burst of failures. A `Retry-After` header on the response is honoured.

## Delivery semantics

Delivery is **at least once**. Workly never drops a task, but your endpoint can see one more than once:

- Workly can stop between sending a request and recording the response (a crash, a deploy, a lost connection to the database). The attempt is recorded as `lost` and the task is delivered again.
- Your endpoint can do the work and then time out, or fail to answer. Workly can't tell that apart from not doing it.

So make handlers safe to run twice. Every delivery carries headers to help:

| Header | |
|---|---|
| `Workly-Task-Id` | The task, the same on every attempt. Use it as a deduplication key. |
| `Workly-Queue` | The task's queue. |
| `Workly-Attempt` | The attempt number, from 1. |
| `Workly-Timestamp`, `Workly-Signature` | The signature (see [Signing](#signing)). |
| `traceparent` | The trace context of the request that enqueued the task, if it had one, so your handler's spans join that trace. |

Your own `headers` are sent too; `Workly-*` names are reserved. Schedule runs carry `Workly-Run-Id`, `Workly-Schedule` and `Workly-Scheduled-For` instead of the task headers ([schedules](https://workly.run/docs/schedules/#handling-a-run)), and a [workflow](https://workly.run/docs/workflows/#what-a-step-receives)'s steps carry `Workly-Workflow`, `Workly-Workflow-Run-Id`, `Workly-Workflow-Step`, `Workly-Workflow-Step-Id` and `Workly-Workflow-Run-Attempt` as well.

Order isn't guaranteed. Tasks start roughly in the order they're due, but with several deliveries at once, and with retries, they can finish in any order. If order matters, set `max_concurrency: 1`; a retry still lets later tasks go first.

## Signing

Every delivery is signed with a secret that Workly and your endpoint share (`WORKLY_SIGNING_SECRET` when you self-host; the API keys page on workly.run). Check the signature before trusting a request: anyone who can reach your endpoint can call it, but only Workly knows the secret.

The [SDKs](https://workly.run/docs/quickstart/#handle-it) check it with `verify_signature` in Python and `verifySignature` in TypeScript. In another language it's a few lines.

### Verifying a signature without an SDK

A delivery carries two headers:

```text
Workly-Timestamp: 1790000000
Workly-Signature: v1=00f7255c88a43b3e2d47b211ff8e71cf935601cabff5e7adad0de2cc72a4e3c6
```

The signature is the hex HMAC-SHA256, keyed with the secret, of the timestamp, a `.`, and the request body exactly as it arrived:

```text
v1=hex(HMAC-SHA256(secret, "<Workly-Timestamp>.<body>"))
```

To verify a request:

1. Read the **raw body**, before any JSON parsing: re-serializing it changes the bytes and breaks the signature.
2. Compute the HMAC of `timestamp + "." + body` with your secret.
3. Split `Workly-Signature` on commas. It holds more than one signature while the secret is being [rotated](https://workly.run/docs/configuration/#rotating-the-signing-secret); accept the request if any `v1=` value matches. Compare in constant time.
4. Reject the request if the timestamp is more than 5 minutes from your clock, so a captured request can't be replayed later. The SDKs use the same tolerance.

The timestamp is the time of the attempt, so every retry has a fresh one and a new signature.

Test vector: the secret `secret`, the timestamp `1790000000` and the body `{"user_id":"123"}` give the signature above. Check it from a shell:

```sh
printf '%s' '1790000000.{"user_id":"123"}' | openssl dgst -sha256 -hmac secret
```

In Go:

```go
func verify(secret []byte, r *http.Request, body []byte) bool {
	ts, err := strconv.ParseInt(r.Header.Get("Workly-Timestamp"), 10, 64)
	if err != nil || time.Since(time.Unix(ts, 0)).Abs() > 5*time.Minute {
		return false
	}
	mac := hmac.New(sha256.New, secret)
	fmt.Fprintf(mac, "%d.", ts)
	mac.Write(body)
	want := mac.Sum(nil)
	for _, part := range strings.Split(r.Header.Get("Workly-Signature"), ",") {
		sig, ok := strings.CutPrefix(strings.TrimSpace(part), "v1=")
		got, err := hex.DecodeString(sig)
		if ok && err == nil && hmac.Equal(got, want) {
			return true
		}
	}
	return false
}
```

In Ruby (3.0 or later):

```ruby
require "openssl"

def verify(secret, headers, body)
  ts = headers["Workly-Timestamp"].to_s
  return false unless ts.match?(/\A\d+\z/) && (Time.now.to_i - ts.to_i).abs <= 300
  want = "v1=" + OpenSSL::HMAC.hexdigest("SHA256", secret, "#{ts}.#{body}")
  headers["Workly-Signature"].to_s.split(",").any? { |sig| OpenSSL.secure_compare(sig.strip, want) }
end
```

Answer a request that doesn't verify with a 401. Workly doesn't retry 4xx responses, so a wrong secret shows up at once as failed tasks with a 401, instead of an hour of retries.
