Concepts
Queues, tasks and attempts, what happens on each delivery, and how your endpoint checks that a request came from Workly. Schedules and task groups build on the same pieces.
Queues
Section titled “Queues”A queue holds tasks and the policy for delivering them. It’s created with the defaults below the first time you enqueue into it. Change its policy with PUT /v1/queues/{queue}, workly queues put or a workly.yaml.
| Setting | Default | |
|---|---|---|
max_attempts |
5 | Total attempts before a task fails. |
min_backoff_seconds, max_backoff_seconds, backoff_multiplier |
2, 3600, 2 | The wait between attempts (see Retries). |
timeout_seconds |
30 | How long one attempt may take. |
max_concurrency |
3 | Deliveries in flight at once, up to 10,000. |
rate_per_second |
none | Deliveries started per second. |
retry_on_4xx |
false | Retry 4xx responses other than 408 and 429, which are always retried. |
auto_pause |
none | {after_failures: n}: pause the queue by itself when its endpoint looks down (see Auto-pause). |
The full list is in the API reference. Queue names are lowercase letters, digits, - and _; names starting with _ are Workly’s own.
A paused queue keeps accepting tasks but doesn’t start deliveries; those already in flight finish. Workly shares deliveries fairly between queues, so a busy queue doesn’t hold up a quiet one.
Auto-pause
Section titled “Auto-pause”When an endpoint goes down, retries keep using up each task’s attempts, and an outage longer than the retry window leaves every task failed, to retry by hand afterwards. A queue with auto_pause stops instead:
queues: billing: auto_pause: after_failures: 20- What pauses it:
after_failuresattempts in a row that end with a 5xx, a timeout or a network error. These say the endpoint itself is unwell. Other 4xx responses (usually about one task), 429 (the endpoint is pacing you, andRetry-Afterhandles it) and lost attempts don’t count, and only a success resets the count. - While it’s paused: no deliveries start, and tasks wait as
pendingwithout using attempts. One task at a time goes as a probe: 30 seconds after the pause, then after twice as long each time a probe fails, up to every 10 minutes. A probe is a normal attempt of the queue’s oldest due task. - Resuming: the first success resumes the queue on its own, and the waiting tasks are delivered within its rate and concurrency limits. Resuming it by hand (
POST /v1/queues/{queue}/resume,workly queues resume, or Resume now in the dashboard) does it at once. Pausing it by hand stops the probes too. - Seeing it: the queue’s
auto_pausedsays since when, the error, how many probes failed and when the next goes; it’s absent when the queue isn’t auto-paused. Alerts go out when a queue pauses itself and when it resumes, and theworkly_queue_auto_pausedmetric is 1 while it’s paused.
Use it for a queue whose tasks go to one service. In a queue that delivers to many endpoints, such as webhooks to your customers’ URLs, one of them being down would hold up the rest. Workly’s own queues (_workflows, _callbacks) can’t have it. A workflow step in an auto-paused queue waits like any task, and its run still fails after its max_duration_seconds.
A task is one HTTP request Workly owes your endpoint: a URL, a method (POST by default), headers, and a body, either payload sent as JSON or body with its own content_type. delay_seconds or run_at sets when it’s first due. It can override its queue’s max_attempts and timeout_seconds, and with on_complete call you back each time it finishes (see Callbacks).
A task is in one of these states:
| State | |
|---|---|
pending |
Waiting to be delivered: due now, waiting for its run_at, or waiting to be retried. |
running |
An attempt is in flight. |
succeeded |
An attempt got a 2xx response. |
failed |
It used up its attempts, or got a response that isn’t retried. |
cancelled |
Cancelled before it succeeded or failed. A running task’s current attempt still finishes and is recorded, but isn’t retried. |
failed isn’t the end: once the endpoint is fixed, retry one task (POST /v1/tasks/{id}/retry) or every failure in a queue (workly queues retry-failed), and it gets a fresh set of attempts. Finished tasks are kept, with their attempts, for WORKLY_RETENTION (7 days by default), then deleted.
Idempotency keys
Section titled “Idempotency keys”Enqueue a task with an idempotency_key that’s already in the queue and you get the existing task back (200 instead of 201). A key holds as long as its task is kept, so retrying an enqueue after a timeout or a crash is safe. This deduplicates enqueues; deduplicating deliveries is your handler’s job (see Delivery semantics).
Attempts
Section titled “Attempts”Each delivery of a task is an attempt, recorded with its start and end, duration, status code, response headers, the first 4 KB of the response body, and how it ended:
| Outcome | |
|---|---|
success |
A 2xx response. |
http_error |
Any other response. |
timeout |
No response within timeout_seconds. |
network_error |
The connection failed: DNS, refused, reset, TLS. |
lost |
Workly stopped mid-delivery, so the result is unknown. The endpoint may have done the work. |
This history is what the dashboard, the CLI and MCP show when you ask why a task failed.
Retries
Section titled “Retries”- Retried: 5xx, 408 and 429 responses, timeouts, network errors and lost attempts.
- Not retried: other 4xx responses fail the task at once, since sending the same request again won’t change the answer. Set
retry_on_4xxon the queue if your endpoint uses them for temporary problems. Connections the server refuses to make, such as to a private address on a server that only delivers to the public internet, aren’t retried either. - Redirects: Workly doesn’t follow them. A 3xx response (or a 1xx) fails the task at once, since it usually means the URL is wrong: point the task at where the redirect goes.
- Backoff: retry n waits a random time between
min_backoff_secondsandmin_backoff_seconds × backoff_multiplier^(n−1), capped atmax_backoff_seconds. The randomness (jitter) spreads out a burst of failures. ARetry-Afterheader on the response is honoured.
Delivery semantics
Section titled “Delivery semantics”Delivery is at least once. Workly never drops a task, but your endpoint can see one more than once:
- Workly can stop between sending a request and recording the response (a crash, a deploy, a lost connection to the database). The attempt is recorded as
lostand the task is delivered again. - Your endpoint can do the work and then time out, or fail to answer. Workly can’t tell that apart from not doing it.
So make handlers safe to run twice. Every delivery carries headers to help:
| Header | |
|---|---|
Workly-Task-Id |
The task, the same on every attempt. Use it as a deduplication key. |
Workly-Queue |
The task’s queue. |
Workly-Attempt |
The attempt number, from 1. |
Workly-Timestamp, Workly-Signature |
The signature (see Signing). |
traceparent |
The trace context of the request that enqueued the task, if it had one, so your handler’s spans join that trace. |
Your own headers are sent too; Workly-* names are reserved. Schedule runs carry Workly-Run-Id, Workly-Schedule and Workly-Scheduled-For instead of the task headers (schedules), and a workflow’s steps carry Workly-Workflow, Workly-Workflow-Run-Id, Workly-Workflow-Step, Workly-Workflow-Step-Id and Workly-Workflow-Run-Attempt as well.
Order isn’t guaranteed. Tasks start roughly in the order they’re due, but with several deliveries at once, and with retries, they can finish in any order. If order matters, set max_concurrency: 1; a retry still lets later tasks go first.
Signing
Section titled “Signing”Every delivery is signed with a secret that Workly and your endpoint share (WORKLY_SIGNING_SECRET when you self-host; the API keys page on workly.run). Check the signature before trusting a request: anyone who can reach your endpoint can call it, but only Workly knows the secret.
The SDKs check it with verify_signature in Python and verifySignature in TypeScript. In another language it’s a few lines.
Verifying a signature without an SDK
Section titled “Verifying a signature without an SDK”A delivery carries two headers:
Workly-Timestamp: 1790000000Workly-Signature: v1=00f7255c88a43b3e2d47b211ff8e71cf935601cabff5e7adad0de2cc72a4e3c6The signature is the hex HMAC-SHA256, keyed with the secret, of the timestamp, a ., and the request body exactly as it arrived:
v1=hex(HMAC-SHA256(secret, "<Workly-Timestamp>.<body>"))To verify a request:
- Read the raw body, before any JSON parsing: re-serializing it changes the bytes and breaks the signature.
- Compute the HMAC of
timestamp + "." + bodywith your secret. - Split
Workly-Signatureon commas. It holds more than one signature while the secret is being rotated; accept the request if anyv1=value matches. Compare in constant time. - Reject the request if the timestamp is more than 5 minutes from your clock, so a captured request can’t be replayed later. The SDKs use the same tolerance.
The timestamp is the time of the attempt, so every retry has a fresh one and a new signature.
Test vector: the secret secret, the timestamp 1790000000 and the body {"user_id":"123"} give the signature above. Check it from a shell:
printf '%s' '1790000000.{"user_id":"123"}' | openssl dgst -sha256 -hmac secretIn Go:
func verify(secret []byte, r *http.Request, body []byte) bool { ts, err := strconv.ParseInt(r.Header.Get("Workly-Timestamp"), 10, 64) if err != nil || time.Since(time.Unix(ts, 0)).Abs() > 5*time.Minute { return false } mac := hmac.New(sha256.New, secret) fmt.Fprintf(mac, "%d.", ts) mac.Write(body) want := mac.Sum(nil) for _, part := range strings.Split(r.Header.Get("Workly-Signature"), ",") { sig, ok := strings.CutPrefix(strings.TrimSpace(part), "v1=") got, err := hex.DecodeString(sig) if ok && err == nil && hmac.Equal(got, want) { return true } } return false}In Ruby (3.0 or later):
require "openssl"
def verify(secret, headers, body) ts = headers["Workly-Timestamp"].to_s return false unless ts.match?(/\A\d+\z/) && (Time.now.to_i - ts.to_i).abs <= 300 want = "v1=" + OpenSSL::HMAC.hexdigest("SHA256", secret, "#{ts}.#{body}") headers["Workly-Signature"].to_s.split(",").any? { |sig| OpenSSL.secure_compare(sig.strip, want) }endAnswer a request that doesn’t verify with a 401. Workly doesn’t retry 4xx responses, so a wrong secret shows up at once as failed tasks with a 401, instead of an hour of retries.