Skip to content

Concepts

Queues, tasks and attempts, what happens on each delivery, and how your endpoint checks that a request came from Workly. Schedules and task groups build on the same pieces.

A queue holds tasks and the policy for delivering them. It’s created with the defaults below the first time you enqueue into it. Change its policy with PUT /v1/queues/{queue}, workly queues put or a workly.yaml.

Setting Default
max_attempts 5 Total attempts before a task fails.
min_backoff_seconds, max_backoff_seconds, backoff_multiplier 2, 3600, 2 The wait between attempts (see Retries).
timeout_seconds 30 How long one attempt may take.
max_concurrency 3 Deliveries in flight at once, up to 10,000.
rate_per_second none Deliveries started per second.
retry_on_4xx false Retry 4xx responses other than 408 and 429, which are always retried.
auto_pause none {after_failures: n}: pause the queue by itself when its endpoint looks down (see Auto-pause).

The full list is in the API reference. Queue names are lowercase letters, digits, - and _; names starting with _ are Workly’s own.

A paused queue keeps accepting tasks but doesn’t start deliveries; those already in flight finish. Workly shares deliveries fairly between queues, so a busy queue doesn’t hold up a quiet one.

When an endpoint goes down, retries keep using up each task’s attempts, and an outage longer than the retry window leaves every task failed, to retry by hand afterwards. A queue with auto_pause stops instead:

queues:
billing:
auto_pause:
after_failures: 20
  • What pauses it: after_failures attempts in a row that end with a 5xx, a timeout or a network error. These say the endpoint itself is unwell. Other 4xx responses (usually about one task), 429 (the endpoint is pacing you, and Retry-After handles it) and lost attempts don’t count, and only a success resets the count.
  • While it’s paused: no deliveries start, and tasks wait as pending without using attempts. One task at a time goes as a probe: 30 seconds after the pause, then after twice as long each time a probe fails, up to every 10 minutes. A probe is a normal attempt of the queue’s oldest due task.
  • Resuming: the first success resumes the queue on its own, and the waiting tasks are delivered within its rate and concurrency limits. Resuming it by hand (POST /v1/queues/{queue}/resume, workly queues resume, or Resume now in the dashboard) does it at once. Pausing it by hand stops the probes too.
  • Seeing it: the queue’s auto_paused says since when, the error, how many probes failed and when the next goes; it’s absent when the queue isn’t auto-paused. Alerts go out when a queue pauses itself and when it resumes, and the workly_queue_auto_paused metric is 1 while it’s paused.

Use it for a queue whose tasks go to one service. In a queue that delivers to many endpoints, such as webhooks to your customers’ URLs, one of them being down would hold up the rest. Workly’s own queues (_workflows, _callbacks) can’t have it. A workflow step in an auto-paused queue waits like any task, and its run still fails after its max_duration_seconds.

A task is one HTTP request Workly owes your endpoint: a URL, a method (POST by default), headers, and a body, either payload sent as JSON or body with its own content_type. delay_seconds or run_at sets when it’s first due. It can override its queue’s max_attempts and timeout_seconds, and with on_complete call you back each time it finishes (see Callbacks).

A task is in one of these states:

State
pending Waiting to be delivered: due now, waiting for its run_at, or waiting to be retried.
running An attempt is in flight.
succeeded An attempt got a 2xx response.
failed It used up its attempts, or got a response that isn’t retried.
cancelled Cancelled before it succeeded or failed. A running task’s current attempt still finishes and is recorded, but isn’t retried.

failed isn’t the end: once the endpoint is fixed, retry one task (POST /v1/tasks/{id}/retry) or every failure in a queue (workly queues retry-failed), and it gets a fresh set of attempts. Finished tasks are kept, with their attempts, for WORKLY_RETENTION (7 days by default), then deleted.

Enqueue a task with an idempotency_key that’s already in the queue and you get the existing task back (200 instead of 201). A key holds as long as its task is kept, so retrying an enqueue after a timeout or a crash is safe. This deduplicates enqueues; deduplicating deliveries is your handler’s job (see Delivery semantics).

Each delivery of a task is an attempt, recorded with its start and end, duration, status code, response headers, the first 4 KB of the response body, and how it ended:

Outcome
success A 2xx response.
http_error Any other response.
timeout No response within timeout_seconds.
network_error The connection failed: DNS, refused, reset, TLS.
lost Workly stopped mid-delivery, so the result is unknown. The endpoint may have done the work.

This history is what the dashboard, the CLI and MCP show when you ask why a task failed.

  • Retried: 5xx, 408 and 429 responses, timeouts, network errors and lost attempts.
  • Not retried: other 4xx responses fail the task at once, since sending the same request again won’t change the answer. Set retry_on_4xx on the queue if your endpoint uses them for temporary problems. Connections the server refuses to make, such as to a private address on a server that only delivers to the public internet, aren’t retried either.
  • Redirects: Workly doesn’t follow them. A 3xx response (or a 1xx) fails the task at once, since it usually means the URL is wrong: point the task at where the redirect goes.
  • Backoff: retry n waits a random time between min_backoff_seconds and min_backoff_seconds × backoff_multiplier^(n−1), capped at max_backoff_seconds. The randomness (jitter) spreads out a burst of failures. A Retry-After header on the response is honoured.

Delivery is at least once. Workly never drops a task, but your endpoint can see one more than once:

  • Workly can stop between sending a request and recording the response (a crash, a deploy, a lost connection to the database). The attempt is recorded as lost and the task is delivered again.
  • Your endpoint can do the work and then time out, or fail to answer. Workly can’t tell that apart from not doing it.

So make handlers safe to run twice. Every delivery carries headers to help:

Header
Workly-Task-Id The task, the same on every attempt. Use it as a deduplication key.
Workly-Queue The task’s queue.
Workly-Attempt The attempt number, from 1.
Workly-Timestamp, Workly-Signature The signature (see Signing).
traceparent The trace context of the request that enqueued the task, if it had one, so your handler’s spans join that trace.

Your own headers are sent too; Workly-* names are reserved. Schedule runs carry Workly-Run-Id, Workly-Schedule and Workly-Scheduled-For instead of the task headers (schedules), and a workflow’s steps carry Workly-Workflow, Workly-Workflow-Run-Id, Workly-Workflow-Step, Workly-Workflow-Step-Id and Workly-Workflow-Run-Attempt as well.

Order isn’t guaranteed. Tasks start roughly in the order they’re due, but with several deliveries at once, and with retries, they can finish in any order. If order matters, set max_concurrency: 1; a retry still lets later tasks go first.

Every delivery is signed with a secret that Workly and your endpoint share (WORKLY_SIGNING_SECRET when you self-host; the API keys page on workly.run). Check the signature before trusting a request: anyone who can reach your endpoint can call it, but only Workly knows the secret.

The SDKs check it with verify_signature in Python and verifySignature in TypeScript. In another language it’s a few lines.

A delivery carries two headers:

Workly-Timestamp: 1790000000
Workly-Signature: v1=00f7255c88a43b3e2d47b211ff8e71cf935601cabff5e7adad0de2cc72a4e3c6

The signature is the hex HMAC-SHA256, keyed with the secret, of the timestamp, a ., and the request body exactly as it arrived:

v1=hex(HMAC-SHA256(secret, "<Workly-Timestamp>.<body>"))

To verify a request:

  1. Read the raw body, before any JSON parsing: re-serializing it changes the bytes and breaks the signature.
  2. Compute the HMAC of timestamp + "." + body with your secret.
  3. Split Workly-Signature on commas. It holds more than one signature while the secret is being rotated; accept the request if any v1= value matches. Compare in constant time.
  4. Reject the request if the timestamp is more than 5 minutes from your clock, so a captured request can’t be replayed later. The SDKs use the same tolerance.

The timestamp is the time of the attempt, so every retry has a fresh one and a new signature.

Test vector: the secret secret, the timestamp 1790000000 and the body {"user_id":"123"} give the signature above. Check it from a shell:

printf '%s' '1790000000.{"user_id":"123"}' | openssl dgst -sha256 -hmac secret

In Go:

func verify(secret []byte, r *http.Request, body []byte) bool {
ts, err := strconv.ParseInt(r.Header.Get("Workly-Timestamp"), 10, 64)
if err != nil || time.Since(time.Unix(ts, 0)).Abs() > 5*time.Minute {
return false
}
mac := hmac.New(sha256.New, secret)
fmt.Fprintf(mac, "%d.", ts)
mac.Write(body)
want := mac.Sum(nil)
for _, part := range strings.Split(r.Header.Get("Workly-Signature"), ",") {
sig, ok := strings.CutPrefix(strings.TrimSpace(part), "v1=")
got, err := hex.DecodeString(sig)
if ok && err == nil && hmac.Equal(got, want) {
return true
}
}
return false
}

In Ruby (3.0 or later):

require "openssl"
def verify(secret, headers, body)
ts = headers["Workly-Timestamp"].to_s
return false unless ts.match?(/\A\d+\z/) && (Time.now.to_i - ts.to_i).abs <= 300
want = "v1=" + OpenSSL::HMAC.hexdigest("SHA256", secret, "#{ts}.#{body}")
headers["Workly-Signature"].to_s.split(",").any? { |sig| OpenSSL.secure_compare(sig.strip, want) }
end

Answer a request that doesn’t verify with a 401. Workly doesn’t retry 4xx responses, so a wrong secret shows up at once as failed tasks with a 401, instead of an hour of retries.