This is the full developer documentation for Workly
# Workly documentation
> Workly is an open-source HTTP task queue and cron scheduler: enqueue a task with a URL, and Workly delivers it with retries, rate limits and full history.
Workly is an open-source HTTP task queue and scheduler. You enqueue a task with a URL and a payload; Workly delivers it to your endpoint and retries with backoff until the endpoint answers 2xx. There are no workers to run and no broker to operate.
It also runs [schedules](https://workly.run/docs/schedules/), which call an endpoint on a cron timetable; [task groups](https://workly.run/docs/groups/), which call you back once a set of tasks has finished; and [workflows](https://workly.run/docs/workflows/), which run HTTP steps in dependency order.
The same binary runs everywhere: `workly dev` on your laptop (SQLite, no configuration), `workly serve` with Postgres in your own infrastructure, or hosted at [workly.run](https://workly.run). Moving between them means changing environment variables, not code.
## Start here
* [Quickstart](https://workly.run/docs/quickstart/): run Workly locally, enqueue a task and handle it.
* [Concepts](https://workly.run/docs/concepts/): queues, tasks, attempts, retries, delivery semantics and signing, including verifying signatures without an SDK.
* [Schedules](https://workly.run/docs/schedules/): call an endpoint on a cron timetable.
* [Task groups](https://workly.run/docs/groups/): fan out tasks and get one callback when they’ve all finished.
* [Workflows](https://workly.run/docs/workflows/): run HTTP steps in dependency order, each delivered as a task, waiting for groups where they fan out.
* [Callbacks](https://workly.run/docs/callbacks/): be called with the outcome each time a task or a schedule’s run finishes.
* [Alerts](https://workly.run/docs/alerts/): hear about failed tasks and runs by email or in Slack.
* [Configuration as code](https://workly.run/docs/deploy/): keep queues, workflows and schedules in a `workly.yaml` and deploy it with `workly deploy`.
* [Self-hosting](https://workly.run/docs/self-hosting/): run it in production on Docker, Compose or Kubernetes.
* [MCP](https://workly.run/docs/mcp/): let coding agents see inside your queues and act on them.
## How delivery works
* **Tasks** belong to a **queue**, created on first use. The queue holds the delivery settings: retries, timeout, rate and concurrency limits. A task can override its retries and timeout.
* Each delivery is an **attempt**. A 2xx response marks the task succeeded. A 5xx, 408 or 429 response, a timeout or a connection error is retried with exponential backoff and jitter, honouring `Retry-After`. Other 4xx responses fail the task straight away, unless the queue sets `retry_on_4xx`.
* By default a queue delivers up to 3 tasks at once, makes up to 5 attempts, waits between 2 seconds and 1 hour between them, and times attempts out after 30 seconds. Raise `max_concurrency` (up to 10,000) for endpoints that can take more.
* A task that runs out of attempts is **failed**. It keeps every attempt’s status, error and response body, and once the endpoint is fixed you can retry it, or all of a queue’s failures at once.
* Delivery is **at least once**: if the server stops mid-delivery, the attempt is recorded as `lost` and the task is delivered again. Make handlers safe to run twice, using the `Workly-Task-Id` header, which is the same on every attempt, as a deduplication key.
* Every delivery is **signed** with `Workly-Signature`. The SDKs check it in one call, and [Concepts](https://workly.run/docs/concepts/#verifying-a-signature-without-an-sdk) shows how in any language.
## Reference
* SDKs: [Python](https://workly.run/docs/sdks/python/) and [TypeScript](https://workly.run/docs/sdks/typescript/), with every method.
* [CLI](https://workly.run/docs/cli/): every `workly` command and flag.
* [API reference](https://workly.run/docs/api/): the HTTP API, generated from its OpenAPI spec.
* [Configuration](https://workly.run/docs/configuration/): every `WORKLY_*` setting.
* [Metrics](https://workly.run/docs/metrics/): Prometheus metrics and example alerts.
* [Performance](https://workly.run/docs/performance/): load test results.
## For AI agents
* [workly.run/docs/llms-full.txt](https://workly.run/docs/llms-full.txt) has all of these docs in one file, SDKs and CLI included. [llms-small.txt](https://workly.run/docs/llms-small.txt) leaves out the reference pages and self-hosting.
* Each page is also Markdown at its address plus `.md`, such as [workly.run/docs/quickstart.md](https://workly.run/docs/quickstart.md).
* The API is described by [openapi.yaml](https://workly.run/docs/api/openapi.yaml).
* An agent connected over [MCP](https://workly.run/docs/mcp/) can read these pages with the `get_docs` tool.
# Quickstart
> Run Workly locally with workly dev, enqueue a task from Python, TypeScript or curl, and handle it in your own endpoint. About five minutes.
Run Workly on your machine, enqueue a task and handle it in your own code.
## Install
```sh
pip install "workly[cli]" # the Python SDK plus the workly binary
# or just the binary:
curl -fsSL https://raw.githubusercontent.com/worklydev/workly/main/install.sh | sh
```
Binaries for Linux, macOS and Windows are also on the [releases page](https://github.com/worklydev/workly/releases). With Docker, `docker run -p 7337:7337 ghcr.io/worklydev/workly dev`.
SDKs, the CLI and the server work together when their first two version numbers match: any 0.4.x SDK or CLI works with any 0.4.x server. Each is released on its own, so the last number can differ. workly.run runs the latest server; `workly --version` prints the CLI’s.
## Run it
```sh
workly dev
```
This starts a server and the dashboard on , with SQLite storage in `.workly/` and no API key. It prints its settings, including the signing secret (`workly-dev-secret`).
## Enqueue a task
`workly dev` has a built-in echo target at `/_dev/echo`. With `fail_rate=0.5` it fails half the time, so you can watch retries:
```sh
curl -X POST localhost:7337/v1/queues/demo/tasks \
-d '{"url": "http://localhost:7337/_dev/echo?fail_rate=0.5", "payload": {"hello": "world"}}'
```
The queue `demo` is created on first use. Queue names are lowercase letters, digits, `-` and `_`; names starting with `_` are reserved for Workly. See the task’s attempts in the dashboard, or with the API:
```sh
curl localhost:7337/v1/tasks/
curl localhost:7337/v1/tasks//attempts
```
## Enqueue from your code
Python (`pip install workly`):
```python
from workly import Workly
client = Workly() # WORKLY_URL, else http://localhost:7337; WORKLY_API_KEY
task = client.tasks.enqueue(
"emails",
url="http://localhost:8000/send-email",
payload={"user_id": 123},
)
```
TypeScript (`npm install @worklydev/sdk`):
```ts
import { Workly } from '@worklydev/sdk';
const workly = new Workly(); // WORKLY_URL, else http://localhost:7337; WORKLY_API_KEY
const task = await workly.tasks.enqueue('emails', {
url: 'http://localhost:8000/send-email',
payload: { user_id: 123 },
});
```
Tasks can also be delayed, deduplicated with an idempotency key, and given their own retries and timeout: the [Python](https://workly.run/docs/sdks/python/#tasks) and [TypeScript](https://workly.run/docs/sdks/typescript/#tasks) SDK pages cover every option.
## Handle it
Your endpoint receives a `POST` with the JSON payload. Answer 2xx when the work is done; anything else is retried. Check the signature against the raw body first:
```python
from flask import Flask, abort, request
from workly import SignatureVerificationError, verify_signature
app = Flask(__name__)
@app.post("/send-email")
def send_email():
try:
verify_signature(request.get_data(), request.headers) # secret from WORKLY_SIGNING_SECRET
except SignatureVerificationError:
abort(401)
user_id = request.get_json()["user_id"]
...
return "", 204
```
```ts
import { SignatureVerificationError, verifySignature } from '@worklydev/sdk';
export async function POST(request: Request) {
const body = await request.text();
try {
await verifySignature(body, request.headers); // secret from WORKLY_SIGNING_SECRET
} catch (err) {
if (err instanceof SignatureVerificationError) return new Response(null, { status: 401 });
throw err;
}
const { user_id } = JSON.parse(body);
// ...
return new Response(null, { status: 204 });
}
```
Run your handler with `WORKLY_SIGNING_SECRET=workly-dev-secret`. Delivery is at least once, so make the handler safe to run twice; `Workly-Task-Id` is the same on every attempt.
## Shape the queue
A new queue delivers up to 3 tasks at once, with no rate limit:
```sh
workly queues ls
curl -X PUT localhost:7337/v1/queues/emails -d '{"rate_per_second": 5, "max_concurrency": 2}'
workly queues pause emails
```
When a handler has been failing, see its failures grouped by error, fix it, and retry them:
```sh
workly queues failures emails
workly queues retry-failed emails
```
## Run something on a schedule
[Schedules](https://workly.run/docs/schedules/) call an endpoint on a cron timetable, separately from queues. This one calls the echo target every minute:
```sh
workly schedules put heartbeat http://localhost:7337/_dev/echo --cron '* * * * *'
workly schedules trigger heartbeat # run it now, too
workly schedules get heartbeat # its next times and recent runs
```
## Next
* Connect your coding agent over [MCP](https://workly.run/docs/mcp/) to enqueue test tasks and read failures while you work on a handler.
* Go to production with [self-hosting](https://workly.run/docs/self-hosting/), or point `WORKLY_URL` at a hosted server.
# Concepts
> How Workly queues, tasks and attempts work: retries and backoff, at-least-once delivery, delivery headers, and verifying signatures without an SDK.
Queues, tasks and attempts, what happens on each delivery, and how your endpoint checks that a request came from Workly. [Schedules](https://workly.run/docs/schedules/) and [task groups](https://workly.run/docs/groups/) build on the same pieces.
## Queues
A **queue** holds tasks and the policy for delivering them. It’s created with the defaults below the first time you enqueue into it. Change its policy with `PUT /v1/queues/{queue}`, `workly queues put` or a [`workly.yaml`](https://workly.run/docs/deploy/).
| Setting | Default | |
| ------------------------------------------------------------------ | ---------- | -------------------------------------------------------------------------------------------------------------- |
| `max_attempts` | 5 | Total attempts before a task fails. |
| `min_backoff_seconds`, `max_backoff_seconds`, `backoff_multiplier` | 2, 3600, 2 | The wait between attempts (see [Retries](#retries)). |
| `timeout_seconds` | 30 | How long one attempt may take. |
| `max_concurrency` | 3 | Deliveries in flight at once, up to 10,000. |
| `rate_per_second` | none | Deliveries started per second. |
| `retry_on_4xx` | false | Retry 4xx responses other than 408 and 429, which are always retried. |
| `auto_pause` | none | `{after_failures: n}`: pause the queue by itself when its endpoint looks down (see [Auto-pause](#auto-pause)). |
The full list is in the [API reference](https://workly.run/docs/api/). Queue names are lowercase letters, digits, `-` and `_`; names starting with `_` are Workly’s own.
A **paused** queue keeps accepting tasks but doesn’t start deliveries; those already in flight finish. Workly shares deliveries fairly between queues, so a busy queue doesn’t hold up a quiet one.
### Auto-pause
When an endpoint goes down, retries keep using up each task’s attempts, and an outage longer than the retry window leaves every task `failed`, to retry by hand afterwards. A queue with `auto_pause` stops instead:
```yaml
queues:
billing:
auto_pause:
after_failures: 20
```
* **What pauses it:** `after_failures` attempts in a row that end with a 5xx, a timeout or a network error. These say the endpoint itself is unwell. Other 4xx responses (usually about one task), 429 (the endpoint is pacing you, and `Retry-After` handles it) and lost attempts don’t count, and only a success resets the count.
* **While it’s paused:** no deliveries start, and tasks wait as `pending` without using attempts. One task at a time goes as a probe: 30 seconds after the pause, then after twice as long each time a probe fails, up to every 10 minutes. A probe is a normal attempt of the queue’s oldest due task.
* **Resuming:** the first success resumes the queue on its own, and the waiting tasks are delivered within its rate and concurrency limits. Resuming it by hand (`POST /v1/queues/{queue}/resume`, `workly queues resume`, or **Resume now** in the dashboard) does it at once. Pausing it by hand stops the probes too.
* **Seeing it:** the queue’s `auto_paused` says since when, the error, how many probes failed and when the next goes; it’s absent when the queue isn’t auto-paused. [Alerts](https://workly.run/docs/alerts/) go out when a queue pauses itself and when it resumes, and the `workly_queue_auto_paused` metric is 1 while it’s paused.
Use it for a queue whose tasks go to one service. In a queue that delivers to many endpoints, such as webhooks to your customers’ URLs, one of them being down would hold up the rest. Workly’s own queues (`_workflows`, `_callbacks`) can’t have it. A workflow step in an auto-paused queue waits like any task, and its run still fails after its `max_duration_seconds`.
## Tasks
A **task** is one HTTP request Workly owes your endpoint: a URL, a method (`POST` by default), headers, and a body, either `payload` sent as JSON or `body` with its own `content_type`. `delay_seconds` or `run_at` sets when it’s first due. It can override its queue’s `max_attempts` and `timeout_seconds`, and with `on_complete` call you back each time it finishes (see [Callbacks](https://workly.run/docs/callbacks/)).
A task is in one of these states:
| State | |
| ----------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `pending` | Waiting to be delivered: due now, waiting for its `run_at`, or waiting to be retried. |
| `running` | An attempt is in flight. |
| `succeeded` | An attempt got a 2xx response. |
| `failed` | It used up its attempts, or got a response that isn’t retried. |
| `cancelled` | Cancelled before it succeeded or failed. A running task’s current attempt still finishes and is recorded, but isn’t retried. |
`failed` isn’t the end: once the endpoint is fixed, retry one task (`POST /v1/tasks/{id}/retry`) or every failure in a queue (`workly queues retry-failed`), and it gets a fresh set of attempts. Finished tasks are kept, with their attempts, for `WORKLY_RETENTION` (7 days by default), then deleted.
### Idempotency keys
Enqueue a task with an `idempotency_key` that’s already in the queue and you get the existing task back (`200` instead of `201`). A key holds as long as its task is kept, so retrying an enqueue after a timeout or a crash is safe. This deduplicates enqueues; deduplicating deliveries is your handler’s job (see [Delivery semantics](#delivery-semantics)).
## Attempts
Each delivery of a task is an **attempt**, recorded with its start and end, duration, status code, response headers, the first 4 KB of the response body, and how it ended:
| Outcome | |
| --------------- | ------------------------------------------------------------------------------------------- |
| `success` | A 2xx response. |
| `http_error` | Any other response. |
| `timeout` | No response within `timeout_seconds`. |
| `network_error` | The connection failed: DNS, refused, reset, TLS. |
| `lost` | Workly stopped mid-delivery, so the result is unknown. The endpoint may have done the work. |
This history is what the dashboard, the CLI and [MCP](https://workly.run/docs/mcp/) show when you ask why a task failed.
### Retries
* **Retried:** 5xx, 408 and 429 responses, timeouts, network errors and lost attempts.
* **Not retried:** other 4xx responses fail the task at once, since sending the same request again won’t change the answer. Set `retry_on_4xx` on the queue if your endpoint uses them for temporary problems. Connections the server refuses to make, such as to a private address on a server that only delivers to the public internet, aren’t retried either.
* **Redirects:** Workly doesn’t follow them. A 3xx response (or a 1xx) fails the task at once, since it usually means the URL is wrong: point the task at where the redirect goes.
* **Backoff:** retry *n* waits a random time between `min_backoff_seconds` and `min_backoff_seconds × backoff_multiplier^(n−1)`, capped at `max_backoff_seconds`. The randomness (jitter) spreads out a burst of failures. A `Retry-After` header on the response is honoured.
## Delivery semantics
Delivery is **at least once**. Workly never drops a task, but your endpoint can see one more than once:
* Workly can stop between sending a request and recording the response (a crash, a deploy, a lost connection to the database). The attempt is recorded as `lost` and the task is delivered again.
* Your endpoint can do the work and then time out, or fail to answer. Workly can’t tell that apart from not doing it.
So make handlers safe to run twice. Every delivery carries headers to help:
| Header | |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `Workly-Task-Id` | The task, the same on every attempt. Use it as a deduplication key. |
| `Workly-Queue` | The task’s queue. |
| `Workly-Attempt` | The attempt number, from 1. |
| `Workly-Timestamp`, `Workly-Signature` | The signature (see [Signing](#signing)). |
| `traceparent` | The trace context of the request that enqueued the task, if it had one, so your handler’s spans join that trace. |
Your own `headers` are sent too; `Workly-*` names are reserved. Schedule runs carry `Workly-Run-Id`, `Workly-Schedule` and `Workly-Scheduled-For` instead of the task headers ([schedules](https://workly.run/docs/schedules/#handling-a-run)), and a [workflow](https://workly.run/docs/workflows/#what-a-step-receives)’s steps carry `Workly-Workflow`, `Workly-Workflow-Run-Id`, `Workly-Workflow-Step`, `Workly-Workflow-Step-Id` and `Workly-Workflow-Run-Attempt` as well.
Order isn’t guaranteed. Tasks start roughly in the order they’re due, but with several deliveries at once, and with retries, they can finish in any order. If order matters, set `max_concurrency: 1`; a retry still lets later tasks go first.
## Signing
Every delivery is signed with a secret that Workly and your endpoint share (`WORKLY_SIGNING_SECRET` when you self-host; the API keys page on workly.run). Check the signature before trusting a request: anyone who can reach your endpoint can call it, but only Workly knows the secret.
The [SDKs](https://workly.run/docs/quickstart/#handle-it) check it with `verify_signature` in Python and `verifySignature` in TypeScript. In another language it’s a few lines.
### Verifying a signature without an SDK
A delivery carries two headers:
```text
Workly-Timestamp: 1790000000
Workly-Signature: v1=00f7255c88a43b3e2d47b211ff8e71cf935601cabff5e7adad0de2cc72a4e3c6
```
The signature is the hex HMAC-SHA256, keyed with the secret, of the timestamp, a `.`, and the request body exactly as it arrived:
```text
v1=hex(HMAC-SHA256(secret, "."))
```
To verify a request:
1. Read the **raw body**, before any JSON parsing: re-serializing it changes the bytes and breaks the signature.
2. Compute the HMAC of `timestamp + "." + body` with your secret.
3. Split `Workly-Signature` on commas. It holds more than one signature while the secret is being [rotated](https://workly.run/docs/configuration/#rotating-the-signing-secret); accept the request if any `v1=` value matches. Compare in constant time.
4. Reject the request if the timestamp is more than 5 minutes from your clock, so a captured request can’t be replayed later. The SDKs use the same tolerance.
The timestamp is the time of the attempt, so every retry has a fresh one and a new signature.
Test vector: the secret `secret`, the timestamp `1790000000` and the body `{"user_id":"123"}` give the signature above. Check it from a shell:
```sh
printf '%s' '1790000000.{"user_id":"123"}' | openssl dgst -sha256 -hmac secret
```
In Go:
```go
func verify(secret []byte, r *http.Request, body []byte) bool {
ts, err := strconv.ParseInt(r.Header.Get("Workly-Timestamp"), 10, 64)
if err != nil || time.Since(time.Unix(ts, 0)).Abs() > 5*time.Minute {
return false
}
mac := hmac.New(sha256.New, secret)
fmt.Fprintf(mac, "%d.", ts)
mac.Write(body)
want := mac.Sum(nil)
for _, part := range strings.Split(r.Header.Get("Workly-Signature"), ",") {
sig, ok := strings.CutPrefix(strings.TrimSpace(part), "v1=")
got, err := hex.DecodeString(sig)
if ok && err == nil && hmac.Equal(got, want) {
return true
}
}
return false
}
```
In Ruby (3.0 or later):
```ruby
require "openssl"
def verify(secret, headers, body)
ts = headers["Workly-Timestamp"].to_s
return false unless ts.match?(/\A\d+\z/) && (Time.now.to_i - ts.to_i).abs <= 300
want = "v1=" + OpenSSL::HMAC.hexdigest("SHA256", secret, "#{ts}.#{body}")
headers["Workly-Signature"].to_s.split(",").any? { |sig| OpenSSL.secure_compare(sig.strip, want) }
end
```
Answer a request that doesn’t verify with a 401. Workly doesn’t retry 4xx responses, so a wrong secret shows up at once as failed tasks with a 401, instead of an hour of retries.
# Schedules
> Call an HTTP endpoint on a cron timetable, in your timezone, with retries, signed requests, run history and a run-now button: cron for HTTP endpoints.
A schedule calls an HTTP endpoint on a cron timetable: a nightly report, a sync every 15 minutes, a weekly cleanup. Each call is a **run**, delivered, signed and retried like a task, with every attempt recorded. It’s cron for HTTP endpoints, with a history and a “run now” button.
Schedules are separate from queues: a schedule has its own settings (retries, timeout, what to do when runs overlap) and history, and no queue’s limits apply to it. Its only tasks are [callbacks](https://workly.run/docs/callbacks/), if you ask for them.
```text
Queue → Task → Attempt
Schedule → Run → Attempt
```
## Create a schedule
```sh
curl -X PUT localhost:7337/v1/schedules/nightly-report -d '{
"cron": "0 3 * * *",
"timezone": "Europe/Stockholm",
"url": "https://api.example.com/reports/nightly",
"payload": {"kind": "daily"}
}'
```
`PUT` is declarative, like queues: the schedule ends up with exactly the settings given, and omitted ones go back to their defaults. The response includes `next_runs`, the next five times it will run, so you can check the expression.
From code:
```python
client.schedules.put(
"nightly-report",
cron="0 3 * * *",
timezone="Europe/Stockholm",
url="https://api.example.com/reports/nightly",
payload={"kind": "daily"},
)
```
```ts
await workly.schedules.put('nightly-report', {
cron: '0 3 * * *',
timezone: 'Europe/Stockholm',
url: 'https://api.example.com/reports/nightly',
payload: { kind: 'daily' },
});
```
Or from the CLI:
```sh
workly schedules put nightly-report https://api.example.com/reports/nightly \
--cron '0 3 * * *' --timezone Europe/Stockholm -d '{"kind": "daily"}'
```
Or keep it in a `workly.yaml` and deploy it with `workly deploy` ([configuration as code](https://workly.run/docs/deploy/)):
```yaml
version: 1
schedules:
nightly-report:
cron: "0 3 * * *"
timezone: Europe/Stockholm
url: https://api.example.com/reports/nightly
payload: {kind: daily}
```
## Cron expressions
Five fields: minute, hour, day of month, month and day of week.
| Expression | Runs |
| ----------------- | ------------------------------------ |
| `*/15 * * * *` | every 15 minutes |
| `0 * * * *` | every hour, on the hour |
| `0 3 * * *` | every day at 03:00 |
| `0 9 * * mon-fri` | weekdays at 09:00 |
| `30 4 * * sun` | Sundays at 04:30 |
| `0 0 1 * *` | the first of every month at midnight |
Fields take `*`, numbers, lists (`1,15`), ranges (`1-5`), steps (`*/10`, `0-30/5`) and three-letter names for months and days (`jan`, `mon`). Day of week runs from 0 (Sunday) to 6, and 7 is Sunday too. The macros `@hourly`, `@daily`, `@weekly`, `@monthly` and `@yearly` work as usual. When both day fields are restricted, as in `0 0 13 * fri`, a day matches if either does (the 13th, or any Friday), as in classic cron. Seconds aren’t supported; the shortest interval is a minute.
The expression is read in the schedule’s `timezone`, an IANA name such as `Europe/Stockholm` (default `UTC`). Across daylight saving changes, a time in the skipped hour runs right after it (02:30 runs at 03:00), and a time in the repeated hour runs once.
## Settings
| Setting | Default | |
| ------------------------------------------------------------------ | --------------- | ------------------------------------------------------------------------------------------------- |
| `cron` | required | When to run. |
| `timezone` | `UTC` | |
| `url` | required | The endpoint to call. |
| `workflow` | none | Instead of `url`: start a run of this [workflow](https://workly.run/docs/workflows/), with `payload` as its input. |
| `method`, `headers`, `payload` or `body`, `content_type` | `POST` | The request, as for [tasks](https://workly.run/docs/quickstart/). |
| `timeout_seconds` | `30` | How long to wait for a response. |
| `max_attempts` | `3` | Attempts per run before it fails. |
| `min_backoff_seconds`, `max_backoff_seconds`, `backoff_multiplier` | `5`, `300`, `2` | Backoff between retries, as for queues. |
| `retry_on_4xx` | `false` | Retry 4xx responses other than 408 and 429. |
| `overlap` | `skip` | What to do when a run is due while the previous one is still going. |
| `missed` | `skip` | What to do about times missed while the server was down. |
| `on_complete` | none | A [callback](https://workly.run/docs/callbacks/), `{url, headers}`, called with a summary each time a run finishes. |
With `workflow`, a schedule sends no request of its own. Each run starts a workflow run, with the schedule run’s ID as its idempotency key, and succeeds once it has started; a workflow that doesn’t exist yet is retried like a failed request. The schedule run links to the workflow run (`workflow_run_id`), and `workflow_run_status` says whether it’s still going or how it ended. For `overlap`, the schedule counts as running until the workflow run finishes, so with `skip` a slow workflow never has two scheduled runs at once. `method`, `headers`, `body` and `content_type` don’t apply.
Retries are fewer and shorter than a queue’s by default, so a failing run gives up before a frequent schedule’s next one.
**Overlap.** With `skip`, a run that comes up while the previous run is still pending or running (or, for a schedule with `workflow`, while the workflow run it started is still going) is recorded as `skipped`, with `skip_reason: overlap`, and not delivered. With `allow`, it runs alongside. Runs you start by hand always run.
**Missed runs.** A run is missed if no server could start it within a minute of its time. `skip` records the most recent missed time as a `skipped` run with `skip_reason: missed` and carries on; `catch_up` runs it once. Either way, a server that was down for a day doesn’t make up every run it missed.
## Runs
Each run has a state: `pending`, `running`, `succeeded`, `failed`, `cancelled` or `skipped`. A run is created once per scheduled time, even if the server fails over at that moment; like tasks, its delivery is at least once.
```sh
workly runs ls --schedule nightly-report # newest first
workly runs ls --state failed # failed runs of every schedule
workly runs get # the run and its attempts
```
In the API, `GET /v1/runs` takes the same filters: `?schedule=nightly-report`, `?state=failed`. A run’s `started_at` is when its first attempt started, and `last_duration_ms` how long its latest attempt took.
A run keeps the request as it was when the run was created, so it shows what was sent even after the schedule changes. Retries use the schedule’s current retry settings.
* **Run now:** `POST /v1/schedules/{name}/trigger` (or `workly schedules trigger`, or the dashboard’s Run now) creates a run with `trigger: manual` and delivers it at once, even while the schedule is paused. It doesn’t change when the schedule next runs.
* **Retry a run:** `POST /v1/runs/{id}/retry` delivers a failed, cancelled, succeeded or skipped run again, with a fresh set of attempts. Retrying a skipped run delivers the time it skipped.
* **Pause:** `POST /v1/schedules/{name}/pause` stops new runs; runs already pending finish. A paused schedule has `paused_at`, no `next_run_at` and empty `next_runs`; `next_runs_if_resumed` are the times it would run if resumed now. Times that pass while it’s paused aren’t run.
* **Delete:** deleting a schedule deletes its runs.
Finished runs are kept for `WORKLY_RUN_RETENTION` (by default `WORKLY_RETENTION`, 7 days), except each schedule’s latest run, which is always kept, so a monthly job still shows how it last went.
## Handling a run
Your endpoint gets the request with these headers:
| Header | |
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `Workly-Run-Id` | The run, the same on every attempt. Use it to deduplicate. |
| `Workly-Schedule` | The schedule’s name. |
| `Workly-Scheduled-For` | The scheduled time (RFC 3339). Use it, not the clock, to decide which day or hour a late or retried run is for. |
| `Workly-Attempt` | The attempt number. |
| `Workly-Signature`, `Workly-Timestamp` | The signature, as for tasks: verify it with the SDKs’ `verify_signature` / `verifySignature`. |
Runs don’t carry `Workly-Task-Id` or `Workly-Queue`, so one handler can tell a scheduled call from a task if it receives both.
## Monitoring
The dashboard’s Schedules page shows each schedule’s next run, last run and recent runs; a schedule’s page has its upcoming times and every run with its attempts. Over [MCP](https://workly.run/docs/mcp/), `get_schedule` answers “why didn’t the 03:00 report run?” from the skipped and failed runs.
For alerts, `workly_schedule_last_success_timestamp_seconds` tells you when each schedule last succeeded; alert when it’s older than the schedule’s period. See [metrics](https://workly.run/docs/metrics/).
# Task groups
> Fan out tasks and get one callback when they've all finished, with counts of what succeeded and failed: fan-out and fan-in for an HTTP task queue.
A task group tracks a set of tasks and calls you back once all of them have finished. Enqueue a task per item and get one callback with how they went: fan-out and fan-in, without writing the bookkeeping yourself.
```text
create group → enqueue tasks into it → close it → … tasks finish … → callback task
```
A group is one execution, not a reusable definition: a nightly job creates a new group each night, and can give them all the same `name`. A group has no dependencies between its tasks, no chains and no nested groups. To run steps in order with a group in the middle, use a [workflow](https://workly.run/docs/workflows/): a step can hand its completion to a group with `Workly-Wait-For`, and the next step starts once the group has succeeded.
## Example: refresh stock prices, then rebuild the views
Every night, a [schedule](https://workly.run/docs/schedules/) calls `/jobs/start-price-update`. That endpoint enqueues one task per company into a group, and the group’s callback rebuilds the views that depend on the prices once every update has finished.
In Python, with Flask:
```python
from flask import Flask, abort, request
from workly import Callback, SignatureVerificationError, Workly, verify_signature
app = Flask(__name__)
workly = Workly()
API = "https://api.example.com"
SYMBOLS = ["NET", "DDOG", "SNOW", ...] # the companies to update
def verified() -> dict:
try:
verify_signature(request.get_data(), request.headers)
except SignatureVerificationError:
abort(401)
return request.get_json()
@app.post("/jobs/start-price-update")
def start_price_update():
verified()
day = request.headers["Workly-Scheduled-For"][:10]
# Creates the group, enqueues the tasks into it in batches and closes it. The
# idempotency key makes a retried run finish the same group rather than start another.
workly.groups.enqueue(
"stock-prices",
({"url": f"{API}/jobs/update-stock-price", "payload": {"symbol": s}} for s in SYMBOLS),
name="daily-stock-prices",
idempotency_key=f"stock-prices-{day}",
on_complete=Callback(url=f"{API}/jobs/refresh-views"),
)
return "", 204
@app.post("/jobs/update-stock-price")
def update_stock_price():
symbol = verified()["symbol"]
... # fetch and store the price
return "", 204
@app.post("/jobs/refresh-views")
def refresh_views():
summary = verified()
# {"group_id": "grp_…", "name": "daily-stock-prices", "status": "succeeded",
# "total": 500, "succeeded": 500, "failed": 0, "cancelled": 0}
if summary["status"] == "succeeded":
... # refresh the materialized views
else:
... # tell someone: summary["failed"] updates failed
return "", 204
```
The same in TypeScript:
```ts
import { Workly } from '@worklydev/sdk';
const workly = new Workly();
const API = 'https://api.example.com';
export async function startPriceUpdate(day: string, symbols: string[]) {
await workly.groups.enqueue(
'stock-prices',
symbols.map((symbol) => ({ url: `${API}/jobs/update-stock-price`, payload: { symbol } })),
{
name: 'daily-stock-prices',
idempotencyKey: `stock-prices-${day}`,
onComplete: { url: `${API}/jobs/refresh-views` },
},
);
}
```
`groups.enqueue` does three steps in one call: create the group, enqueue the tasks into it (in batches of 100, the most one request takes) and close it. With the group’s `idempotency_key` it’s safe to retry. The group is found again rather than created; a task without its own `idempotency_key` gets one from the group and its position, so none is counted twice; and a group that’s already closed is returned as it is. It needs SDK 0.4.3 or later. With an older SDK, or over HTTP, do the three steps yourself:
```sh
curl -X POST localhost:7337/v1/groups -d '{
"name": "daily-stock-prices",
"on_complete": {"url": "https://api.example.com/jobs/refresh-views"}
}'
# {"id": "grp_01J9…", "status": "open", "total": 0, …}
curl -X POST localhost:7337/v1/queues/stock-prices/tasks -d '{
"url": "https://api.example.com/jobs/update-stock-price",
"payload": {"symbol": "NET"},
"group_id": "grp_01J9…"
}'
curl -X POST localhost:7337/v1/groups/grp_01J9…/close
curl localhost:7337/v1/groups/grp_01J9…
```
Or from the CLI:
```sh
group=$(workly groups create --name daily-stock-prices \
--callback-url https://api.example.com/jobs/refresh-views)
workly tasks enqueue stock-prices https://api.example.com/jobs/update-stock-price -d '{"symbol": "NET"}' --group $group
workly groups close $group
workly groups get $group
```
To try it locally, point the URLs at `http://localhost:7337/_dev/echo`.
## Lifecycle
| Status | |
| ----------- | -------------------------------------------------------------- |
| `open` | Accepting tasks. Its tasks run as soon as they’re enqueued. |
| `running` | Closed, with tasks still pending or running. |
| `succeeded` | Every task succeeded. |
| `failed` | Every task finished, and at least one failed or was cancelled. |
* **Close the group once you’ve enqueued everything.** It can’t complete while it’s open, since more tasks may come, so you don’t need to know the number of tasks up front. Closing a closed group does nothing, so it’s safe to retry.
* A group whose tasks have all finished when it’s closed completes at once, and so does a closed group with no tasks.
* Tasks in a group are ordinary tasks, with their queue’s retries, limits and history. A task waiting to retry counts as remaining. A task that fails for good doesn’t stop the others.
* `succeeded` and `failed` are final. Once a group has completed, nothing changes it.
## The callback
`on_complete` is a [callback](https://workly.run/docs/callbacks/): a URL and optional headers, as for tasks and schedules. When the group completes, Workly enqueues a `POST` to the URL with the group’s summary as its JSON body:
```json
{
"group_id": "grp_01J9ZK3M5Q8R2T4V6W8X0Y2Z4A",
"name": "daily-stock-prices",
"status": "failed",
"total": 500,
"succeeded": 498,
"failed": 1,
"cancelled": 1
}
```
It’s enqueued exactly once: the group completes in the same database transaction that finishes its last task (or closes it), and that transaction enqueues the callback. A crash can’t lose it, and two completions racing can’t enqueue two.
From there it’s an ordinary task in the [`_callbacks` queue](https://workly.run/docs/callbacks/#the-_callbacks-queue): signed, retried and recorded, with the group’s `callback_task_id` pointing to it. Delivery is at least once, so make the handler safe to run twice; `group_id` identifies the group. Headers in `on_complete` are sent with it.
The callback fires whether the group succeeded or failed: check `status`. A group without `on_complete` still completes, and you can poll it instead.
The body carries counts, not the tasks’ results. To see which tasks failed, list them: `GET /v1/tasks?group_id=grp_…&state=failed`.
## Retries, cancellations and deleted queues
* **Retrying a task while its group is open or running** counts it as remaining again, so the group waits for the retry. That’s how to fix a failure before the group completes.
* **Retrying a task after its group completed** delivers the task again but doesn’t change the group. To redo the work as a group, create a new group.
* **Cancelling** a task counts it as cancelled, which makes the group `failed` once it completes.
* **Deleting a queue** takes its tasks out of their groups (`total` goes down), and a group may complete without them.
## Adding tasks safely
An enqueue checks that the group is open in the same transaction, so a task can’t slip in after the group closes and be missed. Enqueueing into a closed or completed group fails with `group_closed` (409), and into a missing one with `group_not_found` (404). In a batch, all the tasks are enqueued or none.
Make it safe to retry (`groups.enqueue` in the SDKs does all of this):
* Give tasks an `idempotency_key`. A task enqueued again with the same key isn’t counted twice, so you can resend a batch after a timeout.
* Give the group an `idempotency_key`. Creating it again returns the same group, so a retried request (or schedule run, as in the example) continues it rather than starting a second.
## Watching groups
The dashboard’s **Groups** page lists groups with their progress. A group’s page shows its counts, its callback and how it was delivered, and its tasks by state. `GET /v1/groups` lists groups newest first, filtered by `status` and `name`; `GET /v1/groups/{id}` returns one with `total`, `succeeded`, `failed`, `cancelled` and `remaining`. Agents get `list_groups` and `get_group` over [MCP](https://workly.run/docs/mcp/), and `list_tasks` takes a `group_id`.
Completed groups are deleted after `WORKLY_RETENTION`, like finished tasks. Their tasks may go first; the group’s counts stay accurate without them.
# Workflows
> Run HTTP steps in dependency order, each delivered and retried as a task, with steps that wait for a task group: DAG workflows for an HTTP task queue.
A workflow is a set of HTTP steps that run in dependency order: a step is sent once the steps it depends on have succeeded. Each step is an ordinary [task](https://workly.run/docs/concepts/), delivered, retried and signed like any other: a workflow adds ordering to tasks, not a new kind of worker.
```text
prices ─┐
├─→ analytics ─┐
fx ─────┘ ├─→ notify
news ──────────────────┘
```
A step can also hand its completion to a [task group](https://workly.run/docs/groups/): its endpoint starts the work, answers `202` with `Workly-Wait-For: group/`, and the step finishes when the group does. That is how one step fans out over hundreds of tasks and the next step waits for all of them.
A workflow is a static DAG declared up front: no conditions, loops, outputs passed between steps or nested workflows, and Workly never runs your code. If a step needs data from an earlier one, read it from where that step stored it.
## Define a workflow
Declare workflows in [`workly.yaml`](https://workly.run/docs/deploy/) and apply them with `workly deploy`:
```yaml
version: 1
workflows:
daily-refresh:
max_duration_seconds: 7200
steps:
- name: prices
url: https://api.example.com/jobs/start-price-update
- name: fx
url: https://api.example.com/jobs/refresh-fx
- name: news
url: https://api.example.com/jobs/import-news
queue: imports
- name: analytics
url: https://api.example.com/jobs/rebuild-analytics
depends_on: [prices, fx]
- name: notify
url: https://api.example.com/jobs/notify
depends_on: [analytics, news]
payload: {channel: "#data"}
schedules:
nightly-refresh:
cron: "0 3 * * *"
timezone: Europe/Stockholm
workflow: daily-refresh
```
Or with `PUT /v1/workflows/{name}` or `workflows.put` in the SDKs. Steps without `depends_on` start as soon as a run does: here `prices`, `fx` and `news` run in parallel, `analytics` once `prices` and `fx` have both succeeded, and `notify` once `analytics` and `news` have.
| Step setting | Default | |
| -------------------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------- |
| `name` | required | Unique within the workflow: up to 64 lowercase letters, digits, `-` and `_`. |
| `url` | required | The endpoint to call. |
| `depends_on` | none | The steps that must succeed first. |
| `method`, `headers`, `payload` or `body`, `content_type` | `POST` | The request, as for [tasks](https://workly.run/docs/quickstart/). Without a `payload` or `body`, the step is sent the run’s context (below). |
| `queue` | `_workflows` | The queue its task goes to, for that queue’s retries, timeout, rate and concurrency limits. |
| `max_attempts`, `timeout_seconds` | the queue’s | Overrides for this step. |
A workflow has at most 100 steps, which must form a DAG: Workly rejects unknown or self dependencies and names the cycle in one (`steps form a cycle: a → b → a`). `max_duration_seconds` (default a day) is how long a run may take before it fails.
Steps go to the `_workflows` queue unless they name another. Like [`_callbacks`](https://workly.run/docs/callbacks/), it’s created when first needed and configured like any queue, but you can’t enqueue into it. A step that names one of your queues shares that queue’s limits with its other tasks: use this to keep a workflow from overloading an endpoint that other work also calls.
Changing a workflow doesn’t touch runs already going: a run keeps the definition (and `version`) it started with.
## Run a workflow
```sh
workly workflows trigger daily-refresh --input '{"date": "2026-11-13"}'
```
```python
run = workly.workflows.trigger("daily-refresh", input={"date": "2026-11-13"}, idempotency_key="2026-11-13")
```
```typescript
const run = await workly.workflows.trigger('daily-refresh', { input: { date: '2026-11-13' }, idempotencyKey: '2026-11-13' });
```
Or with the dashboard’s Run now, or a [schedule](https://workly.run/docs/schedules/) with `workflow:` instead of `url:`, whose `payload` becomes each run’s input. An `idempotency_key` already used for a run of the workflow returns that run and starts nothing. A schedule uses its run’s ID as the key, so a retried schedule run never starts a second workflow run, and with `overlap: skip` (the default) it skips a slot while its last workflow run is still going.
## What a step receives
Each step’s task is delivered with the usual [headers](https://workly.run/docs/concepts/) plus:
| Header | |
| ----------------------------- | ------------------------------------------------------------------------------------------- |
| `Workly-Workflow` | The workflow’s name. |
| `Workly-Workflow-Run-Id` | The run, `wfr_…`. |
| `Workly-Workflow-Step` | The step’s name. |
| `Workly-Workflow-Step-Id` | The step, `wfs_…`: the same on every delivery of the step, so use it as an idempotency key. |
| `Workly-Workflow-Run-Attempt` | `1`, then one more each time the run is [retried](#retry-a-run). |
A step without a body of its own is sent the run’s context as JSON:
```json
{
"workflow_run_id": "wfr_01M7AH8EP90ZQ4E9228337GGVR",
"workflow": "daily-refresh",
"step": "analytics",
"step_id": "wfs_01M7AH8EPATE4EPK37FWTA5TBQ",
"run_attempt": 1,
"input": {"date": "2026-11-13"}
}
```
A 2xx response succeeds the step. Anything else is retried with its queue’s policy, and a step whose task fails (out of attempts, or a 4xx that isn’t retried) fails the run.
## Waiting for a group
To hand a step’s completion to a group, answer 2xx (typically `202`) with `Workly-Wait-For: group/`. The step stays `waiting` until the group completes, then succeeds if every task in the group succeeded, or fails, failing the run, if any failed or was cancelled. Close the group once its tasks are enqueued, or it never completes.
```python
@app.post("/jobs/start-price-update")
def start_price_update():
try:
verify_signature(request.get_data(), request.headers)
except SignatureVerificationError:
abort(401) # a 4xx isn't retried
# The same on every delivery of this step in this run attempt: a redelivered
# request reuses its group, and a retried run starts a fresh one.
key = f'{request.headers["Workly-Workflow-Step-Id"]}-{request.headers["Workly-Workflow-Run-Attempt"]}'
# Creates the group, enqueues the tasks into it and closes it; safe to retry.
group = workly.groups.enqueue(
"stock-prices",
({"url": f"{API}/jobs/update-stock-price", "payload": {"symbol": s}} for s in SYMBOLS),
name="daily-stock-prices",
idempotency_key=key,
)
return "", 202, {"Workly-Wait-For": f"group/{group.id}"}
```
The header names exactly one group in the workspace. A header Workly can’t use (another resource type, two groups, a group that doesn’t exist) fails the step at once, with the reason, rather than retrying. If the group has already completed, the step finishes at once.
This is safe across restarts and replicas: the step’s completion is recorded in the same database transaction as the group’s, so a step can’t miss its group or finish twice.
## When a step fails
A run stops at its first failed step:
* steps still blocked are `skipped`,
* step tasks not yet delivered are cancelled and their steps `skipped`,
* steps waiting for a group are `skipped`; the group itself goes on, and its tasks still run,
* requests already in flight finish, and their steps record how they went, but start nothing.
The run is `failed`, with the step’s error. A run still going after `max_duration_seconds` fails the same way, except that its waiting steps are `failed`, since they didn’t finish in time. Cancelling a run (`workly workflows cancel`, or the dashboard) skips what hasn’t started.
Step tasks can’t be retried one at a time: [retry the run](#retry-a-run) instead. A step’s task names its step and run (`workflow_step` in the API, and a link on its dashboard page), so a failure found in a queue leads back to its run.
## Retry a run
Once you’ve fixed what failed, retry the run rather than starting over: the dashboard’s Retry run, `workly workflows retry ID`, `workflow_runs.retry(id)` in the SDKs or `retry_workflow_run` over [MCP](https://workly.run/docs/mcp/). A failed or cancelled run goes back to `running`, with the definition and input it started with, even if the workflow has changed since:
* steps that succeeded stay succeeded, and aren’t sent again,
* failed and skipped steps run again once their dependencies have succeeded, each on the task it had, with a fresh set of attempts,
* a step that was waiting for a group follows the group: it succeeds if the group has succeeded since, waits again if it is still going, and runs again if it failed.
The run’s `attempt` goes up by one, and its `max_duration_seconds` starts over. Steps get the attempt as `Workly-Workflow-Run-Attempt`, so work keyed on the step ID and the attempt, like the group above, is redone on a retry rather than reusing what failed. To run a changed definition, start a new run.
## Statuses
| Step | |
| --------------------- | ------------------------------------------------------------- |
| `blocked` | Waiting for the steps it depends on. |
| `pending` | Its task is queued, or waiting to retry. |
| `running` | Its request is being delivered. |
| `waiting` | Its request succeeded, and it waits for the group it named. |
| `succeeded`, `failed` | Finished. |
| `skipped` | Never ran, or was waiting for a group: the run stopped first. |
A run is `running`, then `succeeded`, `failed` or `cancelled`. `GET /v1/workflow-runs/{id}` (`workly workflows run ID`, `get_workflow_run` over [MCP](https://workly.run/docs/mcp/)) shows each step with its task, its last status and error, what it is blocked by or waiting for, and how long each phase took.
## Why hasn’t my run finished?
Open the run in the dashboard, or `workly workflows run ID`. Each step says why it is where it is:
* **Blocked:** the steps named in “waiting for” haven’t succeeded yet; look at those.
* **Pending with attempts:** its endpoint is failing and the task is retrying; the task page has every attempt’s response.
* **Pending without attempts:** its queue is paused, at its concurrency limit or behind; look at the queue.
* **Waiting for a group:** the group’s tasks are still going, or the group was never closed. The group page shows which.
## Delivery is at least once
Steps are tasks, so a step’s request can arrive more than once, for example when a server stops mid-delivery. Workly starts each step’s task once per run, but your endpoints should still treat `Workly-Workflow-Step-Id` as an idempotency key, as the example above does with its group.
## Retention
Finished runs are deleted after `WORKLY_RUN_RETENTION`, like schedule runs ([configuration](https://workly.run/docs/configuration/)); a workflow’s latest run is always kept. Step tasks are deleted with other tasks, so an old run’s steps may no longer link to theirs.
# Callbacks
> Have Workly POST a summary to your endpoint each time a task or scheduled run succeeds, fails or is cancelled: callbacks for an HTTP task queue.
A callback tells your app how something went once it has finished: a task, a schedule’s run, or a [task group](https://workly.run/docs/groups/). Give it an `on_complete` URL and, each time it finishes, Workly sends a `POST` there with a JSON summary: whether it succeeded, how many attempts it took, and the last status code and error.
```text
task or run finishes → callback task in _callbacks → POST to your on_complete URL
```
Use callbacks to react to a result without polling: mark an export as done, email the user that their report failed, or start the next step once this one has finished. To be told when something needs a human, use [alerts](https://workly.run/docs/alerts/) instead.
## On a task
```python
from workly import Callback, Workly
client = Workly()
client.tasks.enqueue(
"exports",
url="https://api.example.com/jobs/export",
payload={"export_id": 42},
idempotency_key="export-42",
on_complete=Callback(url="https://api.example.com/jobs/export-done"),
)
```
```ts
await workly.tasks.enqueue('exports', {
url: 'https://api.example.com/jobs/export',
payload: { exportId: 42 },
idempotencyKey: 'export-42',
onComplete: { url: 'https://api.example.com/jobs/export-done' },
});
```
```sh
curl -X POST localhost:7337/v1/queues/exports/tasks -d '{
"url": "https://api.example.com/jobs/export",
"payload": {"export_id": 42},
"idempotency_key": "export-42",
"on_complete": {"url": "https://api.example.com/jobs/export-done"}
}'
```
The CLI takes `--on-complete URL`, and `enqueue_task` over [MCP](https://workly.run/docs/mcp/) takes `on_complete`. When the task succeeds, fails for good or is cancelled, the callback gets:
```json
{
"task_id": "task_01J9ZK3M5Q8R2T4V6W8X0Y2Z4A",
"queue": "exports",
"state": "failed",
"attempt_count": 5,
"last_status": 500,
"last_outcome": "http_error",
"last_error": "HTTP 500 Internal Server Error",
"idempotency_key": "export-42",
"finished_at": "2026-11-12T10:15:02.114Z"
}
```
`last_error` is present only when the last attempt failed, so it’s absent when the task succeeded. `last_status`, `last_outcome` and `last_error` are all absent when there was no attempt (a task cancelled before it ran), and `idempotency_key` and `group_id` when the task has none. The body carries the outcome, not the response: to see what the endpoint answered, `GET /v1/tasks/{task_id}/attempts`.
## On a schedule
A schedule’s `on_complete` is called for each of its runs:
workly.yaml
```yaml
version: 1
schedules:
nightly-report:
cron: "0 3 * * *"
timezone: Europe/Stockholm
url: https://api.example.com/reports/nightly
on_complete:
url: https://api.example.com/reports/nightly-done
```
It can also be set with `PUT /v1/schedules/{name}`, `client.schedules.put(…, on_complete=…)`, `workly schedules put --on-complete`, `update_schedule` over MCP, or on the schedule’s page in the dashboard. The body:
```json
{
"run_id": "run_01J9ZK3M5Q8R2T4V6W8X0Y2Z4A",
"schedule": "nightly-report",
"state": "succeeded",
"trigger": "schedule",
"scheduled_for": "2026-11-12T02:00:00Z",
"attempt_count": 1,
"last_status": 200,
"last_outcome": "success",
"finished_at": "2026-11-12T02:00:01.873Z"
}
```
The `last_*` fields follow the same rules as a task’s: a failed run adds `last_error` (such as `"last_error": "HTTP 500 Internal Server Error"`), a run that succeeded has none, and a run cancelled before its first attempt has no `last_*` fields at all.
Skipped runs (an overlap, or a slot missed while the server was down) were never delivered and don’t call back. The schedule’s current `on_complete` applies when a run finishes, so a change takes effect for runs already in progress.
## When a callback fires
* **Once each time it finishes.** A task or run calls back when it succeeds, when its last attempt fails or it gets a response Workly doesn’t retry, and when it is cancelled. Attempts that will be retried don’t.
* **Again after a retry by hand.** A retried task or run gets a fresh set of attempts and calls back again when it finishes, with the new outcome, so a fix-and-retry tells your app it finally worked.
* **Not for deleted tasks.** Deleting a queue deletes its tasks without calling back.
* **Groups call back once.** A [group’s](https://workly.run/docs/groups/) callback fires when the group completes, with its counts, and never again.
The callback is enqueued in the same database transaction that finishes the task or run, so a crash can’t lose one, and a task cancelled mid-attempt calls back once, when it’s cancelled.
## The `_callbacks` queue
Every callback is a task in `_callbacks`, a queue Workly creates when it first needs one. Only Workly enqueues into it, and callbacks never have callbacks of their own. Otherwise a callback is an ordinary task: signed, retried with the queue’s backoff, and recorded with every attempt. The task, run or group links to it (`callback_task_id`, the latest one), and the dashboard shows it next to them. A schedule’s runs also carry that task’s state (`callback_state`), so a list of runs says which callbacks were delivered without fetching each one.
`_callbacks` has the default settings. To give callbacks more retries or a longer timeout, configure it like any queue, e.g. in [`workly.yaml`](https://workly.run/docs/deploy/):
```yaml
queues:
_callbacks:
max_attempts: 10
timeout_seconds: 300
```
If it has been deleted, it’s created again with default settings, so no callback is dropped.
## Handling a callback
A callback is delivered like any task, so the handler is an ordinary endpoint: [verify the signature](https://workly.run/docs/concepts/#signing), read the summary, and answer 2xx. Headers in `on_complete` are sent with it, e.g. to route it or authenticate it:
```json
"on_complete": {"url": "https://api.example.com/jobs/done", "headers": {"X-Job": "export"}}
```
Delivery is at least once, so make the handler safe to run twice. The callback’s `Workly-Task-Id` stays the same across its retries; the summary’s `task_id` (or `run_id`) with `finished_at` identifies which finish it is about, since a task retried by hand calls back once per finish.
# Alerts
> Get an email or a Slack message when Workly tasks or schedule runs fail, so you don't have to watch the dashboard.
Workly can tell you when tasks or schedule runs fail, so you don’t have to watch the dashboard. Alerts go to **channels**: email addresses and Slack incoming webhooks. Add them on the dashboard’s Alerts page or through the API.
## What you get
A task or run counts as failed once it has used up its attempts, or got a response Workly doesn’t retry. Retries that later succeed, and cancelled tasks, never alert.
* Within about a minute of a failure, every channel gets an alert. It names the queue or schedule, how many failed and the most common errors, and links to the failures in the dashboard.
* After that, a queue or schedule gets **at most one alert every 30 minutes**, covering everything that failed since its previous alert. A queue that keeps failing reminds you every half hour instead of once per task.
* Failures in several queues and schedules at once arrive as one alert.
Alerts cover failures from the moment the first channel is added; earlier ones aren’t news.
A queue with [auto-pause](https://workly.run/docs/concepts/#auto-pause) also alerts at once when it pauses itself, with the error that paused it, and when it resumes, with how long it was paused. These two don’t wait for the 30 minutes.
## Slack
In Slack, create an [incoming webhook](https://api.slack.com/messaging/webhooks) for the channel alerts should go to, then add its URL:
```sh
curl -X POST localhost:7337/v1/alert-channels -d '{
"type": "slack",
"target": "https://hooks.slack.com/services/T000/B000/XXXX"
}'
```
The webhook URL is a secret: the API and the dashboard only show its end.
## Email
```sh
curl -X POST localhost:7337/v1/alert-channels -d '{"type": "email", "target": "oncall@example.com"}'
```
A self-hosted server sends email through your SMTP server, set with `WORKLY_SMTP_URL` and `WORKLY_SMTP_FROM` (see [Configuration](https://workly.run/docs/configuration/)). Without them, `email_enabled` is `false` in `GET /v1/alert-channels` and adding an email channel fails with `email_unavailable`. Slack channels need no setup.
On workly.run, email alerts go to members of the workspace.
## Check a channel
Send a test alert to make sure a channel works:
```sh
curl -X POST localhost:7337/v1/alert-channels/ach_01J9ZK3M5Q8R2T4V6W8X0Y2Z4A/test
```
If it can’t be sent, the error says why (`alert_failed`), e.g. Slack’s answer for a removed webhook.
From code:
```python
channel = client.alerts.create("slack", "https://hooks.slack.com/services/T000/B000/XXXX")
client.alerts.test(channel.id)
```
```ts
const channel = await workly.alerts.create('slack', 'https://hooks.slack.com/services/T000/B000/XXXX');
await workly.alerts.test(channel.id);
```
## Links
Alerts link to the dashboard when the server knows its address. Set `WORKLY_PUBLIC_URL` to where people open Workly, e.g. `https://workly.example.com`. `workly dev` links to `http://localhost:7337`.
## Delivery
Alerts are sent by the replica that delivers tasks, so a cluster sends each alert once. If every channel fails to take an alert (Slack is down, say), the next check, a minute later, tries again. A channel can get an alert twice if the server stops just after sending it.
# Configuration as code
> Keep Workly queues, workflows and schedules in a workly.yaml next to your code: validate it in CI, see the plan, and apply it with workly deploy.
Keep your queues, workflows and schedules in a `workly.yaml` next to your code, and deploy them with one command:
```sh
workly deploy
```
`workly deploy` compares the file with the server, shows what it will change, and creates and updates queues, workflows and schedules until the server matches. Running it again changes nothing, so it can run on every deploy of your app.
## Example
A nightly job that refreshes stock prices (see [task groups](https://workly.run/docs/groups/) for the job itself) and a morning refresh of financial data:
```yaml
version: 1
queues:
stock-prices:
max_concurrency: 10
rate_per_second: 2 # the price API allows 120 calls a minute
max_attempts: 3
financial-data:
max_concurrency: 5
timeout_seconds: 120
schedules:
daily-stock-prices:
cron: "0 22 * * 1-5"
timezone: Europe/Stockholm
url: ${API_URL}/jobs/start-price-update
headers:
Authorization: Bearer ${JOBS_TOKEN}
refresh-financial-data:
cron: "0 6 * * *"
timezone: Europe/Stockholm
url: ${API_URL}/jobs/refresh-financial-data
payload:
full: true
```
Check it, see the plan, deploy it:
```console
$ workly validate
workly.yaml is valid: 2 queues, 2 schedules.
$ workly plan
Plan for workly.yaml on https://api.workly.run:
+ create queue stock-prices
~ update queue financial-data
max_concurrency: 3 → 5
timeout_seconds: 30 → 120
+ create schedule daily-stock-prices
~ update schedule refresh-financial-data
cron: 0 5 * * * → 0 6 * * *
headers.Authorization: changed
2 to create, 2 to update, 0 unchanged.
$ workly deploy
…
Apply these changes to https://api.workly.run? [y/N] y
Created queue stock-prices
Updated queue financial-data
Created schedule daily-stock-prices
Updated schedule refresh-financial-data
Deployed workly.yaml: 2 created, 2 updated.
```
## Commands
| Command | |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `workly validate` | Checks the file without contacting the server: field names and types, cron expressions, timezones, URLs, headers, and that every `${NAME}` is set. It runs the server’s own validation, so a file that passes is one the server accepts. |
| `workly plan` | Shows what `deploy` would create, update and (with [`prune`](#pruning)) delete, setting by setting, and which `${NAME:-default}` values took their default. Changes nothing. |
| `workly deploy` | Shows the plan, asks, and applies it. `--yes` (`-y`) skips the question, as CI needs; without a terminal to ask on, it’s required. |
All three read `workly.yaml` in the current directory, or the file given with `--file` (`-f`). `plan` and `deploy` connect like the other commands: `--url` or `WORKLY_URL`, and `--api-key` or `WORKLY_API_KEY`. They exit with a non-zero status on an invalid file or a failed change; a deploy that fails partway reports what it had applied, and running it again finishes the job.
## The file
| Key | |
| ----------- | ------------------------------------------------------------------------------------------------------ |
| `version` | Required. `1`. |
| `prune` | `true` to delete what the file leaves out. See [Pruning](#pruning). Default `false`. |
| `queues` | Queue names to their settings. |
| `workflows` | Workflow names to their `steps` (a list) and `max_duration_seconds`. |
| `schedules` | Schedule names to their settings. |
| `x-…` | Ignored, so a key such as `x-defaults` can hold [anchors](#sharing-settings) for the rest of the file. |
Settings are the fields of the `PUT` requests in the [API](https://workly.run/docs/api/), with the same names, defaults and limits:
* **Queues:** `max_attempts`, `min_backoff_seconds`, `max_backoff_seconds`, `backoff_multiplier`, `timeout_seconds`, `retry_on_4xx`, `rate_per_second`, `max_concurrency` and `auto_pause` (`{after_failures: n}`, see [Auto-pause](https://workly.run/docs/concepts/#auto-pause)).
* **Workflows:** `steps`, each with the settings in [Workflows](https://workly.run/docs/workflows/#define-a-workflow), and `max_duration_seconds`.
* **Schedules:** `cron` and `url` or `workflow` (required), `timezone`, `method`, `headers`, `payload` or `body` and `content_type`, the retry settings, `overlap` and `missed` (see [Schedules](https://workly.run/docs/schedules/)).
A `payload` can be any YAML, and is sent as JSON.
Settings work as `PUT` does: **a setting left out of the file gets its default**, not whatever it was before. The file describes each queue and schedule completely, so a change made in the dashboard to something the file declares is undone by the next deploy. To change it for good, change the file.
### Environment variables
`${NAME}` in a value is replaced with the environment variable `NAME`, so one file can serve several environments and keep secrets out of the repository. A variable that isn’t set is an error, not an empty string. An unquoted value is read again after the replacement, so `max_concurrency: ${CONCURRENCY}` is a number.
`${NAME:-default}` uses `default` when `NAME` is unset or empty, as in the shell, so one file can work locally without exporting anything:
```yaml
url: ${API_URL:-http://localhost:8000}/jobs/nightly
```
`validate`, `plan` and `deploy` list every default they used, so a variable CI forgot to set shows up in the plan instead of quietly sending production schedules to `localhost`:
```console
Not set, so their defaults were used:
API_URL: "http://localhost:8000" (schedules.daily-stock-prices.url, schedules.refresh-financial-data.url)
```
Keep secrets as plain `${NAME}`, without a default, so a missing one stops the deploy.
The plan never shows header values or bodies, only that they changed, since they often carry secrets.
### Sharing settings
YAML anchors (`&name`), aliases (`*name`) and merge keys (`<<`) work, so queues and schedules can share settings. Top-level keys starting with `x-` are ignored, which makes them a place to keep them:
```yaml
version: 1
x-price-api: &price-api
max_concurrency: 2
rate_per_second: 1 # one API key, shared by these queues
queues:
stock-prices: *price-api
earnings-dates:
<<: *price-api
max_attempts: 10 # keys given here win over merged ones
```
## Locally
`workly dev --config workly.yaml` applies the file each time the dev server starts, so a fresh database starts with your queues’ limits and there’s no separate deploy step. Restart it to apply changes. With `${NAME:-default}` in the URLs, the same file serves local and production.
## What deploying changes, and what it doesn’t
* It **creates** queues, workflows and schedules the file declares that the server doesn’t have, and **updates** those whose settings differ; one that already matches isn’t touched, so its `updated_at` stays put. Workflows go before schedules, so a schedule can start one declared in the same file. A workflow run already going keeps the definition it started with.
* It **doesn’t delete** anything unless the file sets [`prune: true`](#pruning). The plan lists the workflows and schedules on the server that aren’t in the file, so you notice ones you meant to remove (`workly schedules rm NAME`, `workly workflows rm NAME`). Queues aren’t listed: enqueueing creates them, so most have never been in a file.
* It **never pauses or resumes**. You pause during an incident, and a deploy shouldn’t undo that: a paused queue or schedule stays paused, and the plan says so.
* A section left out of the file isn’t managed by it: a file without `schedules` doesn’t look at schedules at all. An empty one (`schedules: {}`) declares none, which only changes what the plan lists, or with `prune`, deletes every schedule.
### Pruning
With `prune: true` at the top of the file, each section it has is the whole truth: deploying also deletes the queues, workflows and schedules on the server that the section leaves out, whoever created them. The plan lists them:
```plaintext
Plan for workly.yaml on https://api.workly.run:
- delete schedule old-report
- delete workflow old-refresh
- delete queue old-emails (and its 3 failed tasks)
0 to create, 0 to update, 3 to delete, 4 unchanged.
```
* A section left out of the file is still not managed: a file with `prune: true` but no `schedules` leaves every schedule alone.
* Deleting a schedule deletes its runs. Deleting a workflow keeps its runs, and those still going finish.
* Deleting a queue deletes its tasks, so a queue the file leaves out must have nothing left to deliver. If one has tasks pending, scheduled or running, the plan fails and names it, and nothing is deployed: add it under `queues`, or let its tasks finish (or `workly queues cancel-pending NAME`). Its failed tasks are deleted with it, and the plan says how many. Workly’s own queues (`_workflows`, `_callbacks`) are never deleted.
* Since enqueueing creates a queue, every queue your app enqueues to must be in the file, or the next deploy fails while it has work and deletes it when it hasn’t.
* Deletes happen after the creates and updates: schedules first, then workflows, then queues, so nothing left points at something gone.
In CI, `deploy --yes` deletes without asking, so a queue or schedule removed from the file in a pull request is gone once it merges. [Run `workly plan` on pull requests](#review-the-plan-in-pull-requests) so the deletes are part of the review.
## In CI
Deploy the file with your app. With GitHub Actions:
```yaml
- name: Deploy Workly configuration
run: |
curl -fsSL "https://raw.githubusercontent.com/worklydev/workly/$WORKLY_VERSION/install.sh" | sh
workly deploy --yes
env:
WORKLY_VERSION: v0.5.0 # pin the CLI, and bump it like any other dependency
WORKLY_URL: https://api.workly.run
WORKLY_API_KEY: ${{ secrets.WORKLY_API_KEY }}
API_URL: https://api.example.com
JOBS_TOKEN: ${{ secrets.JOBS_TOKEN }}
```
Run it once the app it calls is live: a new schedule or workflow step can be delivered as soon as it’s deployed, and if its endpoint isn’t out yet, those deliveries fail and retry. A step after your app’s deploy step in the same job does this. If your host deploys on its own and reports the deployment to GitHub, as Vercel does, run it on the `deployment_status` event once the production deployment has succeeded:
```yaml
on: deployment_status
jobs:
workly:
if: github.event.deployment_status.state == 'success' && github.event.deployment.environment == 'Production'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4 # the deployed commit
- name: Deploy Workly configuration
run: |
curl -fsSL "https://raw.githubusercontent.com/worklydev/workly/$WORKLY_VERSION/install.sh" | sh
workly deploy --yes
env:
WORKLY_VERSION: v0.5.0
WORKLY_URL: https://api.workly.run
WORKLY_API_KEY: ${{ secrets.WORKLY_API_KEY }}
```
The environment’s name is whatever your host calls it (Vercel’s is `Production`).
GitHub starts this workflow for every status the host reports. Railway, for example, reports `in_progress` and then `success`, so each deploy starts two runs: the first shows as skipped, and the one for `success` deploys.
In a Python project, `workly-cli==0.5.0` in your dev or CI requirements installs the same pinned binary with `pip` (it’s what `workly[cli]` pulls in).
Run `workly validate` in pull requests to catch mistakes before they merge. It needs no server or API key, only the environment variables the file uses.
### Review the plan in pull requests
`workly plan` shows what merging would change. `--format markdown` writes it for a pull request comment or the job’s summary page, with deletes called out at the top, and `--detailed-exitcode` exits 0 when nothing would change, 2 when something would, and 3 when that includes deletes (errors exit 1, as always). With GitHub Actions:
```yaml
on: pull_request
jobs:
workly-plan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Plan Workly configuration
run: |
curl -fsSL "https://raw.githubusercontent.com/worklydev/workly/$WORKLY_VERSION/install.sh" | sh
code=0
workly plan --format markdown --detailed-exitcode >> "$GITHUB_STEP_SUMMARY" || code=$?
# 0: no changes, 2: changes, 3: changes that delete something.
if [ "$code" = 3 ]; then
echo "::warning::This pull request deletes queues, workflows or schedules on merge. See the plan in the job summary."
elif [ "$code" != 0 ] && [ "$code" != 2 ]; then
exit "$code"
fi
env:
WORKLY_VERSION: v0.5.0
WORKLY_URL: https://api.workly.run
WORKLY_API_KEY: ${{ secrets.WORKLY_API_KEY }}
```
To make a pull request that deletes something wait for a closer look, fail the step on 3 instead (`exit 1`) and make the check required.
For environments that differ by more than a few values, keep a file per environment and pick one with `--file`:
```sh
workly deploy --file workly.staging.yaml --yes
```
# Self-hosting
> Run Workly in production as one container plus Postgres, on Docker, Docker Compose or Kubernetes, next to the apps it delivers to.
In production, Workly is one binary (`workly serve`) plus Postgres. Run it as a container next to your apps; endpoints on a private network can be delivered to directly.
It needs two settings, a database and a signing secret; the rest is in [Configuration](https://workly.run/docs/configuration/). Set `WORKLY_API_KEY` too, unless nothing untrusted can reach the server.
## Docker
The image is `ghcr.io/worklydev/workly` (linux/amd64 and arm64). It runs `workly serve` as a non-root user; pin a version tag (`X.Y.Z`) in production.
```sh
docker run -p 7337:7337 \
-e WORKLY_DATABASE_URL=postgres://user:pass@db:5432/workly \
-e WORKLY_SIGNING_SECRET=... \
-e WORKLY_API_KEY=... \
ghcr.io/worklydev/workly:0.5.0
```
To run Workly and Postgres together on one machine, use [`deploy/docker/compose.yaml`](https://github.com/worklydev/workly/blob/v0.5.0/deploy/docker/compose.yaml):
```sh
export WORKLY_SIGNING_SECRET=$(openssl rand -hex 32) WORKLY_API_KEY=$(openssl rand -hex 32)
docker compose -f deploy/docker/compose.yaml up -d
```
The image has a health check (`workly healthcheck`, which probes `/readyz`), so Compose and Docker report when the server is ready.
## Kubernetes
[`deploy/kubernetes/workly.yaml`](https://github.com/worklydev/workly/blob/v0.5.0/deploy/kubernetes/workly.yaml) is a starting point: a Deployment with two replicas, a Service and a PodDisruptionBudget. It reads its settings from a Secret:
```sh
kubectl create secret generic workly \
--from-literal=database-url='postgres://user:pass@host:5432/workly?sslmode=require' \
--from-literal=signing-secret="$(openssl rand -hex 32)" \
--from-literal=api-key="$(openssl rand -hex 32)"
kubectl apply -f deploy/kubernetes/workly.yaml
```
What it sets up, and why:
* **Two replicas, one dispatcher.** Every replica serves the API and dashboard, but only the leader (chosen with a Postgres advisory lock) delivers tasks. If the leader’s pod goes away, a standby takes over within seconds (as soon as Postgres sees its connection close); deliveries it had in flight are retried.
* **Probes.** Readiness is `/readyz` (the database is reachable and migrated), liveness `/healthz`.
* **Shutdown.** On `SIGTERM`, in-flight deliveries get `WORKLY_SHUTDOWN_TIMEOUT` (25s) to finish, within the default 30s `terminationGracePeriodSeconds`. If your queues have long timeouts, raise both. An interrupted delivery isn’t lost: it’s recorded as `lost` and retried.
* **Metrics.** `WORKLY_METRICS_PORT=9090` serves `/metrics` without the API key on a port the Service doesn’t expose, and the pods carry `prometheus.io/*` scrape annotations. See [Metrics](https://workly.run/docs/metrics/) for the metrics and example alerts.
* **Hardening.** Non-root, read-only root filesystem, no capabilities. Workly writes nothing to disk with Postgres.
To expose the dashboard outside the cluster, add an Ingress for the `workly` Service on port 7337, and keep `WORKLY_API_KEY` set: the dashboard asks for it. The same Ingress serves `/mcp`, so developers can point their coding agents at the cluster’s Workly ([MCP](https://workly.run/docs/mcp/)); it’s read-only unless you set `WORKLY_MCP=full`.
The dashboard updates live over a long-lived stream, `GET /v1/events` (server-sent events), which replicas feed each other through Postgres `LISTEN`/`NOTIFY`. If your Ingress or proxy buffers responses or closes idle connections in under 30 seconds, the dashboard falls back to polling every few seconds; the stream sends a heartbeat every 15 seconds.
Endpoints inside the cluster can be targeted directly, for example `http://billing.default.svc.cluster.local/tasks/invoice`.
## Upgrades
New versions migrate the database on startup; replicas starting together take turns. Check the release notes before upgrading across several versions. To migrate separately, for example from a CI job, set `WORKLY_AUTO_MIGRATE=false` and run `workly migrate` (with `WORKLY_DATABASE_URL` set) before rolling out. Until the database is migrated, new replicas report not ready.
## Backups
All state is in Postgres: back it up like any other database. Finished tasks are deleted after `WORKLY_RETENTION` (7 days), so the database stays about the size of your backlog plus a week of history.
# MCP: Workly for coding agents
> Connect Claude Code, Cursor, VS Code or Codex to Workly over MCP, so coding agents can enqueue test tasks, read failures and retry them.
Every Workly server serves [MCP](https://modelcontextprotocol.io) over HTTP at `/mcp`, so there’s nothing to install. Agents such as Claude Code, Codex, Cursor and VS Code use it to read queues, tasks and schedules, replay a delivery against a local handler, and retry what failed.
## Connect
Paste this into your agent and it connects itself:
```prompt
Connect this app to Workly over MCP. Workly (https://workly.run) is an HTTP task queue and scheduler: it delivers HTTP requests to my endpoints with retries, runs cron schedules, and records every attempt.
1. Add Workly's MCP server (streamable HTTP) with this app's own way of adding a remote MCP server. Use the endpoint that fits this project, or ask me which:
- Hosted Workly: https://api.workly.run/mcp. It signs in with OAuth in the browser; no API key.
- `workly dev` on this machine: http://localhost:7337/mcp. No sign-in.
- A self-hosted server: https:///mcp, with the server's WORKLY_API_KEY as a bearer token. Read the key from an environment variable; don't write it into the config.
2. If it needs a sign-in, tell me how to start it in this app.
3. Once connected, call get_docs to learn how Workly works, then help me with what I'm building.
If this app can't connect to remote MCP servers, read https://workly.run/llms.txt and use Workly's HTTP API instead.
```
Or set it up yourself. How an agent signs in depends on where Workly runs:
| Server | Endpoint | Sign-in |
| -------------------------------------------------------------- | ---------------------------- | ----------------------------------- |
| [Hosted: workly.run](#hosted-worklyrun) | `https://api.workly.run/mcp` | OAuth in the browser, or an API key |
| [Local dev server: `workly dev`](#local-dev-server-workly-dev) | `http://localhost:7337/mcp` | None |
| [Self-hosted](#self-hosted) | `https:///mcp` | `WORKLY_API_KEY` as a bearer token |
### Hosted: workly.run
Add the server, then sign in. The console opens in your browser: sign in or create an account, pick a workspace, and choose read-only or read and write.
Claude Code
```sh
claude mcp add --transport http --scope user workly https://api.workly.run/mcp
claude mcp login workly
```
Codex
```sh
codex mcp add workly --url https://api.workly.run/mcp # opens the sign-in
```
Cursor
```jsonc
// .cursor/mcp.json, or use the install link below
{
"mcpServers": {
"workly": { "url": "https://api.workly.run/mcp" }
}
}
```
VS Code
```jsonc
// .vscode/mcp.json
{
"servers": {
"workly": { "type": "http", "url": "https://api.workly.run/mcp" }
}
}
```
Cursor can also install it from a link: [Add to Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=workly\&config=eyJ1cmwiOiJodHRwczovL2FwaS53b3JrbHkucnVuL21jcCJ9). For Claude or ChatGPT, see [chat apps](#claude-chatgpt-and-other-chat-apps).
#### Plugin for Claude Code and Codex
The Workly plugin ([`plugin/`](https://github.com/worklydev/workly/tree/v0.5.0/plugin)) adds the same server along with two skills: one for finding out why deliveries fail and retrying them once they’re fixed, and one for adding Workly to an app and testing the handler end to end. It signs in the same way.
Claude Code
```sh
claude plugin marketplace add worklydev/workly
claude plugin install workly@workly
```
Codex
```sh
codex plugin marketplace add worklydev/workly
codex plugin add workly@workly
```
For an agent that can’t open a browser, such as one in CI, create an API key in the console (**API keys**) and send it as a bearer token, as for a [self-hosted](#self-hosted) server. The agent gets that key’s workspace.
### Local dev server: `workly dev`
The dev server takes no key, and prints the Claude Code command when it starts.
Claude Code
```sh
claude mcp add --transport http workly-dev http://localhost:7337/mcp
```
Codex
```sh
codex mcp add workly-dev --url http://localhost:7337/mcp
```
Cursor
```jsonc
// .cursor/mcp.json
{
"mcpServers": {
"workly-dev": { "url": "http://localhost:7337/mcp" }
}
}
```
VS Code
```jsonc
// .vscode/mcp.json
{
"servers": {
"workly-dev": { "type": "http", "url": "http://localhost:7337/mcp" }
}
}
```
### Self-hosted
Send the server’s `WORKLY_API_KEY` as a bearer token. Each config reads it from an environment variable (VS Code prompts for it once and stores it), so it can be committed and shared with your team.
Claude Code
```sh
claude mcp add --transport http --scope project workly-prod https://workly.example.com/mcp \
--header 'Authorization: Bearer ${WORKLY_API_KEY}'
```
Codex
```sh
codex mcp add workly-prod --url https://workly.example.com/mcp --bearer-token-env-var WORKLY_API_KEY
```
Cursor
```jsonc
// .cursor/mcp.json
{
"mcpServers": {
"workly-prod": {
"url": "https://workly.example.com/mcp",
"headers": { "Authorization": "Bearer ${env:WORKLY_API_KEY}" }
}
}
}
```
VS Code
```jsonc
// .vscode/mcp.json
{
"inputs": [{ "type": "promptString", "id": "workly-key", "description": "Workly API key", "password": true }],
"servers": {
"workly-prod": {
"type": "http",
"url": "https://workly.example.com/mcp",
"headers": { "Authorization": "Bearer ${input:workly-key}" }
}
}
}
```
### Claude, ChatGPT and other chat apps
Chat apps that support remote MCP servers add workly.run as a custom connector: add one with the URL `https://api.workly.run/mcp`. Workly’s console opens; sign in or create an account, pick a workspace, and choose read-only or read and write.
A chat app connects from its vendor’s servers, not from your machine, so it can’t reach `workly dev`. It reaches a self-hosted server only if the server has a public address and the app can send its API key as a bearer token.
## Read-only by default
`WORKLY_MCP` sets which tools the server offers:
| Value | Tools | Default for |
| ----------- | ------------------------ | -------------- |
| `read-only` | Read tools only | `workly serve` |
| `full` | Read and write tools | `workly dev` |
| `off` | None: `/mcp` answers 404 | |
An agent pointed at production can’t enqueue, retry or cancel anything unless whoever runs the server sets `WORKLY_MCP=full`. workly.run serves the full tools to API keys, and to OAuth connections given read and write access; a connection given read-only access gets the read tools. The endpoint takes the same credentials as `/v1`: with `WORKLY_API_KEY` set, an agent needs the key. Anyone with the key can already do everything through the API, so `read-only` is a guard against an agent’s mistakes, not an access control.
## Tools
Tools return a record’s fields with no value as `null` rather than leaving them out, so every task, run or schedule has the same keys.
Read tools (marked read-only, so clients can run them without asking):
| Tool | What it does |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `list_queues` | All queues in summary: paused or not (by hand, or auto-paused after failures in a row), rate and concurrency limits, current work (due, scheduled, running, failed, how far behind) and the last 24 hours (succeeded, failed, and p50 and p95 attempt duration with its sample size) |
| `get_queue` | One queue with its full policy, the last 24 hours, how many tasks succeeded, failed and were cancelled recently, and its failed tasks grouped by error |
| `list_tasks` | Task summaries, newest first, by queue, state, group, idempotency key, last error and creation time. Each says when its first attempt started and how long the latest took, so queue wait and handler time can be told apart |
| `get_task` | A task with its attempts (outcomes, status codes, errors, response bodies, durations), how many attempts it gets and how many are left |
| `get_task_delivery` | The exact signed request for a task, with a curl command to replay it against a local handler |
| `wait_for_task` | Waits for an attempt since the task was enqueued or retried, and returns how the latest went, with whether the task has finished or will be retried. Returns at once with the reason if its queue is paused or it isn’t due before the wait would end |
| `list_groups` | [Task groups](https://workly.run/docs/groups/), newest first, with their status and counts, by status and name |
| `get_group` | One group: its counts, its most recently failed tasks and its callback task |
| `list_schedules` | All [schedules](https://workly.run/docs/schedules/) in summary: cron, timezone, URL, paused (and since when), next run (with the schedule’s UTC offset) and how the last one went |
| `get_schedule` | One schedule, its next five times (with the schedule’s UTC offset) and its recent runs, including skipped ones and why. A paused schedule’s times are `next_runs_if_resumed` |
| `list_runs` | Runs of one schedule or of all of them, newest first, optionally by state |
| `get_run` | A run with its attempts |
| `get_run_delivery` | The exact signed request for a run, with a curl command |
| `wait_for_run` | Waits for an attempt of a run, and returns how the latest went, with whether the run has finished |
| `list_workflows` | All [workflows](https://workly.run/docs/workflows/): their steps and how the last run went |
| `get_workflow` | One workflow’s definition: steps, dependencies, queues |
| `list_workflow_runs` | Runs of one workflow or of all of them, newest first, optionally by status |
| `get_workflow_run` | A run with each step’s status, task, last result and what it is blocked by or waiting for |
| `get_docs` | These docs, for the server’s version, as Markdown: the list of pages, or one page, SDK and CLI references included |
Write tools (`WORKLY_MCP=full`):
| Tool | What it does |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `enqueue_task` | Enqueues a task (creating the queue if needed), optionally into a group |
| `retry_task` | Delivers a task again now; replays a finished task |
| `cancel_task` | Cancels a pending or running task |
| `retry_failed_tasks` | Retries every failed task in a queue, or only those with one error |
| `cancel_pending_tasks` | Cancels a queue’s pending tasks; they stay in the history and can be retried |
| `pause_queue`, `resume_queue` | Stops and restarts a queue’s deliveries; resuming also ends an auto-pause |
| `update_queue_policy` | Changes some of a queue’s settings, keeping the rest, including `auto_pause` |
| `delete_queue` | Deletes a queue and its finished tasks. Refuses while it has tasks due, scheduled or running, and for Workly’s own queues |
| `update_schedule` | Creates a schedule, or changes some of its settings, keeping the rest |
| `trigger_schedule` | Runs a schedule now |
| `pause_schedule`, `resume_schedule` | Stops and restarts a schedule |
| `retry_run`, `cancel_run` | Delivers a run again, or cancels it |
| `delete_schedule` | Deletes a schedule and its runs |
| `trigger_workflow` | Starts a workflow run, with optional input and idempotency key |
| `cancel_workflow_run` | Cancels a workflow run: steps not yet started are skipped |
| `retry_workflow_run` | Reopens a failed or cancelled workflow run: its failed and skipped steps run again |
| `delete_workflow` | Deletes a workflow’s definition; its runs are kept |
Write tools that change one task, run, queue or schedule return it as changed, so an agent can confirm the change without reading it again. The delete tools can’t be undone, so they’re marked destructive, and clients that confirm destructive tools ask first. `delete_queue` won’t delete a queue with work left in it, so an agent can’t lose tasks it didn’t know about; run `cancel_pending_tasks` first if they should go.
Arguments are checked against each tool’s schema before anything runs, and errors name the argument at fault (`limit must be an integer, not string`, `state must be one of …`), so an agent can correct its call.
## What agents see
Task payloads, headers, response bodies and error messages come from your endpoints and whatever enqueues tasks, so they could contain anything. The server tells agents to treat them as untrusted, and shortens them to keep an agent’s context small: response bodies to 2 KB, payloads to 8 KB, and a task’s attempts to the latest 20. The full data is always in the dashboard and the API.
Each tool call is logged (`mcp tool call`, with the tool name, duration and any error).
# Workly for Python
> The Workly Python SDK: enqueue tasks, verify signatures, and manage queues, schedules, task groups, workflows and alerts, sync or async.
The Python SDK for [Workly](https://github.com/worklydev/workly), the open-source HTTP task queue. Enqueue a task, and Workly delivers it to your endpoint over HTTP, retrying with backoff until it succeeds.
```sh
pip install workly # the SDK
pip install "workly[cli]" # the SDK plus the workly binary, for `workly dev`
```
Requires Python 3.11+. The only dependency is [httpx](https://www.python-httpx.org). The `cli` extra adds [`workly-cli`](https://pypi.org/project/workly-cli/), the `workly` binary (about 7 MB): the dev server (`workly dev`) and the CLI, including `workly validate`, `plan` and `deploy` for [configuration as code](https://workly.run/docs/deploy/). Install it on dev machines and in CI; production images that only enqueue tasks don’t need it.
## Quickstart
Start a local server with `workly dev` (SQLite, no auth, on :7337), then:
```python
from workly import Workly
client = Workly() # WORKLY_URL, else http://localhost:7337; WORKLY_API_KEY
task = client.tasks.enqueue(
"emails",
url="https://api.example.com/send-email",
payload={"user_id": 123},
)
print(task.id, task.state)
```
Your endpoint receives a `POST` with the JSON payload. Any 2xx marks the task succeeded. A 5xx, 408, 429, timeout or connection error is retried with backoff. Delivery is at-least-once, so make handlers safe to run twice. `Workly-Task-Id` is stable across attempts and works as a deduplication key.
## Verifying deliveries
Every delivery is signed: with the server’s `WORKLY_SIGNING_SECRET` when you run Workly yourself, or with your workspace’s secret on [workly.run](#hosted-workly). Check the signature against the **raw** body:
```python
from workly import SignatureVerificationError, verify_signature
# Flask
@app.post("/send-email")
def send_email():
try:
verify_signature(request.get_data(), request.headers) # secret from WORKLY_SIGNING_SECRET
except SignatureVerificationError:
abort(401)
...
# FastAPI / Starlette
@app.post("/send-email")
async def send_email(request: Request):
verify_signature(await request.body(), request.headers, secret)
...
```
By default, signatures older than 5 minutes are rejected (`tolerance=`). `workly dev` signs with `workly-dev-secret`. To test your handler without a server, use `workly.sign(body, secret)`, which returns the headers Workly would send.
## Hosted Workly
On [workly.run](https://workly.run), point the client at the hosted API and use the keys from the [console](https://console.workly.run):
```sh
export WORKLY_URL=https://api.workly.run
export WORKLY_API_KEY=... # API keys page
export WORKLY_SIGNING_SECRET=whsec_... # same page; each workspace has its own
```
Nothing else changes. Hosted workspaces have a request rate limit and a monthly task quota: requests over the rate limit are retried for you, and an enqueue over the quota raises `PermissionDeniedError` with the code `quota_exceeded`.
## Tasks
```python
from datetime import datetime, timedelta, timezone
client.tasks.enqueue(
"emails",
url="https://api.example.com/send-email",
payload={"user_id": 123}, # JSON; or body="raw text", content_type="text/csv"
method="POST", # default
headers={"X-Tenant": "acme"}, # sent with every delivery
delay_seconds=60, # or run_at=datetime(..., tzinfo=timezone.utc)
idempotency_key="welcome-123", # an existing task with this key is returned instead
max_attempts=5, # overrides the queue's setting
timeout_seconds=10, # likewise
)
# Up to 100 at once, all or nothing.
client.tasks.enqueue_many(
"emails",
[{"url": "https://api.example.com/send-email", "payload": {"user_id": i}} for i in range(100)],
)
task = client.tasks.get("task_01J9...")
for attempt in client.tasks.attempts(task.id):
print(attempt.number, attempt.outcome, attempt.status_code, attempt.duration_ms)
client.tasks.retry(task.id) # run now; replays a finished task
delivery = client.tasks.delivery(task.id) # the signed request, to replay by hand
client.tasks.cancel(task.id)
# Newest first, across pages. All filters are optional.
for task in client.tasks.list(
queue="emails", state="failed", created_after=datetime.now(timezone.utc) - timedelta(days=1)
):
...
```
## Queues
Queues are created automatically on first enqueue. `put` sets a queue’s configuration declaratively, which makes it easy to keep in code: settings you leave out are reset to their defaults.
```python
client.queues.put(
"emails",
max_attempts=10,
min_backoff_seconds=1,
max_backoff_seconds=3600,
backoff_multiplier=2,
timeout_seconds=30,
rate_per_second=50,
max_concurrency=20,
auto_pause_after=20, # pause after 20 failures in a row, until a probe succeeds
)
queue = client.queues.get("emails")
print(queue.stats.due, queue.stats.oldest_due_at)
client.queues.pause("emails") # stop delivering; enqueueing still works
client.queues.resume("emails")
client.queues.cancel_pending("emails") # cancel pending tasks
client.queues.retry_failed("emails") # replay failed tasks, e.g. after an outage
for group in client.queues.failures("emails"): # failed tasks grouped by error
print(group.count, group.error)
client.queues.retry_failed("emails", last_error="HTTP 503 Service Unavailable") # replay one group
client.queues.activity("emails") # succeeded/failed/cancelled in the last 7 days
client.queues.delete("emails") # delete the queue and all its tasks
```
## Schedules
A schedule calls a URL on a cron timetable; each call is a run, retried and recorded like a task. Schedules are separate from queues. `put` is declarative, like `queues.put`. See [Schedules](https://workly.run/docs/schedules/).
```python
schedule = client.schedules.put(
"nightly-report",
cron="0 3 * * *",
timezone="Europe/Stockholm",
url="https://api.example.com/reports/nightly",
payload={"kind": "daily"},
max_attempts=3,
overlap="skip", # skip a slot while the previous run is still going
)
print(schedule.next_runs) # the next five times
run = client.schedules.trigger("nightly-report") # run it now
for run in client.runs.list(schedule="nightly-report", state="failed"):
print(run.id, run.scheduled_for, run.last_error)
client.runs.attempts(run.id)
client.runs.retry(run.id) # deliver it again, e.g. a failed or skipped run
client.schedules.pause("nightly-report")
client.schedules.resume("nightly-report") # from the next scheduled time
client.schedules.delete("nightly-report") # with all its runs
```
Runs are delivered with `Workly-Run-Id`, `Workly-Schedule` and `Workly-Scheduled-For` headers and the same signature, so `verify_signature` works unchanged.
## Task groups
A group tracks a set of tasks and enqueues a callback task once it’s closed and they’ve all finished: a `POST` of the group’s status and counts. See [Task groups](https://workly.run/docs/groups/). Like every callback, it goes to Workly’s `_callbacks` queue.
`groups.enqueue` does the whole job in one call: it creates the group, enqueues the tasks into it in batches of 100 and closes it, and it’s safe to retry with the same `idempotency_key`.
```python
from workly import Callback
group = client.groups.enqueue(
"stock-prices",
(
{"url": "https://api.example.com/jobs/update-stock-price", "payload": {"symbol": c.symbol}}
for c in companies
),
name="daily-stock-prices",
idempotency_key="stock-prices-2026-10-19", # a retry finishes this group rather than start another
# or a dict: {"url": "https://api.example.com/jobs/refresh-views"}
on_complete=Callback(url="https://api.example.com/jobs/refresh-views"),
)
```
Or step by step: `client.groups.create(...)`, then `client.tasks.enqueue(..., group_id=group.id)` or `enqueue_many` for each task, then `client.groups.close(group.id)`, after which it completes once its tasks have finished.
```python
group = client.groups.get(group.id)
print(group.status, group.total, group.succeeded, group.failed, group.remaining)
for task in client.tasks.list(group_id=group.id, state="failed"):
print(task.id, task.last_error)
for group in client.groups.list(name="daily-stock-prices", status="failed"):
print(group.id, group.finished_at)
```
## Callbacks
Tasks and schedules take the same `on_complete`: each time the task, or a run of the schedule, succeeds, fails or is cancelled, Workly POSTs a summary of it to the URL. See [Callbacks](https://workly.run/docs/callbacks/).
```python
client.tasks.enqueue(
"exports",
url="https://api.example.com/jobs/export",
payload={"export_id": 42},
on_complete=Callback(url="https://api.example.com/jobs/export-done"),
)
client.schedules.put(
"nightly-report",
cron="0 3 * * *",
url="https://api.example.com/reports/nightly",
on_complete={"url": "https://api.example.com/reports/nightly-done"},
)
```
The summary arrives as the callback task’s JSON body: `task_id`, `queue`, `state`, `attempt_count`, `last_status`, `last_error` and `finished_at` for a task; `run_id`, `schedule` and the same outcome fields for a run.
## Workflows
A workflow is a DAG of HTTP steps: each step is delivered as a task once the steps in its `depends_on` have succeeded, and a failed step stops the run. Most workflows live in a `workly.yaml` and are applied with `workly deploy`; `put` does the same from code, declaratively. See [Workflows](https://workly.run/docs/workflows/).
```python
client.workflows.put(
"daily-refresh",
steps=[
{"name": "prices", "url": "https://api.example.com/jobs/prices"},
{"name": "fx", "url": "https://api.example.com/jobs/fx"},
{
"name": "analytics",
"url": "https://api.example.com/jobs/analytics",
"depends_on": ["prices", "fx"],
},
],
)
run = client.workflows.trigger(
"daily-refresh", input={"date": "2026-11-13"}, idempotency_key="refresh-2026-11-13"
)
run = client.workflow_runs.get(run.id)
for step in run.steps:
print(step.name, step.status, step.blocked_by, step.error)
for run in client.workflow_runs.list(workflow="daily-refresh", status="failed"):
client.workflow_runs.retry(run.id) # its failed and skipped steps run again
client.workflow_runs.cancel(run.id) # steps not yet started are skipped
```
A step without `payload` or `body` is sent the run’s context as JSON, with `input`. A step can also wait for a task group: its endpoint answers 2xx with `Workly-Wait-For: group/`, and the step finishes when the group does.
## Alerts
Alert channels get an email or a Slack message when tasks or schedule runs fail: within about a minute, then at most one every 30 minutes per queue or schedule. See [Alerts](https://workly.run/docs/alerts/).
```python
channel = client.alerts.create(
"slack", "https://hooks.slack.com/services/..."
) # or "email", "ops@example.com"
client.alerts.test(channel.id) # send a test alert now
print(client.alerts.list().channels)
client.alerts.delete(channel.id)
```
## Async
`AsyncWorkly` has the same methods, as coroutines:
```python
from workly import AsyncWorkly
async with AsyncWorkly() as client:
task = await client.tasks.enqueue("emails", url="...", payload={"user_id": 123})
async for task in client.tasks.list(queue="emails"):
...
```
## Errors
| Exception | When |
| ----------------------- | ----------------------------------------------------------------------- |
| `BadRequestError` | 400, invalid request |
| `AuthenticationError` | 401, missing or wrong API key |
| `PermissionDeniedError` | 403, not allowed, or a hosted limit was reached (e.g. `quota_exceeded`) |
| `NotFoundError` | 404, no such queue or task |
| `ConflictError` | 409, e.g. retrying a running task |
| `PayloadTooLargeError` | 413 |
| `RateLimitError` | 429, still rate limited after the retries |
| `APIError` | base class of the above; has `.status`, `.code` (stable) and `.message` |
| `APIConnectionError` | the server couldn’t be reached |
Requests are retried up to `max_retries` times (2 by default), honouring `Retry-After`. A 429 is always retried, since the server refused the request without doing anything. Connection errors and 502–504 are retried for requests that are safe to repeat: an enqueue only when it has an `idempotency_key`, or when the connection failed before the request was sent.
## License
[Apache-2.0](https://github.com/worklydev/workly/blob/main/LICENSE)
# Workly for TypeScript
> The Workly TypeScript SDK for Node, Bun, Deno and edge runtimes: enqueue tasks, verify signatures, and manage queues, schedules, groups and workflows.
The TypeScript SDK for [Workly](https://github.com/worklydev/workly), the open-source HTTP task queue. Enqueue a task, and Workly delivers it to your endpoint over HTTP, retrying with backoff until it succeeds.
```sh
npm install @worklydev/sdk
```
No dependencies. It uses `fetch` and Web Crypto, so it runs on Node 22+, Bun, Deno and edge runtimes (Cloudflare Workers, Vercel Edge). ESM, with types included.
## Quickstart
Start a local server with `workly dev` (SQLite, no auth, on :7337; `pip install "workly[cli]"` or `curl -fsSL https://raw.githubusercontent.com/worklydev/workly/main/install.sh | sh`), then:
```ts
import { Workly } from '@worklydev/sdk';
const workly = new Workly(); // WORKLY_URL, else http://localhost:7337; WORKLY_API_KEY
const task = await workly.tasks.enqueue('emails', {
url: 'https://api.example.com/send-email',
payload: { userId: 123 },
});
console.log(task.id, task.state);
```
Your endpoint receives a `POST` with the JSON payload. Any 2xx marks the task succeeded. A 5xx, 408, 429, timeout or connection error is retried with backoff. Delivery is at-least-once, so make handlers safe to run twice. `Workly-Task-Id` is stable across attempts and works as a deduplication key.
## Verifying deliveries
Every delivery is signed: with the server’s `WORKLY_SIGNING_SECRET` when you run Workly yourself, or with your workspace’s secret on [workly.run](#hosted-workly). Check the signature against the **raw** body, before parsing it:
```ts
import { SignatureVerificationError, verifySignature } from '@worklydev/sdk';
// Next.js route handler, Hono, Cloudflare Workers: anything with a Fetch API Request
export async function POST(request: Request) {
const body = await request.text();
try {
await verifySignature(body, request.headers); // secret from WORKLY_SIGNING_SECRET
} catch (err) {
if (err instanceof SignatureVerificationError) return new Response(null, { status: 401 });
throw err;
}
const payload = JSON.parse(body);
// ...
}
// Express: keep the raw body
app.post('/send-email', express.raw({ type: '*/*' }), async (req, res) => {
await verifySignature(req.body, req.headers, secret);
// ...
});
```
By default, signatures older than 5 minutes are rejected (`{ toleranceSeconds }`). `workly dev` signs with `workly-dev-secret`. To test your handler without a server, `await sign(body, secret)` returns the headers Workly would send.
## Hosted Workly
On [workly.run](https://workly.run), point the client at the hosted API and use the keys from the [console](https://console.workly.run):
```sh
export WORKLY_URL=https://api.workly.run
export WORKLY_API_KEY=... # API keys page
export WORKLY_SIGNING_SECRET=whsec_... # same page; each workspace has its own
```
Nothing else changes. Hosted workspaces have a request rate limit and a monthly task quota: requests over the rate limit are retried for you, and an enqueue over the quota throws a `PermissionDeniedError` with the code `quota_exceeded`.
## Tasks
```ts
await workly.tasks.enqueue('emails', {
url: 'https://api.example.com/send-email',
payload: { userId: 123 }, // JSON, sent as is; or body: 'raw text', contentType: 'text/csv'
method: 'POST', // default
headers: { 'X-Tenant': 'acme' }, // sent with every delivery
delaySeconds: 60, // or runAt: new Date(...)
idempotencyKey: 'welcome-123', // an existing task with this key is returned instead
maxAttempts: 5, // overrides the queue's setting
timeoutSeconds: 10, // likewise
});
// Up to 100 at once, all or nothing.
await workly.tasks.enqueueMany(
'emails',
users.map((u) => ({ url: 'https://api.example.com/send-email', payload: { userId: u.id } })),
);
const task = await workly.tasks.get('task_01J9...');
for (const attempt of await workly.tasks.attempts(task.id)) {
console.log(attempt.number, attempt.outcome, attempt.statusCode, attempt.durationMs);
}
await workly.tasks.retry(task.id); // run now; replays a finished task
const delivery = await workly.tasks.delivery(task.id); // the signed request, to replay by hand
await workly.tasks.cancel(task.id);
// Newest first, across pages. All filters are optional.
for await (const task of workly.tasks.list({
queue: 'emails',
state: 'failed',
createdAfter: new Date(Date.now() - 24 * 3600 * 1000),
})) {
// ...
}
```
Field names are camelCase: the SDK converts them to and from the [API](https://github.com/worklydev/workly/blob/main/api/openapi.yaml)’s snake_case (`retry_on_4xx` is `retryOn4xx`). Payloads and headers are your own data, so they are sent and returned exactly as they are. Timestamps are RFC 3339 strings, and fields added by newer servers come through too.
## Queues
Queues are created automatically on first enqueue. `put` sets a queue’s configuration declaratively, which makes it easy to keep in code: settings you leave out are reset to their defaults.
```ts
await workly.queues.put('emails', {
maxAttempts: 10,
minBackoffSeconds: 1,
maxBackoffSeconds: 3600,
backoffMultiplier: 2,
timeoutSeconds: 30,
ratePerSecond: 50,
maxConcurrency: 20,
autoPause: { afterFailures: 20 }, // pause after 20 failures in a row, until a probe succeeds
});
const queue = await workly.queues.get('emails');
console.log(queue.stats?.due, queue.stats?.oldestDueAt);
await workly.queues.pause('emails'); // stop delivering; enqueueing still works
await workly.queues.resume('emails');
await workly.queues.cancelPending('emails'); // cancel pending tasks
await workly.queues.retryFailed('emails'); // replay failed tasks, e.g. after an outage
for (const group of await workly.queues.failures('emails')) {
console.log(group.count, group.error); // failed tasks grouped by error
}
await workly.queues.retryFailed('emails', { lastError: 'HTTP 503 Service Unavailable' }); // replay one group
await workly.queues.activity('emails'); // succeeded/failed/cancelled in the last 7 days
await workly.queues.delete('emails'); // delete the queue and all its tasks
```
## Schedules
A schedule calls a URL on a cron timetable; each call is a run, retried and recorded like a task. Schedules are separate from queues. `put` is declarative, like `queues.put`. See [Schedules](https://workly.run/docs/schedules/).
```ts
const schedule = await workly.schedules.put('nightly-report', {
cron: '0 3 * * *',
timezone: 'Europe/Stockholm',
url: 'https://api.example.com/reports/nightly',
payload: { kind: 'daily' },
maxAttempts: 3,
overlap: 'skip', // skip a slot while the previous run is still going
});
console.log(schedule.nextRuns); // the next five times
const run = await workly.schedules.trigger('nightly-report'); // run it now
for await (const r of workly.runs.list({ schedule: 'nightly-report', state: 'failed' })) {
console.log(r.id, r.scheduledFor, r.lastError);
}
await workly.runs.attempts(run.id);
await workly.runs.retry(run.id); // deliver it again, e.g. a failed or skipped run
await workly.schedules.pause('nightly-report');
await workly.schedules.resume('nightly-report'); // from the next scheduled time
await workly.schedules.delete('nightly-report'); // with all its runs
```
Runs are delivered with `Workly-Run-Id`, `Workly-Schedule` and `Workly-Scheduled-For` headers and the same signature, so `verifySignature` works unchanged.
## Task groups
A group tracks a set of tasks and enqueues a callback task once it’s closed and they’ve all finished: a `POST` of the group’s status and counts. See [Task groups](https://workly.run/docs/groups/). Like every callback, it goes to Workly’s `_callbacks` queue.
`groups.enqueue` does the whole job in one call: it creates the group, enqueues the tasks into it in batches of 100 and closes it, and it’s safe to retry with the same `idempotencyKey`.
```ts
const group = await workly.groups.enqueue(
'stock-prices',
companies.map((c) => ({ url: 'https://api.example.com/jobs/update-stock-price', payload: { symbol: c.symbol } })),
{
name: 'daily-stock-prices',
idempotencyKey: 'stock-prices-2026-10-19', // a retry finishes this group rather than start another
onComplete: { url: 'https://api.example.com/jobs/refresh-views' },
},
);
```
Or step by step: `workly.groups.create(...)`, then `workly.tasks.enqueue(..., { groupId: group.id })` or `enqueueMany` for each task, then `workly.groups.close(group.id)`, after which it completes once its tasks have finished.
```ts
const g = await workly.groups.get(group.id);
console.log(g.status, g.total, g.succeeded, g.failed, g.remaining);
for await (const task of workly.tasks.list({ groupId: group.id, state: 'failed' })) {
console.log(task.id, task.lastError);
}
```
The callback’s body is a `GroupSummary`. It arrives as the task’s payload, so its fields are snake_case: `group_id`, `status`, `total`, `succeeded`, `failed`, `cancelled`.
## Callbacks
Tasks and schedules take the same `onComplete`: each time the task, or a run of the schedule, succeeds, fails or is cancelled, Workly POSTs a summary of it to the URL. See [Callbacks](https://workly.run/docs/callbacks/).
```ts
await workly.tasks.enqueue('exports', {
url: 'https://api.example.com/jobs/export',
payload: { exportId: 42 },
onComplete: { url: 'https://api.example.com/jobs/export-done' },
});
await workly.schedules.put('nightly-report', {
cron: '0 3 * * *',
url: 'https://api.example.com/reports/nightly',
onComplete: { url: 'https://api.example.com/reports/nightly-done' },
});
```
The bodies are a `TaskSummary` (`task_id`, `queue`, `state`, `attempt_count`, `last_status`, `last_error`, `finished_at`, …) and a `RunSummary` (`run_id`, `schedule`, `state`, …), in snake_case as sent.
## Workflows
A workflow is a DAG of HTTP steps: each step is delivered as a task once the steps in its `dependsOn` have succeeded, and a failed step stops the run. Most workflows live in a `workly.yaml` and are applied with `workly deploy`; `put` does the same from code, declaratively. See [Workflows](https://workly.run/docs/workflows/).
```ts
await workly.workflows.put('daily-refresh', {
steps: [
{ name: 'prices', url: 'https://api.example.com/jobs/prices' },
{ name: 'fx', url: 'https://api.example.com/jobs/fx' },
{ name: 'analytics', url: 'https://api.example.com/jobs/analytics', dependsOn: ['prices', 'fx'] },
],
});
const started = await workly.workflows.trigger('daily-refresh', {
input: { date: '2026-11-13' },
idempotencyKey: 'refresh-2026-11-13',
});
const run = await workly.workflowRuns.get(started.id);
for (const step of run.steps) console.log(step.name, step.status, step.blockedBy, step.error);
for await (const failed of workly.workflowRuns.list({ workflow: 'daily-refresh', status: 'failed' })) {
await workly.workflowRuns.retry(failed.id); // its failed and skipped steps run again
}
await workly.workflowRuns.cancel(run.id); // steps not yet started are skipped
```
A step without `payload` or `body` is sent the run’s context as JSON, with `input`. A step can also wait for a task group: its endpoint answers 2xx with `Workly-Wait-For: group/`, and the step finishes when the group does.
## Alerts
Alert channels get an email or a Slack message when tasks or schedule runs fail: within about a minute, then at most one every 30 minutes per queue or schedule. See [Alerts](https://workly.run/docs/alerts/).
```ts
const channel = await workly.alerts.create('slack', 'https://hooks.slack.com/services/...'); // or 'email', 'ops@example.com'
await workly.alerts.test(channel.id); // send a test alert now
console.log((await workly.alerts.list()).channels);
await workly.alerts.delete(channel.id);
```
## Errors
| Error | When |
| ----------------------- | ----------------------------------------------------------------------- |
| `BadRequestError` | 400, invalid request |
| `AuthenticationError` | 401, missing or wrong API key |
| `PermissionDeniedError` | 403, not allowed, or a hosted limit was reached (e.g. `quota_exceeded`) |
| `NotFoundError` | 404, no such queue or task |
| `ConflictError` | 409, e.g. retrying a running task |
| `PayloadTooLargeError` | 413 |
| `RateLimitError` | 429, still rate limited after the retries |
| `APIError` | base class of the above; has `.status`, `.code` (stable) and `.message` |
| `APIConnectionError` | the server couldn’t be reached, or the request timed out |
All of them extend `WorklyError`. Requests are retried up to `maxRetries` times (2 by default), honouring `Retry-After`. A 429 is always retried, since the server refused the request without doing anything. Connection errors and 502–504 are retried for requests that are safe to repeat: an enqueue only when it has an `idempotencyKey`, or when the connection failed before the request was sent.
## Options
```ts
const workly = new Workly({
url: 'https://workly.internal', // default WORKLY_URL, then http://localhost:7337
apiKey: '...', // default WORKLY_API_KEY
timeoutMs: 30_000, // per call, including retries
maxRetries: 2,
fetch: customFetch,
});
// Every method takes an AbortSignal last.
await workly.tasks.get(id, { signal: AbortSignal.timeout(1000) });
```
## License
[Apache-2.0](https://github.com/worklydev/workly/blob/main/LICENSE)
# CLI reference
> Every workly command and flag: the dev server, serve, and the CLI for queues, tasks, groups, workflows, schedules and runs, validate, plan and deploy.
The `workly` binary is the server and the CLI. Install it with `pip install "workly[cli]"`, or see the [quickstart](https://workly.run/docs/quickstart/#install) for the other ways. `workly COMMAND --help` prints the same help as this page.
The commands that call the API (queues, tasks, groups, workflows, schedules, runs, plan and deploy) connect to `--url` or `WORKLY_URL`, `http://localhost:7337` by default, with `--api-key` or `WORKLY_API_KEY`. For hosted Workly, that is `https://api.workly.run` and a key from the console. They print a table, or the API’s JSON response with `--json`.
## workly dev
Run a local development server (SQLite, no auth).
`--config` applies a workly.yaml once the server is listening, as workly deploy would, so a fresh database starts with the file’s queues, workflows and schedules rather than default settings. It’s applied each time the server starts: restart it to apply changes to the file.
```text
workly dev [flags]
```
Flags:
```text
--config string a workly.yaml to apply at startup, as workly deploy would
--database-url string postgres:// or sqlite:// URL (env WORKLY_DATABASE_URL)
--port int port to listen on (env WORKLY_PORT) (default 7337)
```
## workly serve
Run the Workly server (for self-hosting).
Requires WORKLY_DATABASE_URL (postgres\://…) and WORKLY_SIGNING_SECRET. Other settings: WORKLY_API_KEY, WORKLY_PORT, WORKLY_METRICS_PORT, WORKLY_RETENTION, WORKLY_RUN_RETENTION, WORKLY_SHUTDOWN_TIMEOUT, WORKLY_SIGNING_SECRET_PREVIOUS, WORKLY_MCP and more; see
```text
workly serve [flags]
```
Flags:
```text
--database-url string postgres:// or sqlite:// URL (env WORKLY_DATABASE_URL)
--port int port to listen on (env WORKLY_PORT) (default 7337)
```
## workly migrate
Apply database migrations and exit.
```text
workly migrate [flags]
```
Flags:
```text
--database-url string postgres:// or sqlite:// URL (env WORKLY_DATABASE_URL)
--port int port to listen on (env WORKLY_PORT) (default 7337)
```
## workly healthcheck
Exit 0 if the local server is ready (for Docker HEALTHCHECK).
```text
workly healthcheck [flags]
```
Flags:
```text
--url string server URL (default http://localhost:$WORKLY_PORT)
```
## workly queues
List, configure, pause and resume queues.
Also: `queue`, `q`.
Flags for every `workly queues` command:
```text
--api-key string API key (env WORKLY_API_KEY)
--json print the API's JSON response
--url string server URL (env WORKLY_URL, default http://localhost:7337)
```
### workly queues ls
List queues with their current work.
```text
workly queues ls
```
Also: `list`.
### workly queues get
Show a queue’s configuration and current work.
```text
workly queues get NAME
```
### workly queues put
Create or update a queue.
Declarative, like the API: the queue ends up with exactly the settings given, and settings you leave out are reset to their defaults. Pausing is separate (see pause and resume).
```text
workly queues put NAME [flags]
```
Also: `create`, `update`.
Examples:
```sh
workly queues put emails --rate 50 --max-concurrency 10
workly queues put webhooks --max-attempts 5 --timeout 10s --retry-on-4xx
workly queues put billing --auto-pause-after 20
```
Flags:
```text
--auto-pause-after int pause the queue by itself after this many 5xx, timeouts or network errors in a row, until a probe succeeds (default never)
--backoff-multiplier float how fast the wait between retries grows (default 2)
--max-attempts int total delivery attempts before a task fails (default 5)
--max-backoff duration longest wait before a retry (default 1h0m0s)
--max-concurrency int most deliveries in flight at once (default 3)
--min-backoff duration shortest wait before a retry (default 2s)
--rate float most deliveries started per second (default no limit)
--retry-on-4xx retry 4xx responses (by default only 408 and 429 are retried)
--timeout duration how long to wait for a response (default 30s)
```
### workly queues pause
Stop delivering a queue’s tasks (enqueueing still works).
```text
workly queues pause NAME
```
### workly queues resume
Resume delivering a paused queue’s tasks.
```text
workly queues resume NAME
```
### workly queues failures
Group a queue’s failed tasks by error, most common first.
```text
workly queues failures NAME
```
### workly queues retry-failed
Retry every failed task in a queue, e.g. after an endpoint outage.
```text
workly queues retry-failed NAME [flags]
```
Examples:
```sh
workly queues retry-failed emails
workly queues retry-failed emails --error 'HTTP 503 Service Unavailable'
```
Flags:
```text
--error string only tasks whose last error is exactly this (see queues failures)
```
### workly queues cancel-pending
Cancel a queue’s pending tasks, including scheduled ones.
```text
workly queues cancel-pending NAME [flags]
```
Flags:
```text
-y, --yes don't ask for confirmation
```
### workly queues delete
Delete a queue and all its tasks.
```text
workly queues delete NAME [flags]
```
Also: `rm`.
Flags:
```text
-y, --yes don't ask for confirmation
```
## workly tasks
Enqueue, inspect, retry and cancel tasks.
Also: `task`, `t`.
Flags for every `workly tasks` command:
```text
--api-key string API key (env WORKLY_API_KEY)
--json print the API's JSON response
--url string server URL (env WORKLY_URL, default http://localhost:7337)
```
### workly tasks enqueue
Enqueue a task to deliver to URL, and print its ID.
The queue is created with default settings if it doesn’t exist. `--payload` and `--body` take a value, @file, or `-` for stdin.
```text
workly tasks enqueue QUEUE URL [flags]
```
Examples:
```sh
workly tasks enqueue emails https://api.example.com/send -d '{"user_id": 123}'
workly tasks enqueue reports https://api.example.com/build --delay 10m -k report-2026-09
cat event.json | workly tasks enqueue events https://api.example.com/ingest -d -
```
Flags:
```text
--at string deliver at this time (RFC 3339, e.g. 2026-10-01T09:00:00Z)
--body string raw body: a value, @file, or - for stdin
--content-type string content type of --body (default text/plain; charset=utf-8)
--delay duration deliver after this delay, e.g. 30s or 2h
--group string add the task to this open group (see workly groups)
-H, --header stringArray header sent with every delivery, "Name: value" (repeatable)
-k, --idempotency-key string don't enqueue if the queue already has a task with this key
--max-attempts int override the queue's max attempts
-X, --method string HTTP method (default POST)
--on-complete string POST a summary of the task to this URL each time it finishes
-d, --payload string JSON body: a value, @file, or - for stdin
--timeout duration override the queue's response timeout
```
### workly tasks get
Show a task and its delivery attempts.
```text
workly tasks get ID
```
### workly tasks ls
List tasks, newest first.
```text
workly tasks ls [flags]
```
Also: `list`.
Examples:
```sh
workly tasks ls --queue emails --state failed
workly tasks ls -k report-2026-09
```
Flags:
```text
--cursor string continue from a previous page
--error string only tasks whose last error is exactly this (see queues failures)
--group string only this group's tasks
-k, --idempotency-key string only tasks with this idempotency key
--limit int tasks per page (at most 200) (default 20)
--queue string only this queue's tasks
--state string only tasks in this state: pending, running, succeeded, failed or cancelled
```
### workly tasks retry
Deliver a task now; replays a finished task with fresh attempts.
```text
workly tasks retry ID
```
### workly tasks cancel
Cancel a pending or running task.
```text
workly tasks cancel ID
```
## workly groups
A task group tracks a set of tasks and enqueues a callback task once it is closed and all of them have finished. Enqueue into a group with workly tasks enqueue `--group`.
Also: `group`, `g`.
Flags for every `workly groups` command:
```text
--api-key string API key (env WORKLY_API_KEY)
--json print the API's JSON response
--url string server URL (env WORKLY_URL, default http://localhost:7337)
```
### workly groups create
Create an open group and print its ID.
```text
workly groups create [flags]
```
Examples:
```sh
group=$(workly groups create --name daily-prices --callback-url https://api.example.com/refresh)
workly tasks enqueue prices https://api.example.com/price -d '{"symbol": "NET"}' --group $group
workly groups close $group
```
Flags:
```text
--callback-url string URL of the callback task, enqueued when the group completes
-H, --header stringArray header sent with the callback, "Name: value" (repeatable)
-k, --idempotency-key string return the existing group with this key instead of creating one
--name string a label, e.g. the job this group is one run of
```
### workly groups get
Show a group’s status and progress.
```text
workly groups get ID
```
### workly groups ls
List groups, newest first.
```text
workly groups ls [flags]
```
Also: `list`.
Flags:
```text
--cursor string continue from a previous page
--limit int groups per page (at most 200) (default 20)
--name string only groups with this name
--status string only groups in this status: open, running, succeeded or failed
```
### workly groups close
Stop a group accepting tasks, so it completes once they finish.
```text
workly groups close ID
```
## workly workflows
A workflow runs HTTP steps in dependency order, each delivered as a task. A step’s endpoint can hand its completion to a task group by answering with Workly-Wait-For: group/. Define workflows in workly.yaml and apply them with workly deploy.
Also: `workflow`, `wf`.
Flags for every `workly workflows` command:
```text
--api-key string API key (env WORKLY_API_KEY)
--json print the API's JSON response
--url string server URL (env WORKLY_URL, default http://localhost:7337)
```
### workly workflows ls
List workflows and how each last ran.
```text
workly workflows ls
```
Also: `list`.
### workly workflows get
Show a workflow’s steps and their dependencies.
```text
workly workflows get NAME
```
### workly workflows rm
Delete a workflow’s definition; its runs are kept.
```text
workly workflows rm NAME [flags]
```
Flags:
```text
-y, --yes delete without asking
```
### workly workflows trigger
Start a run of a workflow and print its ID. `--input` is JSON passed to every step without a body of its own; it takes a value, @file, or `-` for stdin.
```text
workly workflows trigger NAME [flags]
```
Also: `start`.
Examples:
```sh
workly workflows trigger daily-refresh --input '{"date": "2026-11-13"}'
run=$(workly workflows trigger daily-refresh -k 2026-11-13) && workly workflows run $run
```
Flags:
```text
-k, --idempotency-key string return the existing run with this key instead of starting one
-i, --input string JSON input for the run: a value, @file or - for stdin
```
### workly workflows runs
List workflow runs, newest first.
```text
workly workflows runs [NAME] [flags]
```
Flags:
```text
--cursor string continue from a previous page
--limit int runs per page (at most 200) (default 20)
--status string only runs in this status: running, succeeded, failed or cancelled
```
### workly workflows run
Show a workflow run: each step, and why it is where it is.
```text
workly workflows run ID
```
### workly workflows cancel
Cancel a workflow run: steps not yet started are skipped.
```text
workly workflows cancel ID
```
### workly workflows retry
Reopen a failed or cancelled workflow run, under the definition it started with and with the same input. Steps that succeeded stay; the rest run again, each on its old task with a fresh set of attempts, and a step that was waiting for a group follows the group.
```text
workly workflows retry ID
```
## workly schedules
Create, inspect, run and pause schedules.
A schedule calls a URL on a cron timetable. Each call is a run, with its own attempts and retries. Schedules are separate from queues and tasks.
Also: `schedule`, `s`.
Flags for every `workly schedules` command:
```text
--api-key string API key (env WORKLY_API_KEY)
--json print the API's JSON response
--url string server URL (env WORKLY_URL, default http://localhost:7337)
```
### workly schedules ls
List schedules with their next and last run.
```text
workly schedules ls
```
Also: `list`.
### workly schedules get
Show a schedule’s settings, next slots and recent runs.
```text
workly schedules get NAME
```
### workly schedules put
Create or update a schedule that calls URL, or with `--workflow` (and no URL) starts a run of that workflow, with `--payload` as the run’s input.
Declarative, like the API: the schedule ends up with exactly the settings given, and settings you leave out are reset to their defaults. Pausing is separate (see pause and resume). `--payload` and `--body` take a value, @file, or `-` for stdin.
`--cron` takes five fields (minute hour day-of-month month day-of-week), or @hourly, @daily, @weekly, @monthly or @yearly.
```text
workly schedules put NAME [URL] [flags]
```
Also: `create`, `update`.
Examples:
```sh
workly schedules put nightly-report https://api.example.com/reports/nightly \
--cron '0 3 * * *' --timezone Europe/Stockholm -d '{"kind": "daily"}'
workly schedules put sync https://api.example.com/sync --cron '*/15 * * * mon-fri' --overlap allow
workly schedules put daily-refresh --workflow daily-refresh --cron '0 3 * * *'
```
Flags:
```text
--backoff-multiplier float how fast the wait between retries grows (default 2)
--body string raw body: a value, @file, or - for stdin
--content-type string content type of --body (default text/plain; charset=utf-8)
--cron string when to run, e.g. '0 3 * * *' or @hourly (required)
-H, --header stringArray header sent with every run, "Name: value" (repeatable)
--max-attempts int total delivery attempts before a run fails (default 3)
--max-backoff duration longest wait before a retry (default 5m0s)
-X, --method string HTTP method (default POST)
--min-backoff duration shortest wait before a retry (default 5s)
--missed string slots missed while the server was down: skip, or catch_up (run the latest once) (default "skip")
--on-complete string POST a summary of each run to this URL when it finishes
--overlap string a slot that comes while the previous run is still going: skip or allow (default "skip")
-d, --payload string JSON body: a value, @file, or - for stdin
--retry-on-4xx retry 4xx responses (by default only 408 and 429 are retried)
--timeout duration how long to wait for a response (default 30s)
--timezone string IANA timezone the cron expression is read in, e.g. Europe/Stockholm (default "UTC")
--workflow string start a run of this workflow instead of calling a URL
```
### workly schedules pause
Stop a schedule from running on its timetable.
```text
workly schedules pause NAME
```
### workly schedules resume
Resume a paused schedule from its next slot.
```text
workly schedules resume NAME
```
### workly schedules trigger
Run a schedule now, outside its timetable, and print the run’s ID.
The run is delivered right away, even if the schedule is paused or a run is in flight. It doesn’t change when the schedule next runs.
```text
workly schedules trigger NAME
```
Also: `run`.
### workly schedules delete
Delete a schedule and all its runs.
```text
workly schedules delete NAME [flags]
```
Also: `rm`.
Flags:
```text
-y, --yes don't ask for confirmation
```
## workly runs
Inspect, retry and cancel schedule runs.
Also: `run`, `r`.
Flags for every `workly runs` command:
```text
--api-key string API key (env WORKLY_API_KEY)
--json print the API's JSON response
--url string server URL (env WORKLY_URL, default http://localhost:7337)
```
### workly runs ls
List runs of every schedule, or of one, newest first.
```text
workly runs ls [flags]
```
Also: `list`.
Examples:
```sh
workly runs ls --state failed
workly runs ls --schedule nightly-report
```
Flags:
```text
--cursor string continue from a previous page
--limit int runs per page (at most 200) (default 20)
--schedule string only this schedule's runs
--state string only runs in this state: pending, running, succeeded, failed, cancelled or skipped
```
### workly runs get
Show a run and its delivery attempts.
```text
workly runs get ID
```
### workly runs retry
Deliver a run now; a finished or skipped run gets fresh attempts.
```text
workly runs retry ID
```
### workly runs cancel
Cancel a pending or running run.
```text
workly runs cancel ID
```
## workly validate
Check workly.yaml, without contacting the server: its fields, cron expressions, timezones, URLs and headers, and that every ${NAME} it uses is set in the environment. It lists the ${NAME:-default} references that took their default.
```text
workly validate [flags]
```
Flags:
```text
-f, --file string the configuration file (default "workly.yaml")
```
## workly plan
Show what workly deploy would change, without changing anything.
`--format` markdown writes the plan for a pull request comment or GitHub’s step summary ($GITHUB_STEP_SUMMARY). `--detailed-exitcode` exits 0 when there’s nothing to change, 2 when there are changes, and 3 when they include deletes, so CI can ask for a closer look; errors exit 1.
```text
workly plan [flags]
```
Flags:
```text
--api-key string API key (env WORKLY_API_KEY)
--detailed-exitcode exit 2 when there are changes, 3 when they include deletes
-f, --file string the configuration file (default "workly.yaml")
--format string text, or markdown for a pull request or GitHub's step summary (default "text")
--url string server URL (env WORKLY_URL, default http://localhost:7337)
```
## workly deploy
Create and update the queues, workflows and schedules in workly.yaml, so the server matches it. Shows the plan and asks first; `--yes` skips the question, as CI needs. Running it again changes nothing. It never pauses or resumes anything, and leaves what the file doesn’t declare alone, unless the file sets prune: true: then it deletes the queues, workflows and schedules a section of the file leaves out. Workflow runs already going keep the definition they started with.
```text
workly deploy [flags]
```
Flags:
```text
--api-key string API key (env WORKLY_API_KEY)
-f, --file string the configuration file (default "workly.yaml")
--url string server URL (env WORKLY_URL, default http://localhost:7337)
-y, --yes apply without asking
```
## workly version
Print the version.
```text
workly version
```
# Configuration
> Every Workly server setting: the WORKLY_* environment variables for the database, signing secrets, API key, retention, payload limits, email and logging.
The server is configured with environment variables. `--port` and `--database-url` flags override their variables.
| Variable | Default | Description |
| -------------------------------- | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `WORKLY_DATABASE_URL` | `sqlite://.workly/dev.db` in `dev`; required for `serve` | `postgres://user:pass@host:5432/db` or `sqlite://path/to/file.db` (`sqlite:///abs/path.db` for an absolute path). |
| `WORKLY_SIGNING_SECRET` | `workly-dev-secret` in `dev`; required for `serve` | Signs every delivery (`Workly-Signature`). Generate one with `openssl rand -hex 32`. |
| `WORKLY_SIGNING_SECRET_PREVIOUS` | unset | During a secret rotation, also signs deliveries with the old secret. See [Rotating the signing secret](#rotating-the-signing-secret). |
| `WORKLY_API_KEY` | unset | If set, `/v1` and `/metrics` require `Authorization: Bearer `. `serve` logs a warning when it’s unset. Always ignored by `dev`. |
| `WORKLY_PORT` | `7337` | Port for the API and dashboard. |
| `WORKLY_METRICS_PORT` | unset | Serve `/metrics` on this port instead, without the API key. Keep it internal. |
| `WORKLY_RETENTION` | `168h` | How long succeeded, failed and cancelled tasks (with their attempts) are kept. `0` keeps them forever. |
| `WORKLY_RUN_RETENTION` | `WORKLY_RETENTION` | How long finished [schedule](https://workly.run/docs/schedules/) and [workflow](https://workly.run/docs/workflows/) runs are kept. Each schedule’s and workflow’s latest run is always kept. `0` keeps them forever. |
| `WORKLY_SHUTDOWN_TIMEOUT` | `25s` | How long in-flight requests and deliveries get to finish on shutdown. Deliveries still running after it are retried. |
| `WORKLY_MAX_PAYLOAD_BYTES` | `1048576` | Largest task body, up to 100 MiB. |
| `WORKLY_MAX_IN_FLIGHT` | `100` | Most task deliveries in flight at once, across all queues, up to 10,000. Throughput to slow endpoints is at most this divided by their response time; queues’ `max_concurrency` limits them further. Schedule runs have their own cap of 20. |
| `WORKLY_AUTO_CREATE_QUEUES` | `true` | Enqueueing to a missing queue creates it with defaults. With `false`, it’s a `404 queue_not_found`. |
| `WORKLY_AUTO_MIGRATE` | `true` | Apply database migrations on startup. With `false`, run `workly migrate` before starting new versions. |
| `WORKLY_MCP` | `read-only` (`full` in `dev`) | Tools served to coding agents at `/mcp`: `read-only`, `full` (adds enqueue, run now, retry, cancel, pause and settings changes) or `off`. See [MCP](https://workly.run/docs/mcp/). |
| `WORKLY_PUBLIC_URL` | `http://localhost:7337` in `dev`; unset | Where people open Workly, e.g. `https://workly.example.com`. [Alerts](https://workly.run/docs/alerts/) link to its dashboard; without it they have no links. |
| `WORKLY_SMTP_URL` | unset | SMTP server for email [alerts](https://workly.run/docs/alerts/): `smtp://user:pass@host:587` (STARTTLS, required before the password is sent, except to localhost) or `smtps://user:pass@host:465`. Without it, alerts go to Slack only. |
| `WORKLY_SMTP_FROM` | unset | The address alerts are sent from, e.g. `Workly `. Required with `WORKLY_SMTP_URL`. |
| `WORKLY_LOG_LEVEL` | `info` | `debug`, `info`, `warn` or `error`. |
| `WORKLY_LOG_FORMAT` | `json` (`text` in `dev`) | `json` or `text`. |
Durations use Go’s syntax: `90s`, `15m`, `168h`.
## Postgres
Workly needs Postgres 14 or newer. It creates its tables on startup (`WORKLY_AUTO_MIGRATE`); replicas starting together take turns.
Connection settings go in the URL:
* `sslmode=require` (or `verify-full`) for a managed database.
* `search_path=workly` keeps Workly’s tables in their own schema. Create the schema first. Separate installs can share a database this way, each in its own schema.
* `pool_max_conns=20` raises the connection pool size (the default is 4, or the number of CPUs if higher). The dispatcher leader holds one more connection for its lock.
## The CLI
The CLI’s API commands (`queues`, `tasks`, `schedules` and the rest) talk to a server over its API:
| Variable | Flag | Default |
| ---------------- | ----------- | ----------------------- |
| `WORKLY_URL` | `--url` | `http://localhost:7337` |
| `WORKLY_API_KEY` | `--api-key` | unset |
The Python and TypeScript SDKs read the same two variables.
## Rotating the signing secret
Endpoints verify deliveries with the signing secret, so changing it takes three steps:
1. Set the new secret as `WORKLY_SIGNING_SECRET` and the old one as `WORKLY_SIGNING_SECRET_PREVIOUS`, and restart. Each delivery now carries both signatures, and endpoints still on the old secret keep verifying them.
2. Move your endpoints to the new secret.
3. Unset `WORKLY_SIGNING_SECRET_PREVIOUS` and restart.
# Metrics
> Workly's Prometheus metrics: queue depth, delivery latency, the age of the oldest due task and schedule health, with example alerting rules.
Workly serves Prometheus metrics at `/metrics`. If the server requires an API key (`WORKLY_API_KEY`), so does `/metrics`: give Prometheus the key as a bearer token.
Alternatively, set `WORKLY_METRICS_PORT` (e.g. `9090`) to serve `/metrics` on a separate port without the key, and not on the main port. Keep that port internal; the [Kubernetes manifests](https://github.com/worklydev/workly/blob/v0.5.0/deploy/kubernetes/workly.yaml) do this and add `prometheus.io/*` scrape annotations.
```yaml
scrape_configs:
- job_name: workly
authorization:
credentials:
static_configs:
- targets: ["workly:7337"]
```
Metric names and labels are stable. Every per-queue metric has `workspace` and `queue` labels; `workspace` is `default` on a self-hosted server.
## Reference
| Metric | Type | Description |
| ------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------- |
| `workly_tasks_enqueued_total` | counter | Tasks enqueued. Idempotent repeats aren’t counted. |
| `workly_attempts_total{outcome}` | counter | Delivery attempts by outcome: `success`, `http_error`, `timeout`, `network_error` or `lost`. |
| `workly_tasks_completed_total{state}` | counter | Tasks that finished delivery: `succeeded` or `failed`. |
| `workly_attempt_duration_seconds` | histogram | Time from request to response. |
| `workly_dispatch_lag_seconds` | histogram | Time from when a task was due to when its delivery started, including waits for rate and concurrency limits. |
| `workly_queue_due` | gauge | Pending tasks that are due now. |
| `workly_queue_scheduled` | gauge | Pending tasks due later, including ones waiting to retry. |
| `workly_queue_running` | gauge | Tasks with a delivery in flight. |
| `workly_queue_failed` | gauge | Failed tasks still retained (`WORKLY_RETENTION`). A rise means an endpoint is rejecting deliveries. |
| `workly_queue_paused` | gauge | 1 if the queue is paused. |
| `workly_queue_auto_paused` | gauge | 1 if the queue paused itself after failures in a row ([auto-pause](https://workly.run/docs/concepts/#auto-pause)) and hasn’t resumed. |
| `workly_queue_auto_pauses_total` | counter | Times the queue paused itself. |
| `workly_queue_oldest_due_age_seconds` | gauge | How long the longest-waiting due task has been due. **The metric to alert on.** |
| `workly_dispatcher_is_leader` | gauge | 1 on the replica that delivers tasks, 0 on standbys. No labels. |
[Schedules](https://workly.run/docs/schedules/) have their own metrics, with `workspace` and `schedule` labels:
| Metric | Type | Description |
| ------------------------------------------------ | --------- | ------------------------------------------------------------------------------------------------------------------------- |
| `workly_schedule_runs_total{state}` | counter | Runs that finished: `succeeded`, `failed` or `skipped`. |
| `workly_schedule_attempts_total{outcome}` | counter | Delivery attempts of runs, by outcome. |
| `workly_schedule_attempt_duration_seconds` | histogram | Time from request to response. |
| `workly_schedule_fire_lag_seconds` | histogram | Time from a scheduled time to when its run was created. |
| `workly_schedule_paused` | gauge | 1 if the schedule is paused. |
| `workly_schedule_last_success_timestamp_seconds` | gauge | When the latest successful run finished (Unix time). Absent until one succeeds. **The metric to alert on** for schedules. |
The queue gauges are read from the database when Prometheus scrapes. They stop counting at 10,000, to keep scrapes cheap with a large backlog; the age of the oldest due task has no such limit.
With several replicas, only the leader delivers, so the attempt and queue metrics come from the leader. `workly_tasks_enqueued_total` is counted by whichever replica served the request: sum it across replicas.
## Example alerts
```yaml
groups:
- name: workly
rules:
- alert: WorklyQueueBehind
expr: workly_queue_oldest_due_age_seconds > 300
for: 5m
annotations:
summary: "Queue {{ $labels.queue }} is {{ $value | humanizeDuration }} behind"
- alert: WorklyTasksFailing
expr: sum by (queue) (rate(workly_tasks_completed_total{state="failed"}[10m])) > 0
annotations:
summary: "Tasks in {{ $labels.queue }} are failing after all retries"
- alert: WorklyScheduleStale
# A daily schedule that hasn't succeeded for over a day.
expr: time() - workly_schedule_last_success_timestamp_seconds{schedule="nightly-report"} > 26 * 3600
annotations:
summary: "Schedule {{ $labels.schedule }} hasn't succeeded in over a day"
- alert: WorklyNoLeader
expr: max(workly_dispatcher_is_leader) == 0
for: 1m
annotations:
summary: "No Workly replica is delivering tasks"
```
A paused queue falls behind by design; add `unless on (workspace, queue) workly_queue_paused == 1` to the first alert to ignore paused queues.
Each delivery attempt is also logged as one structured line (`task attempt`, or `task failed` when a task gives up) with the task ID, queue, attempt number, outcome, status, duration and next retry. Schedule runs log `schedule fired`, `schedule run skipped`, `schedule run attempt` and `schedule run failed` the same way.
# Performance
> Workly's load test results on SQLite and Postgres: enqueue and delivery rates and latencies on one server, batched enqueues, what limits them and how to reproduce them.
These are measurements on one laptop, to show where Workly’s limits are and what moves them. On Postgres, Workly enqueues about 15,000 tasks/s sent one per request and about 60,000 tasks/s sent in batches, and delivers about 4,000 tasks/s to a fast endpoint. SQLite enqueues about 20,000 tasks/s and delivers about 6,000.
## Results
Each run enqueues tasks for 20 s to a no-op endpoint that answers `204`, after a warm-up run against a fresh database. Latency is measured from just before the enqueue request to the moment the endpoint receives the delivery. Repeated runs vary by up to about 20%, and more near a limit.
### Enqueueing
With the queue paused, nothing is delivered, so these runs measure enqueues alone: as many requests as the load test can send, either one task per request (`POST /v1/queues/{queue}/tasks`) or 100 per request (`POST /v1/queues/{queue}/tasks:batch`).
| Requests | SQLite | Postgres |
| -------------------------- | --------------------------------- | -------------------------------- |
| 1 task each, 64 at once | 19,720/s (p50 3.1 ms, p99 6.2 ms) | 14,990/s (p50 3.9 ms, p99 12 ms) |
| 100 tasks each, 4 at once | 20,280/s (p50 15 ms, p99 64 ms) | 37,100/s (p50 9.6 ms, p99 36 ms) |
| 100 tasks each, 16 at once | 19,790/s (p50 60 ms, p99 360 ms) | 56,750/s (p50 25 ms, p99 75 ms) |
| 100 tasks each, 64 at once | 19,840/s (p50 227 ms, p99 1.4 s) | 60,950/s (p50 98 ms, p99 278 ms) |
Latencies are per request. Enqueues that arrive while others are being written share the next statement, so single enqueues from many clients are written much like a batch. On Postgres, batches still enqueue about 4 times as many tasks per second: if you enqueue many tasks at once, send them in batches. SQLite writes about 20,000 tasks/s on its one connection either way, so more concurrent batches only add waiting.
### Enqueue and delivery
One task per request at a fixed rate, with 64 concurrent requests, as a typical client sends them.
**SQLite** (`workly dev`):
| Rate | Enqueue p50 / p99 | Enqueue → delivery p50 / p95 / p99 | Delivered |
| ------- | ----------------- | ---------------------------------- | --------------------------- |
| 2,000/s | 1.0 ms / 6.1 ms | 2.3 ms / 4.5 ms / 11 ms | 1,999/s, all |
| 4,000/s | 2.1 ms / 18 ms | 4.6 ms / 25 ms / 80 ms | 3,996/s, all |
| 5,000/s | 2.2 ms / 11 ms | 5.1 ms / 14 ms / 66 ms | 4,995/s, all |
| 6,000/s | 3.5 ms / 14 ms | 8.6 ms / 320 ms / 382 ms | 5,993/s, all (at the limit) |
| 8,000/s | 6.1 ms / 18 ms | backlog grows | \~6,190/s (saturated) |
**Postgres** (`workly serve`, default connection pool):
| Rate | Enqueue p50 / p99 | Enqueue → delivery p50 / p95 / p99 | Delivered |
| ------- | ----------------- | ---------------------------------- | --------------------------- |
| 1,000/s | 1.1 ms / 8.0 ms | 2.6 ms / 4.7 ms / 16 ms | 999/s, all |
| 2,000/s | 1.7 ms / 21 ms | 4.7 ms / 17 ms / 79 ms | 1,998/s, all |
| 3,000/s | 2.2 ms / 17 ms | 7.5 ms / 74 ms / 213 ms | 2,998/s, all |
| 4,000/s | 2.4 ms / 10 ms | 7.2 ms / 30 ms / 68 ms | 3,995/s, all (at the limit) |
| 6,000/s | 3.5 ms / 17 ms | backlog grows | \~4,510/s (saturated) |
Around the Postgres limit, runs differ: another run at 4,000/s fell behind and delivered 3,270/s, and one at 5,000/s kept up.
No run lost or duplicated a task.
### Test setup
Apple M1 Pro (10 cores, 16 GB), macOS. Postgres 17 runs in Docker Desktop on the same machine, so every query crosses the Docker VM network (roughly 0.5 ms per round trip). Workly, the load generator and the endpoint all share the machine. Workly logs every attempt at info level, as it does by default. Dedicated Postgres on Linux should do better; treat these numbers as a floor.
## What limits throughput
* **The dispatch loop.** One loop claims tasks and one writer records results, each a statement or two per batch. On Postgres they saturate at about 4,000–4,500 deliveries/s here.
* **Enqueue statements.** Each replica runs at most 4 enqueue statements at once on Postgres, and 1 on SQLite. Enqueues that arrive while those are busy wait for the next statement and share it, up to 500 per statement; with few enqueues, each is written at once. A batch request is one statement of its own.
* **The connection pool.** Enqueues, attempt results and the dispatcher share one pool, whose default size is the number of CPUs (minimum 4). Since single enqueues use at most 4 connections however many requests arrive, they leave the rest to the dispatcher and the API’s reads. A larger pool (`pool_max_conns` in `WORKLY_DATABASE_URL`) helps when many other requests run at once, keeping in mind that every replica opens its own pool against Postgres’s `max_connections`.
* **SQLite’s single writer.** Enqueues and attempt results take turns on one connection. Deliveries saturate around 6,000 tasks/s and enqueues around 20,000 tasks/s, far beyond what `workly dev` and small installs need.
* **Global in-flight limit.** At most 100 deliveries run at once by default (`WORKLY_MAX_IN_FLIGHT`), so throughput to a slow endpoint is at most 100 ÷ response time (e.g. 200 tasks/s at 500 ms). Per-queue `max_concurrency` (3 by default) limits it further; the load test raises it so only the global limit applies. To an endpoint as fast as the load test’s, raising the global limit doesn’t help: at 4,000 tasks/s on SQLite, before enqueues shared statements, 400 slots delivered 2,856/s against 2,880/s with 100.
Bottlenecks found while load testing, and fixed:
* Wake-up notifications used to be sent inside the enqueue transaction. A Postgres transaction that sends `NOTIFY` holds a database-wide lock through its commit, so every enqueue waited for the previous one’s WAL flush. Enqueues topped out at about 1,300/s. The server now sends the wake-up after commit, coalesced so that at most one is in flight at a time.
* The dispatcher claimed tasks with one `UPDATE` per task. Claiming happens in one loop, so round trips capped dispatch at about 1,100 tasks/s. It now claims each batch with one statement.
* Each delivery recorded its result in a transaction of its own, so Postgres connections queued on the write-ahead log, one commit per delivery. Results that finish while a write is in flight now share the next one: one transaction and two statements per batch. On the same machine, saturation went from about 1,460 to 1,840 tasks/s with the default pool, from about 1,780 to 2,260–2,670 with 40 connections, and from about 2,090 to 3,180 on SQLite.
* An enqueue took five round trips: a transaction around making sure the queue exists, the insert, and reading the task back. It is now one statement, which inserts the task if its queue exists and returns it; the queue is created only when that finds none. Claiming a batch went from four round trips to one the same way. On Postgres, saturation went from about 1,740 to 2,830 tasks/s with the default pool, and delivery latency at 2,000/s from seconds to milliseconds.
* The batch endpoint inserted its tasks one statement at a time, so 100 tasks took about 100 round trips in one transaction. It now inserts them with one statement; tasks that statement can’t insert (an idempotency key already taken, or a queue that doesn’t exist yet) go through the single-task path. With 64 batches of 100 at once, enqueues went from 12,130 to 25,480 tasks/s on Postgres and from 6,840 to 19,150 on SQLite.
* Each single enqueue was a statement and a commit of its own, so concurrent enqueues queued for connections and, on SQLite, for its one writer. Enqueues that arrive while others are being written now share the next statement. With 64 requests at once, enqueues went from about 5,500 to 15,000 tasks/s on Postgres and from 4,100 to 19,700 on SQLite. SQLite’s delivery limit went from about 2,900 to 6,000 tasks/s, as enqueues take far fewer turns on its connection.
## Reproducing
Start a server, then run the load test against it:
```sh
just dev # SQLite on :7337
just loadtest -rate 1000 -duration 20s
just loadtest -paused -rate 0 -batch 100 -workers 16 -duration 20s # enqueues only
```
`-paused` keeps the queue paused so nothing is delivered, `-rate 0` sends requests as fast as the workers can, and `-batch` sets the tasks per request.
For Postgres, run `just serve`, or `workly serve` with `WORKLY_DATABASE_URL` and `WORKLY_SIGNING_SECRET` set. The load test recreates its queue (`loadtest`) on each run. See `go run ./tools/loadtest -h` for options.