Task groups
Fan out over hundreds of tasks. Get one callback.
A task group tracks a batch of tasks and calls you back once all of them have finished, with how they went. Enqueue a task per customer, file or price to refresh, close the group, and let Workly do the counting.
Create, enqueue, close.
Every night, a schedule calls an endpoint that refreshes 500 stock prices. It creates a group, enqueues one task per company into it, and closes it. When the last price is in, the group's callback rebuilds the views that depend on them, once.
The callback fires whether the group succeeded or failed, so the handler checks status and can tell someone how many failed.
from datetime import date
from workly import Callback, Workly
workly = Workly() # WORKLY_URL, WORKLY_API_KEY
API = "https://api.example.com"
SYMBOLS = ["NET", "DDOG", "SNOW", ...] # the 500 companies to update
day = date.today().isoformat()
# Creates the group, enqueues the tasks in batches of 100 and closes it
workly.groups.enqueue(
"stock-prices",
({"url": f"{API}/jobs/update-stock-price", "payload": {"symbol": s}} for s in SYMBOLS),
name="daily-stock-prices",
idempotency_key=f"stock-prices-{day}", # a retry finishes the same group
on_complete=Callback(url=f"{API}/jobs/refresh-views"),
){
"group_id": "grp_01J9ZK3M5Q8R2T4V6W8X0Y2Z4A",
"name": "daily-stock-prices",
"status": "failed",
"total": 500,
"succeeded": 498,
"failed": 1,
"cancelled": 1
}See a batch while it runs.
The dashboard's Groups page lists groups with their progress. A group's page shows its counts, its callback and how that was delivered, and its tasks by state, with the failed ones a click away. Agents get the same over MCP.
- Total
- 500
- Succeeded
- 409
- Failed
- 1
- Remaining
- 90
Callback: POST /jobs/refresh-views, once the group completes
Workly counts the tasks in the database.
Counting finished jobs yourself means a table, a lock and edge cases: a retry counted twice, a task enqueued after the last one finished, a crash between the last task and the follow-up. Groups handle them in the database.
No count up front
Enqueue as many tasks as you like, in as many batches as you like, then close the group. It can't complete while it's open, so it never finishes early.
Exactly one callback
The group completes in the same database transaction that finishes its last task, and that transaction enqueues the callback. A crash can't lose it, and a race can't send two.
Ordinary tasks inside
Tasks in a group keep their queue's retries, rate limits and history. A task waiting to retry still counts as remaining, and one that fails doesn't stop the others.
Fix before it finishes
Retry a failed task while its group is still running and the group waits for the retry, so you can fix a handler mid-batch and still get a clean result.
Safe to retry
Give the group and its tasks idempotency_keys, and a retried request continues the same group instead of starting a second or counting a task twice.
Nothing slips in late
Enqueueing checks the group is open in the same transaction, so a task can't join after the close and be missed. Batches go in whole or not at all.
What groups don't do.
One level deep
No dependencies between tasks, chains or nested groups. To run steps in order around a group, use a workflow.
The callback has counts only
The callback says how many succeeded and failed, not what each task returned. Tasks store their results where your app reads them; failed ones are a filter away.
One execution
A group is a single batch, not a reusable definition. A nightly job creates a new group each night, and can give them all the same name.
Questions
- What is fan-out and fan-in?
- Fan-out is splitting a job into many smaller tasks that run in parallel, such as one task per customer or per file. Fan-in is doing something once they have all finished, such as building a report from their results. A task group gives you both: enqueue a task per item into the group, close it, and get one callback when every task has finished.
- How do I run something after a batch of background jobs finishes?
- Create a group with an on_complete URL, enqueue the jobs into it, and close it. When the last job finishes, Workly calls the URL once with the group's counts. Without a callback you can poll the group instead.
- What happens if some tasks fail?
- They are retried like any task. A task that runs out of attempts doesn't stop the others; once everything has finished, the group is failed and its callback says how many failed. Retry a failed task while the group is still running and the group waits for it.
- How big can a group be?
- Groups are for batches of hundreds or thousands of tasks; the SDKs' groups.enqueue sends them 100 at a time and closes the group at the end. The tasks are delivered at their queue's rate and concurrency, so a large group never floods the endpoint behind it.
- How is a task group different from a workflow?
- A group runs many tasks of one kind in parallel and tells you when they are done. A workflow runs different steps in dependency order. They work together: a workflow step can hand off to a group, and the next step starts once the group has succeeded.
Try it in five minutes.
workly dev gives you a server and dashboard on your laptop with nothing else to start, and the Python and TypeScript SDKs let your app enqueue to it. Or sign in to hosted Workly and get a workspace and an API key in a minute, free on the Hobby plan.