HTTP-based vs workers
Two ways to run background jobs: workers pull them, or a queue pushes them to you.
Celery, Sidekiq and BullMQ hand jobs to worker processes that pull from a broker. Cloud Tasks, QStash and Workly send each job to your app as an HTTP request. Both retry and both scale, but they ask different things of your infrastructure, and each is better at some work.
The difference is who starts the work.
With workers, a process you run asks the broker for the next job and runs your function. With HTTP push, the queue calls a route in your app and reads the response: 2xx means done, anything else is retried. The job's code is the same; what changes is what you deploy around it.
Workers pull
HTTP push
With workers, the last box is a separate deployment. With HTTP push, it is your app again: no worker processes, and the queue can be hosted.
# tasks.py
from celery import Celery
app = Celery("app", broker="redis://localhost:6379/0")
@app.task(autoretry_for=(Exception,), retry_backoff=True, max_retries=5)
def send_email(user_id):
...
# In a request handler
send_email.delay(123)
# And in production, alongside the app:
# redis-server
# celery -A tasks workerfrom workly import SignatureVerificationError, Workly, verify_signature
workly = Workly() # WORKLY_URL, WORKLY_API_KEY
# In a request handler
workly.tasks.enqueue(
"emails",
url="https://api.example.com/tasks/send-email",
payload={"user_id": 123},
)
# The task is an endpoint in the app you already run
@app.post("/tasks/send-email")
def send_email():
try:
verify_signature(request.get_data(), request.headers)
except SignatureVerificationError:
abort(401)
user_id = request.get_json()["user_id"]
...
return "", 204What HTTP push is good at.
Nothing new to deploy
A task is a route in the app you already run. No worker processes to deploy, scale and keep on the same version of your code.
Any language
Anything that answers HTTP can handle a task, so one queue can serve a Python API and a Node service.
Works on serverless
Vercel, Cloud Run, Lambda and Cloudflare Workers can answer a request but not run a worker that waits for jobs.
Limits across every delivery
The queue sends the requests, so its rate and concurrency limits hold however many instances of your app are running.
A record of every attempt
Each attempt ends in a status code and a response body, which the queue can keep and show you.
And where it falls short.
A request has a time limit
An attempt has to finish within the queue's timeout (30 minutes in Workly) and your platform's own. Work that runs for hours needs splitting into tasks, or a worker.
The queue must reach your endpoint
A hosted queue needs a public URL; a self-hosted one can call private addresses. Either way, the handler checks the signature, as for a webhook.
An HTTP request per task
Each delivery goes through your web stack. Fine for thousands of tasks a second, but not the cheapest way to run millions of tiny jobs.
What workers are good at.
Jobs can run for hours
A worker holds a job as long as it needs: video encoding, large exports, model training.
No inbound endpoint
Workers connect out to the broker, so nothing has to be reachable from outside.
High throughput
A worker fetches jobs from Redis in batches and runs them in-process, with no HTTP request per job.
And what they cost you.
More to run
A broker such as Redis or RabbitMQ, worker processes scaled separately from your app, and often a scheduler process for cron.
One language per queue
Workers run functions from your codebase, so a Celery queue is for Python and a BullMQ queue for Node.
Visibility is extra
What a job did lives in worker logs. Seeing failures and retrying them takes another tool, such as Flower or Bull Board.
Limits per worker
Rate limits are often enforced by each worker on its own, so adding workers raises the total.
Which tools push, and which pull.
Each comparison covers the tool's model in detail, and when it is the better choice.
| Tool | Model | How it runs your jobs | Comparison |
|---|---|---|---|
| Workly | HTTP push | Delivers each task to your endpoint as a signed HTTP request. Open source; self-hosted or at workly.run. | |
| Google Cloud Tasks | HTTP push | A managed HTTP push queue on Google Cloud. | Compare → |
| Upstash QStash | HTTP push | Hosted HTTP messaging: publish a message and QStash calls your URL. | Compare → |
| Inngest | HTTP push | Calls your app's serve endpoint once per step of a function. Its connect mode is worker-style instead. | Compare → |
| Celery, RQ, Dramatiq | workers | Python workers pull jobs from Redis or RabbitMQ. | Compare → |
| BullMQ | workers | Node.js workers pull jobs from Redis. | Compare → |
| PgQueuer, River, pg-boss | workers | A library in your own processes pulls jobs from your Postgres. | Compare → |
| Sidekiq | workers | Ruby workers pull jobs from Redis. | |
| Temporal, Hatchet | workers | Your workers poll the server for workflow and activity tasks. |
From each tool's documentation, November 2026. All comparisons →
Choosing between them.
Push over HTTP when
- your jobs finish in minutes: emails, webhooks, syncs, calls to other APIs;
- your app runs on serverless, or you'd rather not run workers and a broker;
- several services in different languages share background work;
- you want to see every attempt without adding a monitoring tool.
Use workers when
- a single job runs for hours and can't be split;
- nothing may accept inbound requests;
- you run millions of tiny jobs and the cost per job matters most.
Questions
- What is an HTTP-based task queue?
- A queue that delivers each job as an HTTP request to an endpoint in your app, instead of handing it to a worker process that pulls from a broker. The queue retries the request until your endpoint answers 2xx, so your background job is an ordinary route. Google Cloud Tasks, QStash and Workly work this way.
- Is pushing over HTTP slower than workers?
- Per job, yes: each delivery is an HTTP request rather than a function call inside a worker. In practice it is rarely the limit. On one laptop, Workly delivers about 4,000 tasks a second on Postgres and 6,000 on SQLite, and how fast your handlers answer matters more.
- Can an HTTP task queue run long jobs?
- Up to its timeout: 30 minutes per attempt in Workly, 30 minutes in Cloud Tasks, and less on many serverless platforms. Longer work is usually split into tasks, for example one per file or per customer, and a task group tells you when they have all finished. Work that has to run for hours in one go is better in a worker.
- Can I use both?
- Yes. A common split is HTTP tasks for most background work, such as emails, webhooks and syncs, and a few workers for the long jobs. A task can also start a worker job: the endpoint enqueues it and answers 2xx.
Try it in five minutes.
workly dev gives you a server and dashboard on your laptop with nothing else to start, and the Python and TypeScript SDKs let your app enqueue to it. Or sign in to hosted Workly and get a workspace and an API key in a minute, free on the Hobby plan.