Background workers and task queues let a SaaS application move slow, retryable, or bursty work out of the synchronous request path. Instead of making a user wait while an API sends email, generates a report, processes an upload, or calls a third-party service, the web process records a job and a separate worker executes it.
A common small-team architecture is:
user request
→ web/API process
→ enqueue job
→ queue
→ worker process
→ database / object storage / external API
Raff Technologies can host the web process and workers on cloud VMs, while managed databases, Valkey, object storage, private networking, and other services can support the surrounding architecture. The application team still owns queue semantics, retry behavior, idempotency, worker code, and workload-specific monitoring.
This guide explains when background jobs are useful, when a queue is justified, how workers should scale, and which reliability controls matter before the pattern reaches production.
Background jobs solve latency and workload-isolation problems
Not every operation belongs inside an HTTP request.
Good background-job candidates include:
- sending transactional email;
- image or video processing;
- generating exports and reports;
- webhook delivery;
- importing large files;
- calling slow or rate-limited third-party APIs;
- search indexing;
- billing reconciliation;
- scheduled maintenance;
- AI or batch inference jobs;
- retrying work after a temporary dependency failure.
The main goal is not "use a queue because queues are scalable." It is to keep user-facing requests responsive and isolate work whose duration, retry model, or resource profile differs from normal web traffic.
A task that takes 20 seconds because an external API is slow should not necessarily hold an HTTP connection open for 20 seconds.
A worker is not the same thing as a queue
The two concepts are related but separate.
A background worker is a process that executes work outside the main request handler.
A queue is a system that stores jobs until workers can consume them.
You can have workers without a durable queue. For example, a cron process may execute a script every hour. You can also have a queue with several worker types consuming different job classes.
The queue becomes more valuable when work needs:
- buffering;
- retry;
- prioritization;
- concurrency control;
- distribution across processes or servers;
- durability across worker restarts;
- observability of pending/failed work.
For simpler recurring work, a cron job may be enough. Raff already covers that decision in Cron Jobs vs Queues vs Workflow Automation.
The smallest useful SaaS worker architecture
A small SaaS product often starts with the web process and worker on the same VM.
VM
├── reverse proxy
├── web/API service
└── worker service
managed/external services
├── database
├── queue/cache
└── object storage
That is not automatically bad architecture.
Keeping the worker on the same VM is reasonable when:
- job volume is low;
- CPU/memory usage is modest;
- web and worker workloads do not interfere materially;
- one failure domain is acceptable;
- scaling them together is still efficient.
Split workers onto a separate VM when worker load begins competing with request traffic, when worker concurrency needs independent scaling, or when failures should not consume the same resources as the web tier.
This follows the same principle as Single Server vs Multi-Server Architecture: split a component because its workload or failure characteristics justify independence, not because a diagram looks more sophisticated.
Queue choice should follow workload semantics
Different queue systems make different trade-offs.
A small team may use:
- Redis/Valkey-backed queue libraries such as BullMQ, Celery-compatible transports, Sidekiq-style systems, or framework-native queues;
- RabbitMQ or another message broker when routing and messaging semantics justify it;
- a managed cloud queue;
- a database-backed queue for modest workloads where operational simplicity matters more than extreme throughput.
The decision should consider:
- delivery guarantees;
- job durability;
- expected queue depth;
- ordering requirements;
- retry behavior;
- visibility/acknowledgement model;
- delay/scheduling features;
- throughput;
- operational ownership.
Do not select a queue solely because its benchmark throughput is high. Many small SaaS systems care more about predictable retries and simple recovery than maximum messages per second.
If Valkey is already part of the data layer, Redis Cache Strategy for SaaS Apps: Queues, Valkey & Sessions covers the broader data-layer trade-offs.
Design jobs to be idempotent
Retries are normal in queue systems.
A worker can crash after performing an external action but before marking the job complete. A dependency can time out even though the remote service actually processed the request. A message can be delivered again.
Therefore, important jobs should be designed so repeating them does not create an incorrect result.
Examples:
- use an idempotency key for payment or webhook operations;
- store a unique job/business-operation identifier;
- check whether an export has already completed before generating another;
- use database constraints to prevent duplicate records;
- make state transitions explicit.
"Exactly once" should not be assumed simply because a queue library exists.
Retries need limits and backoff
A production queue should define what happens after failure.
A useful retry policy answers:
- Which errors are retryable?
- How many attempts are allowed?
- How long is the delay between attempts?
- Should delay increase after repeated failure?
- What happens after the final attempt?
Retrying a permanent failure immediately can create a retry storm and increase pressure on the failing dependency.
Exponential backoff is commonly used for temporary failures because it spaces repeated attempts. Queue libraries such as BullMQ expose retry attempts and backoff controls; the exact mechanism varies by implementation.
Do not retry validation errors, malformed payloads, or other clearly permanent failures as if they were network timeouts.
Failed jobs need a terminal state
If a job exhausts its retry policy, it should not disappear silently.
Depending on the queue technology, failed work may remain in a failed set or move to a dead-letter queue.
The operational goal is the same:
job fails
→ retry policy
→ still fails
→ terminal failed/dead-letter state
→ alert / inspect / repair / replay
Dead-letter handling is particularly useful when you need to separate poison jobs from normal traffic.
Track at least:
- failed job count;
- oldest failed job;
- failure reason;
- job type;
- attempt count;
- whether replay succeeded.
Queue depth is a capacity signal
Queue length by itself is not enough.
A queue of 1,000 one-second jobs and a queue of 1,000 five-minute jobs have completely different capacity implications.
More useful signals include:
- jobs waiting;
- oldest-job age;
- enqueue rate;
- completion rate;
- average and p95 execution duration;
- retry rate;
- failure rate;
- worker utilization.
The most important operational question is often:
Is work entering the queue faster than workers can finish it?
If yes, backlog age will increase even if workers are technically healthy.
Scale workers from throughput, not guesses
A simple sizing model is:
required throughput ≈ incoming jobs / time
worker throughput ≈ concurrency ÷ average job duration
If jobs arrive at 120 per minute and one worker process completes 30 per minute, you need roughly four equivalent worker units just to keep pace before adding headroom.
Resource type also matters.
CPU-heavy jobs may need lower concurrency and more CPU.
I/O-heavy jobs waiting on APIs or storage may support higher concurrency without the same CPU demand.
Memory-heavy jobs may need concurrency caps even when CPU is idle.
Use measured queue age, job duration, CPU, memory, and dependency limits before increasing worker count.
Separate queues when workload behavior differs
One queue can be enough initially.
Separate queues become useful when job classes have different priorities or resource profiles.
For example:
critical-email
image-processing
report-generation
webhook-delivery
bulk-import
A long-running report should not block password-reset email if both share one low-concurrency queue.
Separate queues or worker pools allow:
- different concurrency;
- different retry policies;
- priority isolation;
- different VM sizes;
- independent scaling.
Avoid creating dozens of queues before the workload needs them. Queue topology is operational complexity too.
Long-running jobs need special treatment
A job that runs for seconds is different from one that runs for hours.
For long-running work, consider:
- heartbeat/lease behavior;
- progress tracking;
- interruption safety;
- timeout configuration;
- checkpointing;
- partial output cleanup;
- worker restart behavior.
If a task is extremely long-lived, compute-heavy, or batch-oriented, compare queue workers with Long-Running Serverless Functions for AI, ETL, and Batch Jobs and ordinary VM-based batch processing.
Cron jobs and queues solve different scheduling problems
Cron answers:
Run this work at a specific time or interval.
A queue answers:
Store this unit of work until a worker can process it.
They can be combined:
cron/scheduler
→ enqueue 10,000 jobs
→ worker pool consumes jobs safely
That is often better than one cron process trying to complete all 10,000 items sequentially.
The existing Cron Jobs vs Queues vs Workflow Automation guide owns that decision boundary.
Monitor workers as production services
A worker can fail while the web application still returns HTTP 200.
That means ordinary uptime monitoring is insufficient.
Monitor:
- worker process health;
- queue depth;
- oldest-job age;
- completion rate;
- failure/retry rate;
- execution latency;
- dependency errors;
- dead-letter/failed jobs.
Alert on symptoms that affect business processing, not only whether the process PID exists.
For broader operational monitoring, continue to Application Observability for Small Teams.
Protect the worker trust boundary
Background workers often have powerful credentials because they send email, access storage, modify databases, or call billing providers.
Apply least privilege.
A worker responsible only for image thumbnails should not automatically inherit billing credentials.
Separate:
- queue credentials;
- database permissions;
- object-storage access;
- third-party API keys;
- environment secrets.
Also treat job payloads as untrusted input. Do not deserialize arbitrary executable objects or run shell commands directly from queue data without strict validation.
When to keep background work synchronous
A queue is not automatically worth the operational cost.
Keep work synchronous when:
- execution is fast and predictable;
- the user needs the result immediately;
- retry semantics would be more confusing than useful;
- queue infrastructure would exceed the complexity of the task.
Moving a 20 ms database write into a queue can make correctness and debugging harder without providing meaningful latency benefit.
Use background workers where isolation, buffering, retry, or independent scaling solves a real problem.
Background worker production checklist
Before relying on workers in production:
- jobs have stable identifiers;
- retries are bounded;
- retryable and permanent failures are distinguished;
- important jobs are idempotent;
- failed work remains inspectable;
- queue depth and oldest-job age are monitored;
- worker concurrency is measured against CPU/memory/dependency limits;
- credentials are least-privilege;
- long-running jobs have timeout/recovery behavior;
- deployment does not abandon in-flight jobs unexpectedly;
- critical and bulk workloads cannot starve each other.
Frequently asked questions
What is a background worker?
A background worker is a process that executes application work outside the user-facing request path, such as email delivery, report generation, file processing, or webhook retries.
What is a task queue?
A task queue stores jobs until workers can process them. It can provide buffering, retries, concurrency control, durability, and workload distribution.
Do small SaaS apps need a queue?
Not always. A queue becomes useful when work is slow, bursty, retryable, or needs independent scaling. Simple fast operations are often better kept synchronous.
Should workers run on the same VM as the web app?
They can at small scale. Split them when worker CPU, memory, concurrency, scaling, or failure behavior needs to be independent from request traffic.
What is a dead-letter queue?
A dead-letter queue or equivalent failed-job state holds jobs that could not be processed successfully after their retry policy, allowing operators to inspect and replay them safely.
How do I scale queue workers?
Measure incoming job rate, execution duration, queue age, CPU, memory, and dependency limits. Add worker concurrency or instances when completion capacity is below sustained incoming work.
Sources
