Back to blog

Building Resilient Background Task Consumers with Message Queues

Learn how to architect robust background queue consumers using idempotency, bounded batches, and dead-letter queues to prevent data loss and system freezes.

Oct 24, 2026
7
directions
notes from delivery practice
3
levels
strategy · process · tools
1
library
guides · checklists · cases
See cases

Why Long-Running Jobs Fail in Web Requests

In modern web engineering, applications frequently need to execute resource-heavy operations: cleaning up expired user uploads, aggregating large-scale analytic reports, or performing extensive database migrations. Attempting to handle these workloads inside standard HTTP request-response cycles inevitably leads to connection timeouts, thread pool starvation, and degraded user experience.

The standard architectural solution is delegating heavy tasks to a message queue. The web layer quickly emits a job descriptor to a broker (such as RabbitMQ, AWS SQS, or Redis) and returns an immediate response. Autonomous background workers consume and execute these jobs asynchronously. However, introducing a queue does not automatically make distributed execution safe unless the consumer is designed for failure recovery from day one.

Core Challenges of Distributed Workers

  • At-least-once delivery: Most message brokers guarantee delivery at least once. Duplicate message deliveries will occur after network blips, unacknowledged timeouts, or process terminations.
  • Database contention: Processing tens of thousands of records in a single monolithic transaction causes severe row locks that can freeze primary application traffic.
  • Crash recovery: When a worker process dies mid-execution, the system must recover smoothly without leaving inconsistent data or running conflicting duplicate routines.

Idempotency: The Core of Fault Tolerance

Every cleanup or background routine must be idempotent. Executing the same job payload twice must leave the system in the exact same state as executing it once. This requires decoupling the queue's message ID from the underlying business operation key (e.g., a compound key consisting of tenant identifier and cleanup period).

By tracking operation status (pending, running, completed, failed) in persistent storage, the worker can handle redeliveries gracefully. A message received with an existing "running" state does not imply the job succeeded; it signals a previous worker crashed, prompting the new consumer to inspect the checkpoint cursor and resume execution safely.

Bounded Batches and Resource Discipline

Rather than processing huge datasets in one monolithic transaction, work should always be partitioned into bounded batches. For instance, removing 20,000 stale database records is best done in chunks of 500 records per transaction commit.

Bounded batching keeps memory footprints low, prevents database locks from escalating, and ensures processing times fit well within the queue's message visibility window. If a consumer crashes mid-way, only the current batch needs re-evaluation, keeping recovery time negligible.

Acknowledgement Timing and Dead-Letter Queues

The exact timing of the queue acknowledgement (ACK) determines whether your pipeline loses data during an outage. An ACK must only be dispatched after all database transactions and external API mutations have reached absolute durability. Acknowledging before persistence causes data loss on crash; acknowledging too late without extending the message lease causes the broker to redeliver the job to another worker, triggering race conditions.

Persistent failures must be isolated through a Dead-Letter Queue (DLQ). When a job repeatedly fails after an exponential backoff policy (for instance, due to corrupted tenant data), routing it to a DLQ prevents poison messages from blocking normal traffic. The DLQ acts as a recovery surface, preserving the error context so DevOps and backend engineers can inspect, fix, and replay tasks without disrupting business continuity.

Services on this topic

Need this for your project?

Development turnkey: analysis → implementation → launch → support. Tell us about your task — we will reply with an estimate.

About the direction

Relevant articles