Event-driven apps with Queues

Some work is slow: sending an email, generating a thumbnail, syncing data to another system. If you do it while the user waits, the page feels sluggish and one hiccup can lose the work. An event-driven design fixes both problems: the request only records 'this needs doing' and returns immediately; the actual work happens later, in the background, reliably.

On Cloudflare the glue is Queues — a reliable waiting line for messages. A producer Worker accepts the request and drops a message into the queue. A consumer Worker is invoked later with a batch of messages and does the heavy lifting (write to D1, store a file in R2, call an external API). The two Workers never call each other directly; the queue sits in the middle. That separation is called decoupling.

Notice the two arrows leaving the producer. One goes into the queue (send the message). The other goes straight back to the caller with 202 Accepted — before any slow work has happened. That fork is the heart of an event-driven app.

An event-driven flow has five concepts you keep meeting: the producer, the queue, the consumer, plus the reliability features — batching, retries and the dead-letter queue. Here is each in one line.

Imagine a sign-up form. The user submits, and we owe them a welcome email. The trick is: reply 202 Accepted the moment the job is queued, and send the email afterwards. Here is the whole round trip — watch how the user is set free before the email is ever sent.

What each step means

• POST /signup: the browser submits the form to the producer Worker.
• queue email job: the producer calls send() to drop a message describing the work.
• 202 Accepted: the producer replies right away — '202' means 'I accepted this; I'll process it later'.
• deliver batch later: Cloudflare invokes the consumer with a batch when messages are ready.
• send the email: the consumer does the actual slow work against the mail service.
• ack the message: on success the consumer acknowledges, so the message leaves the queue for good.

Background work fails sometimes — a mail server is down, an API times out. Queues handles this with a simple rule: a message stays alive until it is acked. If processing throws (or you call retry()), the message is redelivered later. After max_retries failed attempts it is moved to a dead-letter queue (DLQ) instead of being dropped.

Idempotency: surviving duplicates

Queues delivers at-least-once: a message may occasionally be delivered more than once (for example, the consumer succeeds but crashes before acking). So your handler must be idempotent — running it twice with the same message produces the same result as running it once. The classic trick: give every message a unique id, record ids you've finished, and skip any id you've already seen.

Here is a complete, runnable example: a producer Worker that queues a job and returns 202, a consumer Worker that processes a batch idempotently and acks, the wrangler.toml that wires the queue (with retries and a DLQ), the D1 table that tracks finished ids, and the commands to create everything and deploy.

1. 1. Producer Worker — accept & return 202

   On each request it builds a message with a unique id, sends it to the queue via the MY_QUEUE binding, and replies 202 immediately. No slow work happens here.

   jscontent_copyCopy

   // src/producer.js
   export default {
     // PRODUCER: runs on every HTTP request
     async fetch(request, env, ctx) {
       const { email } = await request.json();
   
       // Build a job with a unique id (used later for idempotency)
       const msg = {
         id: crypto.randomUUID(),
         type: "welcome_email",
         email,
         ts: Date.now(),
       };
   
       // Hand the slow work to the queue, then reply immediately
       await env.MY_QUEUE.send(msg);
   
       return new Response(JSON.stringify({ queued: true }), {
         status: 202,
         headers: { "content-type": "application/json" },
       });
     },
   };

2. 2. Consumer Worker — process a batch & ack

   Cloudflare calls queue() with a batch. We loop over batch.messages, skip ids already done (idempotency), do the slow work, then ack on success or retry on failure. Failed messages are redelivered and, after max_retries, go to the DLQ automatically.

   jscontent_copyCopy

   // src/consumer.js
   export default {
     // CONSUMER: Cloudflare calls this with a batch of messages
     async queue(batch, env, ctx) {
       for (const m of batch.messages) {
         try {
           const job = m.body;
   
           // Idempotency: skip if this id was already handled
           const done = await env.DB
             .prepare("SELECT 1 FROM processed WHERE id = ?")
             .bind(job.id)
             .first();
           if (done) {
             m.ack();
             continue;
           }
   
           // Do the slow work: send the email
           await sendEmail(env, job.email);
   
           // Record the id so a retry never sends twice
           await env.DB
             .prepare("INSERT INTO processed (id, ts) VALUES (?, ?)")
             .bind(job.id, Date.now())
             .run();
   
           m.ack(); // success -> remove from the queue
         } catch (err) {
           m.retry(); // failure -> redeliver later, then DLQ
         }
       }
     },
   };

3. 3. wrangler.toml — wire producer, consumer & DLQ

   queues.producers exposes the queue to your code as env.MY_QUEUE. queues.consumers tells Cloudflare to invoke queue() with batches, and sets batch size, retries and the dead_letter_queue where exhausted messages land.

   tomlcontent_copyCopy

   name = "queue-app"
   main = "src/index.js"
   compatibility_date = "2025-01-01"
   
   # PRODUCER: send to the "jobs" queue as env.MY_QUEUE
   [[queues.producers]]
   queue = "jobs"
   binding = "MY_QUEUE"
   
   # CONSUMER: Cloudflare invokes queue() with batches from "jobs"
   [[queues.consumers]]
   queue = "jobs"
   max_batch_size = 10
   max_batch_timeout = 5
   max_retries = 3
   dead_letter_queue = "jobs-dlq"
   
   # D1 used for idempotency bookkeeping (env.DB)
   [[d1_databases]]
   binding = "DB"
   database_name = "queue-app-db"
   database_id = "<paste-your-database-id>"

4. 4. D1 table — remember finished ids

   A tiny table keyed by the message id. The consumer inserts a row after finishing, and checks this table first to stay idempotent across retries and duplicate deliveries.

   sqlcontent_copyCopy

   -- schema.sql
   CREATE TABLE IF NOT EXISTS processed (
     id  TEXT PRIMARY KEY,
     ts  INTEGER NOT NULL
   );

5. 5. Create queues + DLQ, then deploy

   Create the main queue and the dead-letter queue (it's just another queue), create the D1 database, load the schema, deploy, and tail the logs to watch messages being consumed live.

   bashcontent_copyCopy

   npx wrangler queues create jobs
   npx wrangler queues create jobs-dlq
   npx wrangler d1 create queue-app-db
   npx wrangler d1 execute queue-app-db --remote --file=./schema.sql
   npx wrangler deploy
   npx wrangler tail

Things beginners trip over

• Forgetting to ack: an un-acked message is treated as failed and redelivered — call m.ack() on success.
• No idempotency: at-least-once delivery means duplicates happen; guard every side effect with a unique id check.
• Doing the slow work in the producer: keep fetch() fast — only send() and return 202; the consumer does the heavy lifting.
• Tiny batches: tune max_batch_size and max_batch_timeout so consumers process efficiently instead of one message at a time.
• Ignoring the DLQ: messages can sit there silently — add an alert or a periodic check so you notice failures.