Blueprint: an image upload & processing pipeline

Image processing is slow — resizing and converting a big photo can take seconds. If you do it while the user waits, uploads feel sluggish and a single failure loses the whole job. This blueprint splits the work into two halves: the front of the pipeline accepts the file and replies instantly, while the back of the pipeline does the heavy lifting in the background.

The glue between the two halves is a queue. The Upload Worker stores the original in R2 (object storage) and drops a tiny job message into Cloudflare Queues. A separate Consumer Worker picks that job up later, optimizes the image with Cloudflare Images, writes the variants back to R2, and records the result in D1 (a SQL database). This is the classic producer–consumer pattern.

Each product in the pipeline has one clear job. Keeping responsibilities separate is what makes the system reliable: any one part can be slow or briefly fail without breaking the others.

The decoupling boundary

The queue is a one-way valve between the producer and the consumer. The producer never waits for the consumer; if the consumer is busy or fails, jobs simply wait and retry. Neither side needs to know the other is healthy.

D1 tracks every image and its variants. The images table is the headline record (id, R2 key, status, dimensions, variant count). A variants table holds one row per generated size, linked back by image_id.

CREATE TABLE images (
  id        TEXT PRIMARY KEY,
  key       TEXT NOT NULL,
  status    TEXT NOT NULL DEFAULT 'pending',
  width     INTEGER,
  height    INTEGER,
  variants  INTEGER NOT NULL DEFAULT 0,
  created_at TEXT NOT NULL DEFAULT (datetime('now'))
);

CREATE TABLE variants (
  id        TEXT PRIMARY KEY,
  image_id  TEXT NOT NULL REFERENCES images(id),
  label     TEXT NOT NULL,
  key       TEXT NOT NULL,
  width     INTEGER NOT NULL
);

-- Fast lookups of jobs still waiting to be processed
CREATE INDEX idx_images_status ON images (status);

Read this top to bottom. The crucial moment is the 202 reply: the upload returns before any processing happens. Everything below the dashed boundary runs asynchronously in the background.

You'll create three resources (an R2 bucket, a queue, a D1 database), wire them as bindings, then write a producer, a consumer, and a tiny front-end. The Images binding needs no resource — it's a runtime capability.

1. Create the resources

   One R2 bucket holds both originals and variants; one queue carries the jobs; one D1 database stores metadata.

   bashcontent_copyCopy

   npx wrangler r2 bucket create user-images
   npx wrangler queues create image-jobs
   npx wrangler queues create image-jobs-dlq
   npx wrangler d1 create image-meta

2. Wire the bindings

   Declare R2, the queue (as both producer and consumer), D1, and the Images binding in wrangler.jsonc. A dead letter queue catches jobs that keep failing.

   jsonccontent_copyCopy

   {
     "name": "image-pipeline",
     "main": "src/index.js",
     "compatibility_date": "2025-01-01",
   
     "r2_buckets": [
       { "binding": "IMAGES_BUCKET", "bucket_name": "user-images" }
     ],
     "queues": {
       "producers": [
         { "queue": "image-jobs", "binding": "IMAGE_QUEUE" }
       ],
       "consumers": [
         {
           "queue": "image-jobs",
           "max_batch_size": 10,
           "max_batch_timeout": 5,
           "dead_letter_queue": "image-jobs-dlq"
         }
       ]
     },
     "d1_databases": [
       { "binding": "DB", "database_name": "image-meta", "database_id": "<your-d1-id>" }
     ],
     "images": { "binding": "IMAGES" }
   }

3. Create the tables

   Apply the schema to your D1 database.

   bashcontent_copyCopy

   npx wrangler d1 execute image-meta --remote --file=./schema.sql

Front-end — upload the file

<input type="file" id="file" accept="image/*" />
<button id="send">Upload</button>

<script>
  document.getElementById("send").onclick = async () => {
    const file = document.getElementById("file").files[0];

    // Stream the raw bytes straight to the Upload Worker
    const res = await fetch("/upload", {
      method: "POST",
      headers: { "content-type": file.type },
      body: file,
    });

    const job = await res.json(); // { id, status: "pending" }
    console.log("Queued job", job.id, "->", job.status);
  };
</script>

Producer — accept & enqueue

export default {
  // PRODUCER: store the upload, then enqueue a background job
  async fetch(request, env, ctx) {
    if (request.method !== "POST") {
      return new Response("Use POST to upload", { status: 405 });
    }

    const id = crypto.randomUUID();
    const key = `originals/${id}`;
    const bytes = await request.arrayBuffer();
    const contentType = request.headers.get("content-type") || "image/jpeg";

    // 1) Save the ORIGINAL image to R2
    await env.IMAGES_BUCKET.put(key, bytes, {
      httpMetadata: { contentType },
    });

    // 2) Record metadata in D1 with status = pending
    await env.DB.prepare(
      "INSERT INTO images (id, key, status) VALUES (?, ?, 'pending')"
    ).bind(id, key).run();

    // 3) Enqueue a job and reply instantly (do NOT wait for processing)
    await env.IMAGE_QUEUE.send({ id, key });

    return Response.json({ id, status: "pending" }, { status: 202 });
  },
};

Consumer — process the batch

export default {
  // CONSUMER: Cloudflare invokes this with a batch of queued jobs
  async queue(batch, env, ctx) {
    for (const message of batch.messages) {
      try {
        const { id, key } = message.body;

        // 1) Read the original back from R2
        const original = await env.IMAGES_BUCKET.get(key);
        if (!original) {
          message.ack(); // nothing to do
          continue;
        }

        // 2) Build optimized variants with the Images binding
        const sizes = [256, 1024];
        const variantKeys = [];
        for (const width of sizes) {
          const result = await env.IMAGES
            .input(original.body)
            .transform({ width })
            .output({ format: "image/webp" });

          const variantKey = `variants/${id}/w${width}.webp`;
          await env.IMAGES_BUCKET.put(variantKey, result.image());
          variantKeys.push(variantKey);
        }

        // 3) Mark the row ready and record how many variants exist
        await env.DB.prepare(
          "UPDATE images SET status = 'ready', variants = ? WHERE id = ?"
        ).bind(variantKeys.length, id).run();

        message.ack(); // success — never deliver again
      } catch (err) {
        message.retry(); // failure — Queues will redeliver later
      }
    }
  },
};

Serve — an Images transform URL

You can also resize on the fly straight from a URL with Cloudflare Images' transformation path. The browser asks only for the size it needs, and the CDN caches the result at the edge.

<!-- Resize + convert on the fly with a transformation URL -->
<!-- /cdn-cgi/image/<options>/<source-path> -->

<img
  src="https://media.example.com/cdn-cgi/image/width=512,quality=80,format=auto/variants/abc123/w1024.webp"
  width="512"
  alt="optimized photo"
/>

Things to remember

• Always configure a Dead Letter Queue so jobs that keep failing land somewhere you can inspect instead of vanishing.
• Store status (pending / ready / failed) so the front-end can poll or show a spinner until variants exist.
• Keep queue messages small — send only the id and R2 key, never the image bytes (128 KB message limit).
• R2 and Queues charge no egress fees; store the variants you generate instead of transforming on every request.
• Tune max_batch_size and max_batch_timeout to trade latency for throughput when traffic spikes.