sellStorage & Databases

linear_scale

Pipelines

Fire-hose of events in, neatly stored files in R2 out — no servers to babysit.

R2Delivers into R2

HTTPIngest endpoint

OnceExactly-once delivery

BetaOpen beta

lightbulb

What are Pipelines?

Pipelines ingest streaming data (streaming data = a continuous flow of small events arriving non-stop, like clicks or sensor readings), optionally transform it, and write it neatly into R2 object storage for you.

A pipeline has three parts: a stream (where events come in, via an HTTP endpoint or a Worker), an optional SQL transformation (to filter or reshape events), and a sink (the R2 destination where files are saved, as Iceberg tables, Parquet, or JSON).

water_drop

Think of it like…

A water treatment plant. A messy stream of raw events flows in (stream), gets filtered and cleaned (SQL transform), and is bottled into tidy files on a shelf (R2 sink) ready to drink later.

help

Why use Pipelines?

Collecting a non-stop flood of events reliably is hard: you must buffer spikes, batch them into files, and never lose or duplicate data. Pipelines does all of that as a managed service, then hands you clean files in R2.

shield

Durable ingestion

Events are safely buffered so traffic spikes do not drop data.

done_all

Exactly-once delivery

Each event lands in R2 once — no missing rows, no duplicates.

filter_alt

Transform on the way

Use SQL to filter, reshape, or enrich events before they are stored.

savings

Cheap to store

Output lands in R2, which has zero egress fees when you read it back.

target

When should you use it?

ads_click

Clickstream analytics

Capture every page view and click to analyze user behavior later.

receipt_long

Logs & events

Stream server logs or app events into R2 for archiving and querying.

sensors

IoT telemetry

Collect readings from many devices into one organized data lake.

insights

Feeding a data lake

Land structured files in R2 ready for big-data tools to query.

rocket_launch

How do you start?

The guided setup command creates the stream, the SQL transform, and the R2 sink in one go. Then you send events to the HTTP endpoint and query the stored data.

Install Wrangler & log in
Pipelines is available on the Workers Paid plan during open beta.
bash
```
npm install -g wrangler
wrangler login
```
Run the guided setup
This walks you through naming the pipeline, enabling the HTTP endpoint, choosing an R2 bucket as the sink, and picking a transform.
bash
```
npx wrangler pipelines setup --name ecommerce
```

Send events to the endpoint

POST a JSON array of events to your stream's ingest URL. Replace {stream-id} with the id from setup.

bash

curl -X POST https://{stream-id}.ingest.cloudflare.com \
  -H "Content-Type: application/json" \
  -d '[{"user_id":"u_123","event":"purchase","amount":29.99}]'

Send events from a Worker

You can also push events from Worker code by POSTing to the same ingest URL.

export default {
  async fetch(request, env) {
    const event = { user_id: "u_123", event: "page_view", path: "/home" };
    await fetch("https://{stream-id}.ingest.cloudflare.com", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify([event]),
    });
    return new Response("event sent");
  },
};

Query the stored data

Once events land in R2, query them with R2 SQL using your warehouse name.

bash

export WRANGLER_R2_SQL_AUTH_TOKEN=your_api_token

npx wrangler r2 sql query "YOUR_WAREHOUSE_NAME" \
  "SELECT user_id, event, amount FROM default.ecommerce LIMIT 10"

school

Key concepts

input

Stream

The entry point where events arrive — via an HTTP endpoint or a Worker binding.

transform

SQL transformation

An optional SQL step that filters, reshapes, or enriches events as they pass through.

output

Sink

The R2 destination where data is written as Iceberg, Parquet, or JSON files.

inventory

Batching

Many small events are grouped into larger files so storage and querying stay efficient.

tips_and_updates

Tips & status

science

Open beta

Pipelines is in open beta on the Workers Paid plan. During beta you are not billed for Pipelines itself — you only pay standard R2 storage costs for the data it writes.

Send events as a JSON array so you can batch many records in one request.
Pick Parquet or Iceberg output if you plan to query the data with big-data tools.
Because output lives in R2, reading your data back out costs zero egress.
Need a simple message queue between Workers instead? Look at Queues.