---
title: stream
---

`/stream` is hydrant's ordered websocket event stream, and every subscriber gets a full copy of it. it's only in builds with the `indexer_stream` feature.

`/stream` isn't wire or protocol compatible with [`tap`](../concepts/vs-tap.md). tap sends events out through a sharded work queue, tracks acks on the server, and redelivers them. hydrant broadcasts every event to everyone and leaves the cursor to the client.

## GET /stream

subscribe to the ordered event stream.

### query parameters

| param | type | description |
| :--- | :--- | :--- |
| `cursor` | integer | `u64` event id to replay durable record events from, inclusive. |
| `stream` | string | the stream `cursor` came from (see [`/stream/head`](#get-streamhead)). if it isn't this one, hydrant sends a `StreamMismatch` error frame and closes the connection. |

without `cursor`, the connection starts at the current head and only gets new events. with it, hydrant first replays the durable record events from the cursor up to the head as it was when you connected, then switches to live events.

`cursor=0` replays every record event still stored. ids always go up but can skip numbers, because identity and account events use ids without being stored, and events hydrant can't render are skipped.

### event envelope

events are json text frames shaped like this:

```json
{
  "id": 42,
  "type": "record",
  "record": {}
}
```

| field | type | description |
| :--- | :--- | :--- |
| `id` | integer | `u64` stream event id. |
| `type` | string | `record`, `identity`, or `account`. |
| `record` | object | present when `type` is `record`. |
| `identity` | object | present when `type` is `identity`. |
| `account` | object | present when `type` is `account`. |

only the payload matching `type` is present.

### record

record events are durable, and a cursor replay can get them.

```json
{
  "id": 42,
  "type": "record",
  "record": {
    "live": true,
    "did": "did:plc:abc123xyz",
    "rev": "3kpjxabc123",
    "collection": "app.bsky.feed.post",
    "rkey": "3kpjxabc123",
    "action": "create",
    "record": {
      "text": "hello, world!",
      "$type": "app.bsky.feed.post",
      "createdAt": "2026-05-27T12:00:00.000Z"
    },
    "cid": "bafyreihy..."
  }
}
```

| field | type | description |
| :--- | :--- | :--- |
| `live` | boolean | `true` for live events, `false` for backfill. |
| `did` | string | repository did. |
| `rev` | string | repository revision, a tid. |
| `collection` | string | collection nsid. |
| `rkey` | string | record key. |
| `action` | string | `create`, `update`, or `delete`. |
| `record` | object | the decoded dag-cbor record, with keys in the record's cbor order (shorter first) instead of alphabetical. left out when there's no stored body. |
| `cid` | string | cid of `record`. left out when there's no stored body. |

`record` and `cid` are left out for deletes and when hydrant doesn't store record bodies, such as with `HYDRANT_ONLY_INDEX_LINKS=true`.

once a redaction finishes, hydrant stops serving the body, except in events already queued for a reader.

### identity

identity events are live only, and a cursor replay doesn't get them.

```json
{
  "id": 43,
  "type": "identity",
  "identity": {
    "did": "did:plc:abc123xyz",
    "handle": "user.bsky.social"
  }
}
```

| field | type | description |
| :--- | :--- | :--- |
| `did` | string | repository did. |
| `handle` | string | current handle, left out when hydrant doesn't have one. |

### account

account events are live only, and a cursor replay doesn't get them.

```json
{
  "id": 44,
  "type": "account",
  "account": {
    "did": "did:plc:abc123xyz",
    "active": false,
    "status": "deactivated"
  }
}
```

| field | type | description |
| :--- | :--- | :--- |
| `did` | string | repository did. |
| `active` | boolean | whether the repository is active. |
| `status` | string | account status, left out when hydrant doesn't have one. |

### websocket behavior

- hydrant sends an empty ping every 30 seconds and answers client pings with pongs.
- clients can't send text or binary frames. hydrant closes the connection if one does.
- when `/stream` is in `HYDRANT_THROTTLED_STREAMS` (it isn't by default), a replay from a cursor runs at most `HYDRANT_STREAM_REPLAY_SPEED` times the bandwidth the live stream would take (10 by default), or `HYDRANT_STREAM_REPLAY_FLOOR_BYTES` a second (about 80 Mbit/s by default) if that's more. the live tail never waits for it.
- a subscriber that falls behind the live broadcast catches up from the database at its own pace. it never misses a `record` event, but it misses `identity` and `account` events from while it was behind.
- if hydrant can't read the next events from the database, it closes the connection instead of skipping them. reconnect with the last `id` you got as the cursor.
- when hydrant stops, it closes the connection with close code `1012` (service restart). reconnect the same way. under systemd socket activation, the reconnect waits for the next hydrant instead of being refused.
- if a send is stuck for `HYDRANT_STREAM_SEND_TIMEOUT` (30 seconds by default), hydrant sends this error frame and closes the connection:

```json
{
  "type": "error",
  "error": "ConsumerTooSlow",
  "message": "stream socket send blocked for at least 30 seconds"
}
```

## GET /stream/head

get the id of the newest durable record event, and the stream it belongs to.

```json
{ "id": 289455, "stream": "5b0e6c8f2a1d4e7b9c3f0a6d8e2b4c71" }
```

`id` is `null` while no events are stored. `stream` is the id of this database's stream. every database gets a fresh one, including after a wipe, and two nodes never share one. an event id only means something on the stream it came from. `id` is the newest event a cursor replay can deliver, not the id counter, because identity and account events use ids without being stored.

to know when a `cursor=0` replay has caught up, get the head before subscribing and wait for a frame with an `id` at or past it. keep an idle timeout as a fallback too, because if the head event itself can't be rendered, it's skipped and never arrives.

## chunks

with `HYDRANT_STREAM_CHUNK_SPAN` set, hydrant also seals the stream into immutable files that a subscriber starting from scratch can read instead of replaying `/stream` from 0. chunks are compressed and cacheable, and serving one is just a file read instead of rendering every event again for each subscriber.

each chunk covers the ids `first` through `last` and holds every record event in that range. like a replay, it has no `identity` or `account` events. reading the chunks in order and then `/stream` from the id after the last one gets you exactly what `/stream?cursor=0` would. the file layout is under [chunk format](#chunk-format).

### GET /stream/manifest

list the chunks to read, in order.

```json
{
  "stream": "5b0e6c8f2a1d4e7b9c3f0a6d8e2b4c71",
  "generation": 41,
  "through": 917503,
  "chunks": [
    { "first": 786432, "last": 851967, "events": 47512, "bytes": 5104330, "sha256": "9f2c..." },
    { "first": 851968, "last": 917503, "events": 61664, "bytes": 11420553, "sha256": "41ab..." }
  ]
}
```

(the list is shortened here.) the chunks run from id 0 through `through` with no holes. while hydrant is rebuilding a chunk, the list and `through` stop just before it. read all the chunks, then connect to `/stream?cursor={through + 1}&stream={stream}`. if a chunk fails partway through, you don't have to start over. connect to `/stream` from the id after the last event you got. `generation` changes whenever the list does. returns `404` when chunks are off.

### GET /stream/chunk/{sha256}

get a chunk's bytes as stored, with content type `application/zstd`. its sha256 is both the name and the `ETag`. chunks never change, can be cached forever, and support `Range` requests. after a merge, the old chunks stay readable for 15 minutes for subscribers still working through an older list. a chunk hit by a redaction or erase returns `404` right away.

### how the list changes

a new chunk is one span long. whenever the newest 4 chunks are the same size, hydrant merges them into one, up to 16 spans. past that, the list grows by one chunk every 16 spans, which gives a subscriber several chunks to fetch in parallel. hydrant builds chunks on a low priority thread and seals only one span every 30 seconds while ingest is behind. building chunks for a whole stream (after turning chunks on, or after a wipe) mostly waits for ingest to catch up.

a redaction or erase hits every chunk that has events for that repository. those chunks drop out of the list until hydrant rebuilds them from the event rows without the removed bodies, matching what a replay sends. the stream has no redaction event, and a subscriber that already read an affected chunk keeps what it got, just like one connected to `/stream` at the time.

## chunk format

a chunk is a [STAR](https://dasl.ing/star.html) archive compressed with zstd. the file has several zstd frames, and some decoders stop after the first one unless you tell them to read across frames. the archive's header `meta` is a DRISL map with these keys:

| key | value |
| :--- | :--- |
| `$type` | `"systems.gaze.hydrant.streamChunk"` |
| `stream` | the stream id, as in `/stream/head` |
| `first` | the first id the chunk covers |
| `last` | the last id the chunk covers |

### events

each pair is one event, in id order between `first` and `last`. the key is the event id as 8 bytes big-endian, and the value is a DRISL map of the event's fields, with the record as hydrant stores it:

| key | DRISL | `/stream` field |
| :--- | :--- | :--- |
| `live` | bool | `record.live` |
| `did` | text | `record.did` |
| `rev` | text | `record.rev` |
| `collection` | text | `record.collection` |
| `rkey` | text | `record.rkey` |
| `action` | text | `record.action` |
| `record` | byte string, left out when there's no body | `record.record` |
| `cid` | tag 42 cid link, left out for deletes and when it matches the cidv1 (dag-cbor, sha2-256) of `record`'s raw bytes | `record.cid`, as a base32 string |

### as a /stream frame

a chunk event has the same fields as a `/stream` record frame. you only need this part to rebuild the exact json `/stream` sends.

the json frame decodes `record`'s dag-cbor bytes into json. maps keep their cbor key order, a tag 42 link becomes `{"$link": "<base32 cid>"}`, bytes become `{"$bytes": "<padded base64>"}`, and ints, strings, bools, nulls and arrays stay as they are. if a body can't be represented that way without losing information, hydrant serializes it the way jacquard's `RawData` does instead. that happens when a body has a float, an int past 64 bits, a `$link` or `$bytes` map key, keys out of dag-cbor order or repeated, or a `blob` map with any field besides `$type`, `ref`, `mimeType` and `size`. if a body isn't valid dag-cbor at all, `record` is left out of the json frame. the whole frame is

```
{"id":<id>,"type":"record","record":{"live":..,"did":..,"rev":..,"collection":..,"rkey":..,"action":..[,"record":<body>][,"cid":"<cid>"]}}
```

with no whitespace, byte for byte what `/stream` sends.