---
`/stream` is hydrant's ordered websocket event stream, and every subscriber gets a full copy of it. it's only in builds with the `indexer_stream` feature.
`/stream` isn't wire or protocol compatible with [`tap`](../concepts/vs-tap.md). tap hands events out through a sharded work queue, with acks the server tracks and redelivery. hydrant broadcasts every event to everyone and leaves the cursor to the client.
`/stream` isn't wire or protocol compatible with [`tap`](../concepts/vs-tap.md). tap sends events out through a sharded work queue, tracks acks on the server, and redelivers them. hydrant broadcasts every event to everyone and leaves the cursor to the client.
## GET /stream
subscribe to the ordered event stream.
| `rev` | string | repository revision, a tid. |
| `collection` | string | collection nsid. |
| `rkey` | string | record key. |
| `action` | string | `create`, `update`, or `delete`. |
| `record` | object | the decoded dag-cbor record, its keys in the record's cbor order (shorter first), not alphabetical. left out when there's no stored body. |
| `record` | object | the decoded dag-cbor record, with keys in the record's cbor order (shorter first) instead of alphabetical. left out when there's no stored body. |
| `cid` | string | cid of `record`. left out when there's no stored body. |
`record` and `cid` are left out for deletes, and when hydrant doesn't store record bodies, like with `HYDRANT_ONLY_INDEX_LINKS=true`.
`record` and `cid` are left out for deletes and when hydrant doesn't store record bodies, such as with `HYDRANT_ONLY_INDEX_LINKS=true`.
once a redaction finishes, the body isn't served anymore, except in events that were already queued for a reader by then.
once a redaction finishes, hydrant stops serving the body, except in events already queued for a reader.
### identity
identity events are live only, and a cursor replay doesn't get them.
- hydrant sends an empty ping every 30 seconds and answers client pings with pongs.
- clients can't send text or binary frames. hydrant closes the connection if one does.
- when `/stream` is in `HYDRANT_THROTTLED_STREAMS` (it isn't by default), a replay from a cursor runs at most `HYDRANT_STREAM_REPLAY_SPEED` times the bandwidth the live stream would take (10 by default), or `HYDRANT_STREAM_REPLAY_FLOOR_BYTES` a second (about 80 Mbit/s by default) if that's more. the live tail never waits for it.
- a subscriber that falls behind the live broadcast catches up from the database at its own pace. it never misses a `record` event, but it does miss the `identity` and `account` events from while it was behind.
- a subscriber that falls behind the live broadcast catches up from the database at its own pace. it never misses a `record` event, but it misses `identity` and `account` events from while it was behind.
- if hydrant can't read the next events from the database, it closes the connection instead of skipping them. reconnect with the last `id` you got as the cursor.
- when hydrant stops, it closes the connection with close code `1012` (service restart). reconnect the same way. under systemd socket activation, the reconnect waits for the next hydrant instead of being refused.
- if a send is stuck for `HYDRANT_STREAM_SEND_TIMEOUT` (30 seconds by default), hydrant sends this error frame and closes the connection:
to know when a `cursor=0` replay has caught up, get the head before subscribing and wait for a frame with an `id` at or past it. keep an idle timeout as a fallback too, because if the head event itself can't be rendered, it's skipped and never arrives.
## chunks
with `HYDRANT_STREAM_CHUNK_SPAN` set, hydrant also seals the stream into immutable files that a subscriber starting from nothing can read instead of replaying `/stream` from 0. they're compressed, a cache can hold them, and serving one is just a file read for hydrant, where a replay renders every event again for every subscriber.
with `HYDRANT_STREAM_CHUNK_SPAN` set, hydrant also seals the stream into immutable files that a subscriber starting from scratch can read instead of replaying `/stream` from 0. chunks are compressed and cacheable, and serving one is just a file read instead of rendering every event again for each subscriber.
each chunk covers the ids `first` through `last` and holds every record event in that range. like a replay, it has no `identity` or `account` events. reading the chunks in order and then `/stream` from the id after the last one gets you exactly what `/stream?cursor=0` would. the file layout is under [chunk format](#chunk-format).
### GET /stream/manifest
]
}
```
(the list is shortened here.) the chunks run from id 0 through `through` with no holes. while a chunk is being rebuilt, the list stops just before it, and `through` with it. read them all, then go on with `/stream?cursor={through + 1}&stream={stream}`. if a chunk fails partway through, you don't have to start over. continue with `/stream` from the id after the last event you got. `generation` changes whenever the list does. returns `404` when chunks are off.
(the list is shortened here.) the chunks run from id 0 through `through` with no holes. while hydrant is rebuilding a chunk, the list and `through` stop just before it. read all the chunks, then connect to `/stream?cursor={through + 1}&stream={stream}`. if a chunk fails partway through, you don't have to start over. connect to `/stream` from the id after the last event you got. `generation` changes whenever the list does. returns `404` when chunks are off.
### GET /stream/chunk/{sha256}
get a chunk's bytes as stored, as `application/zstd`. its sha256 is both the name and the `ETag`. chunks never change and can be cached forever, and `Range` requests work too. after a merge, the chunks it replaced stay readable for 15 minutes for subscribers still partway through an older list. a chunk hit by a redaction or erase is `404` right away.
get a chunk's bytes as stored, with content type `application/zstd`. its sha256 is both the name and the `ETag`. chunks never change, can be cached forever, and support `Range` requests. after a merge, the old chunks stay readable for 15 minutes for subscribers still working through an older list. a chunk hit by a redaction or erase returns `404` right away.
### how the list changes
a new chunk is one span long. whenever the newest 4 chunks are the same size, hydrant merges them into one, up to 16 spans. past that the list grows by one chunk every 16 spans, and stopping there leaves a subscriber several chunks to fetch at once. hydrant builds chunks on a low priority thread and seals only one span every 30 seconds while ingest is behind. chunking a whole stream at once (after turning chunks on, or after a wipe) mostly waits for ingest to catch up.
a new chunk is one span long. whenever the newest 4 chunks are the same size, hydrant merges them into one, up to 16 spans. past that, the list grows by one chunk every 16 spans, which gives a subscriber several chunks to fetch in parallel. hydrant builds chunks on a low priority thread and seals only one span every 30 seconds while ingest is behind. building chunks for a whole stream (after turning chunks on, or after a wipe) mostly waits for ingest to catch up.
a redaction or erase hits every chunk with that repository's events in it. those chunks drop out of the list until hydrant rebuilds them from the event rows without the bodies, the same as a replay would show them. there's no redaction event on the stream, and a subscriber that already read one of those chunks keeps what it got, just like one that was connected to `/stream` at the time.
a redaction or erase hits every chunk that has events for that repository. those chunks drop out of the list until hydrant rebuilds them from the event rows without the removed bodies, matching what a replay sends. the stream has no redaction event, and a subscriber that already read an affected chunk keeps what it got, just like one connected to `/stream` at the time.
## chunk format
a chunk is a [STAR](https://dasl.ing/star.html) archive compressed with zstd. the file has several zstd frames, and some decoders stop after the first one unless you tell them to read across frames. the archive's header `meta` is a DRISL map with these keys:
| `last` | the last id the chunk covers |
### events
each pair is one event, in id order between `first` and `last`. the key is the event id as 8 bytes big-endian, and the value is a DRISL map with the same fields as the event's `/stream` frame, with the record as hydrant stores it:
each pair is one event, in id order between `first` and `last`. the key is the event id as 8 bytes big-endian, and the value is a DRISL map of the event's fields, with the record as hydrant stores it:
| key | DRISL | json frame |
| key | DRISL | `/stream` field |
| :--- | :--- | :--- |
| `live` | bool | `record.live` |
| `did` | text | `record.did` |
| `rev` | text | `record.rev` |
| `collection` | text | `record.collection` |
| `rkey` | text | `record.rkey` |
| `action` | text | `record.action` |
| `record` | byte string, left out when there's no body | `record.record` |
| `cid` | tag 42 cid link, left out for a delete and when it's just the cidv1 (dag-cbor, sha2-256) of `record`'s bytes as they are | `record.cid`, as a base32 string |
| `cid` | tag 42 cid link, left out for deletes and when it matches the cidv1 (dag-cbor, sha2-256) of `record`'s raw bytes | `record.cid`, as a base32 string |
`record` holds the record's dag-cbor bytes, and the json frame turns them into json. maps keep their cbor key order, a tag 42 link becomes `{"$link": "<base32 cid>"}`, bytes become `{"$bytes": "<padded base64>"}`, and ints, strings, bools, nulls and arrays stay as they are. a body that can't be written that way faithfully is written the way jacquard's `RawData` serializes it instead. that's a body with a float, an int past 64 bits, a `$link` or `$bytes` map key, keys out of dag-cbor order or repeated, or a `blob` map with any field besides `$type`, `ref`, `mimeType` and `size`. a body that isn't dag-cbor at all is left out of the json frame. the whole frame is
### as a /stream frame
a chunk event has the same fields as a `/stream` record frame. you only need this part to rebuild the exact json `/stream` sends.
the json frame decodes `record`'s dag-cbor bytes into json. maps keep their cbor key order, a tag 42 link becomes `{"$link": "<base32 cid>"}`, bytes become `{"$bytes": "<padded base64>"}`, and ints, strings, bools, nulls and arrays stay as they are. if a body can't be represented that way without losing information, hydrant serializes it the way jacquard's `RawData` does instead. that happens when a body has a float, an int past 64 bits, a `$link` or `$bytes` map key, keys out of dag-cbor order or repeated, or a `blob` map with any field besides `$type`, `ref`, `mimeType` and `size`. if a body isn't valid dag-cbor at all, `record` is left out of the json frame. the whole frame is
```
{"id":<id>,"type":"record","record":{"live":..,"did":..,"rev":..,"collection":..,"rkey":..,"action":..[,"record":<body>][,"cid":"<cid>"]}}
```