a chunk covers the ids `first` through `last`. like a replay, it holds only record events, never `identity` or `account` ones, and every one of its range. reading the chunks in order, then `/stream` from the id after the last one, delivers exactly what `/stream?cursor=0` does.
### chunk format
a chunk is a [STAR](https://dasl.ing/star.html)(DASL's streaming archive, from its [pull 54](https://tangled.org/dasl.ing/dasl.ing/pulls/54)), compressed with zstd. decompressed, it is
a chunk is a [STAR](https://dasl.ing/star.html) archive, compressed with zstd. its header `meta`is a DRISL map:
```
2a 6c 01 00 | varint len | meta | ( varint 8 | id | varint len | event )*
| `stream` | the stream id, as in `/stream/head` |
| `first` | the first id the chunk covers |
| `last` | the last id the chunk covers |
- the four magic bytes, then the header: an unsigned varint (LEB128, no superfluous trailing `00`byte, at most 9 bytes) holding the length of `meta`, a DRISL map:
each pair is one event, in id order, each id between `first` and `last`and greater than the one before. the key is 8 bytes, the id big-endian. the value is a DRISL map of the event's `/stream` frame fields, with the record as hydrant stores it:
| `stream` | the stream id, as in `/stream/head` |
| `first` | the first id the chunk covers |
| `last` | the last id the chunk covers |
| key | DRISL | json frame |
| :--- | :--- | :--- |
| `live` | bool | `record.live` |
| `did` | text | `record.did` |
| `rev` | text | `record.rev`|
| `collection` | text | `record.collection`|
| `rkey` | text | `record.rkey` |
| `action` | text | `record.action` |
| `record` | byte string, absent when there's no body | `record.record` |
| `cid` | tag 42 cid link, absent for a delete and when it's the cidv1 (dag-cbor, sha2-256) of `record`'s bytes | `record.cid`, as a base32 string |
- then one pair per event, in id order, each id between `first`and `last` and greater than the one before. the key is 8 bytes, the id big-endian. the value is a DRISL map of the event's `/stream` frame fields, with the record as hydrant stores it:
a missing `cid` next to a `record`is the cidv1 of the bytes as they are, not of a re-encode of them.
| key | DRISL | json frame |
| :--- | :--- | :--- |
| `live` | bool | `record.live` |
| `did` | text | `record.did` |
| `rev` | text | `record.rev` |
| `collection` | text | `record.collection` |
| `rkey` | text | `record.rkey` |
| `action` | text | `record.action` |
| `record` | byte string, absent when there's no body | `record.record` |
| `cid` | tag 42 cid link, absent for a delete and when it's the cidv1 (dag-cbor, sha2-256) of `record`'s bytes | `record.cid`, as a base32 string |
the compressed file is several zstd frames back to back, which decompress as one stream: one for the magic and header, then any number holding the pairs, then one for the end byte. that is how a merge works: a new header frame, then each chunk's frames between its header and its end, as they are, then a new end frame.
a missing `cid` next to a `record` is the cidv1 of the bytes as they are, not of a re-encode of them. the archive ends after the last pair, with nothing after it. STAR can't tell an archive cut short at a pair boundary from a whole one, so check a chunk against its sha256 before trusting it.
hydrant versions before the end byte wrote chunks without the `01` and `00` bytes. a newer hydrant rebuilds those from the event rows when it opens, and serves none of them meanwhile.
the compressed file is several zstd frames back to back, which decompress as one stream: one for the magic and header, then one or more holding the pairs. that is how a merge works: a new header frame, then each chunk's frames after its own header, as they are.
`record` holds the record's dag-cbor bytes. the json frame has them as json: maps keep their cbor key order, a tag 42 link becomes `{"$link": "<base32 cid>"}`, bytes become `{"$bytes": "<padded base64>"}`, ints, strings, bools, nulls and arrays map as themselves. a body this can't express faithfully (a float, an int past 64 bits, a map key `$link` or `$bytes`, keys out of dag-cbor order or repeated, a `blob` map with more or other fields than `$type`, `ref`, `mimeType` and `size`) is written the way jacquard's `RawData` serializes it, and one that isn't dag-cbor at all leaves `record` out of the json frame. the json frame is then