[jetstream] send the live json as built, without a redaction read per reader

jetstream already builds a live commit's json once at the commit and shares it, but every reader checked the record's redaction marker before sending it, a point read per reader per live event, and rendered it from the db when the marker was there. that check is gone, so the json goes out as built. a redaction that completes after the commit doesn't reach events already queued then, the same window /stream has, and docs/api/jetstream.md says so. the test that pinned the old read now pins this: a commit queued before the redaction still has its record. the redaction keys JetstreamLive and JetstreamEphemeral carried for it are gone too.

[stream] build each live /stream frame once, for every reader

each /stream reader rendered every live event itself: read the record head back, hashed it against the event's cid, decoded it into a RawData tree, turned that into JSON, queued the decoded event, and serialized it into its own frame on the socket. four readers did all of that four times and each held its own ~700 B decoded event per queued item. now /stream readers get frames. the first reader to render a live event writes its frame, the record transcoded in place, and swaps it in for the event on the broadcast's shared LiveRecordEvent (a mutex not held while it encodes). every other reader clones those bytes into its queue and the websocket sends them as they are. with no /stream reader nothing is built. replays write frames the same way straight from the db, and frame_bytes is now the real frame length for the replay cap. the frame is built from the body the commit had, without reading it back, so a redaction that completes after the broadcast doesn't stop events already queued then. docs/api/stream.md says so. after the swap only the frame is kept, so an in-process subscriber (Hydrant::subscribe, ffi) that meets a swapped event reads it back from the db the way a replay does. Hydrant::subscribe keeps its Event type, /stream uses a pub(crate) subscribe_frames next to it. memory, per live event held in the broadcast slots or a stalled reader's backlog, measured over 30,000 real firehose messages (29,628 record events): the inline body was a slice of its commit's car, so the event held the car. before after the swap struct + strings ~267 B 256 B commit car 5,215 B - frame - 593 B total 5,291 B 849 B (512-event window) 5,293 B 850 B (4096-event window) car p50 5,037 B, p99 11,421 B. the frame is shrunk to its length, it was written into a guess of 256 B + 2x the body. a reader's queue now holds a 32 B handle per frame instead of a ~700 B decoded event. WsAction::Skip is gone, nothing sends one now that frames can't fail to serialize on the socket.

[types] stream record json straight from the cbor

every record body that goes out as JSON (/stream live and replay, jetstream live and replay, the relay's jetstream) went through a RawData tree first, then to_raw_value. write_record_json now reads the DAG-CBOR and writes the JSON as it goes. the prototype of it took 0.47 us a record against 1.10 for the tree on the cost test's records, this one is timed with the next commit. keys come out in the record's CBOR order instead of sorted, which the docs now say. anything RawData would turn into something else falls back to it, truncating whatever the streamed attempt wrote: keys out of DAG-CBOR order or repeated (RawData keeps the last), floats, ints past 64 bits, $link and $bytes keys, and blobs with other than their four fields, which RawData reshapes. so apart from key order the JSON is what it was. over the 288,266 blocks of two real repo cars every one parses to the same value as before, 60,711 byte for byte. the replay fixture in stream/indexer.rs changes only in its record's key order.

[stream] cap how fast network readers replay from a cursor

a replay reads and encodes rows as fast as its reader takes them, and since the full-channel sleep went away that's a whole core of db reads and encoding for one client asking /subscribe for cursor 0. a websocket reader's cursor replay is now held to HYDRANT_STREAM_REPLAY_SPEED (10) times what keeping up with the live stream would cost it, or HYDRANT_STREAM_REPLAY_FLOOR_BYTES (10 MB/s, about 80 Mbit/s) when that's more. the cost is bytes sent, and every row a filter drops costs 32 bytes, so a filtered jetstream replay that scans a million rows to send ten still pays for the scan. 32 comes from measuring jetstream replay: about 250 ns for a dropped row against 8 us for a sent 660 byte one, rounded up for disk reads. the live rate is the stream's position counter, sampled by the reader about once a second, so the commit path pays nothing. HYDRANT_THROTTLED_STREAMS picks which streams are capped, jetstream and relay by default, empty for none. an unknown name or a zero speed or floor is a startup error. in-process subscribers (Hydrant::subscribe and friends) are never capped. only the replay the cursor asked for is capped. markers and catch ups after falling behind the tail are as big as what got written meanwhile, so they stay live work, and the live tail never waits. the budget is per reader, so many readers replaying at once are bounded by how many connect, not by this. HYDRANT_STREAM_REPLAY_CHUNK_PAUSE is gone, and so is the stream_replay_chunk_pause config field (pre-1.0). the env var is still read so env files keep working, but setting it logs that it does nothing.

[indexer] comments that still describe the lock or old names

rewrite_prefix, the mini-doc reconcile and the txn tests still explained themselves through the repo lock and BackfillFinished, both gone. also drops docspeak and a doc that restated its fn name.

[db] drop the test that greps for the lock's old names

the test searched the source tree for the names of the removed repo lock protocol. the compiler already stops anyone calling them, a comeback under another name would pass it, and its comment only told the history. also fixes a test comment that still explained a pds count read through the lock.

[indexer] drop the repo results identity and account never returned

handle_commit never returns Deleted, and nothing read what handle_identity and handle_account returned besides the error, so the arm and the variant go and both return unit.

[db] derive counting ephemeral records from the event origin

record_scope and record_txn took count_ephemeral_records next to the origin, and every caller passed true for backfills and false for live writes, so it comes from the origin now and record_scope's one remaining job moves into its two callers.

[indexer] refuse in place writes while a stopped hydrant's shards still run

with no shard sender the repo writers run in place, meant for before the shards start. shutdown takes the sender away too while a shard can still be in its last step, so a late writer like the retry worker ran in place next to it. shard threads now count themselves, and once the stop is set an in place write fails until they're gone.

[indexer] don't requeue mid apply when the busy check fails

a chain break or a sync during a backfill's steps marks the repo to backfill again once they finish, because swapping its queue key now would throw the half written repo away. requeue_after_steps read a failed queue lookup as not busy, so on a db read error it swapped the key anyway. the error goes to the caller now: a chain break fails the commit, which then gets buffered, and a sync logs it.

[indexer] buffer a new repo's first commit behind the backfill it queued

since the relay's row change lands in the indexer txn, a new repo's row and its first commit land in one txn, and making the row queues its backfill there. the commit then decided whether to buffer from the metadata in the db, where that backfill isn't yet, so it was applied live while every later commit was buffered. a car older than that commit then deleted its records and left them gone with a synced row. the commit now counts as backfilling once its txn queued the backfill.

[indexer] keep a repo busy after its erase or wipe step fails

a failed erase or wipe step dropped the job and the repo's busy entry but kept the marker for the next start. until that restart live commits and a track landed on the half erased repo, and a track left it tracked, which made the next start drop the erase marker as moved on, so the erase never finished. the repo now stays busy after a failed step, until a restart or a retried erase runs the job again.

[indexer] a backfill step leaves a running erase or wipe alone

a backfill's first step marks its repo busy with an apply and writes its job marker, and it only checked for an apply under its own queue key first. an erase or a wipe running for the repo had its busy entry and marker replaced. the crawler can queue such a backfill mid erase, and from there live commits applied again, records the backfill wrote behind the erase cursor outlived the erase, and a restart no longer resumed the erase because its marker was gone. the step now answers superseded while the repo's erase or wipe runs.

[db] keep erasing a kind of entry after a step the byte cap cut short

an erase works through each kind of entry (heads, history, event bodies, redactions, buffered commits) a step at a time, and erase_chunk stops a step at the entry limit or at 32 MiB. but erase_step only stayed on a kind when the step took the whole entry limit, so a step cut short by bytes moved on to the next kind, and everything after it under that prefix was never erased. the erase still reported success, and the leftover heads stayed servable by getRecord and listRecords. this came in with erasing repos in steps on their indexer shard. erase_chunk now says whether it stopped at a cap with more to go, and the erase only moves on once a kind is empty. the step's txn only exists for the last phase that uses it, and a kind that fits one step no longer spends an empty step after it, so the erase test counts one step per kind.

[tests] the hanging did:web host test asserts order, not wall time

the test bounded each other did's latency at 100 ms of wall time, which a loaded box broke (195.9 ms in a full run) without anything being wrong. it now asserts the order instead: every other did's message, one at a time and then the burst, is applied while the slow did's lookup is still in flight and the cursor still waits on it. the lookup deadline is 10 s, only as a backstop. a shard that sleeps past the deadline after parking, like one that blocked on the lookup, fails with "...k0 was applied only after the slow lookup ended".

[ingest] one background refresh per did until its answer is in

a background refresh leaves the pool's in-flight set just before its answer is queued for the shard, so a message the shard runs in between finds no job for the did and starts a second one, a second plc request for nothing. a_key_past_its_ttl_verifies_and_refreshes_once_in_the_ background caught it once under a loaded full run (2 plc hits). the shard skips a refresh while it still has an origin for the did, which it holds from start to answer.

[ingest] relay comments say what holds, not how it got there

finding numbers, "now", "used to" and "like it always did" were history in comments and test docs. they say what the code or the test pins instead. the log line for a failed lookup nobody waits on stops calling every such lookup a background refresh, since a held did's lookup lands there too once its queue was dropped.

[ingest] relay shard: drop the dead info branch, one did helper

the reader never forwards an info frame (firehose.rs takes a ticket only for commit, identity, account and sync), so the branch that handled one in the relay shard never ran. without it, the three `.did().expect(..)` spellings are one WorkerMessage::did(). the CountDeltas resets in step and apply_answer went too: commit takes them and discard resets them already.

[ingest] relay sink forgets a failed commit's events and rows the indexer never gets

a relay commit that failed left the frame's events in the sink. the next frame's hand_off and hand_row took the first own-frame entry, which was the failed one, so its ticket moved to the wrong event and its row change landed on another did's row through that event. the sink is discarded with the failed commit now. when the indexer shard is gone, blocking_send fails and the row change never reaches applied(), so the relay kept reading it over the durable row from the overlay. flush now lets go of it and logs why. each queued message keeps whether it's the frame's own event in a named field, and the identity sinks call that parameter own_frame instead of advances_cursor, which it hasn't meant since tickets.

[ingest] relay shard checks held queues on every wake, not on a timer

the shard slept on a tokio timer until the oldest held queue was due. hydrant shuts its runtime down before Hydrant::shutdown, and the time driver's shutdown wakes every timer, so a shard holding a did polled that sleep on a dead driver and panicked. and with a backlog the biased select always picked a message over the timer, so the sweep only ran when idle. the shard checks HOLD_MAX at the top of every loop instead. nothing needs the timer: the held did's own lookup answers within its 12 s deadline, which wakes the shard anyway.

[crawler] the firehose dispatch time follows the cursor watermark

note_dispatched ran at the top of the relay's handle_commit: before validation, with the event time as the pds stamped it, before the indexer applied anything, and only for commits that got that far. so a commit stamped in the future ended the crawler's waits on that relay for good, a held commit counted as dispatched, and with a sparse filter almost nothing moved the mark. tickets now carry their frame's event time, and the mark moves with the cursor's watermark: when every frame read up to one is done, its time is noted, clamped to now. any frame moves it, skipped ones included, a held one stops it, and nothing counts before it's applied. firehose_dispatched also answers None while the firehose is paused, since nothing moves then. SubscribeReposMessage::position() hands out a frame's seq and time for the reader, the relay's identity announce and the pipeline tests, instead of a match each.

[ingest] lookup pool: plc's own permits and breaker, host permit first, bounded hosts

plc went through the per-host semaphore before its own one, so every did:plc lookup in the process ran two at a time instead of 16. plc now has a host entry sized PLC_PERMITS under a key of its own, so a did:web host named plc (did:web:plc:x) can't trip or share its breaker. a job took its global permit before waiting on the host, so one did:web host that hangs, with its 64 parked dids, held all 32 global permits and every other lookup timed out behind it. the host permit comes first now (for the pds probe too), and the breaker is checked again once the permit is held. the host map kept every host that ever failed, so a stream of new did:web hosts grew it without bound, with a full scan per new host at the cap. at the cap it now forgets hosts nobody is asking whose breaker is shut, then open ones too. backoff: an open breaker no longer backs the did off, it's about the host. a failed background refresh doesn't either, since nobody waits on it and the row keeps its key, but it holds off the next background refresh for that did for 60 s so a did whose refreshes keep failing can't start one per message. a did plc says isn't there starts at 5 s and doubles to 10 min, since a mirror can lag a few seconds behind a new account.

[backfill] comments that still describe one batch

the budget, the fetch and a test name still talked about the repo's write batch committing, which steps replaced. the outcome docs claimed records were committed for a stop that can come before any step. also merges two impl blocks and drops time words from two test comments.

[backfill] match a task's fetch outcome directly

did_task mapped the fetch outcome into an option of a handoff and then matched that again, which said nothing the fetched value didn't. it matches Fetched directly now.

[backfill] stop the retry pass once hydrant is stopping

the retry worker only looked at stop between passes. once shutdown takes the shards away, a pass halfway through its candidates ran each requeue in place, next to a shard that may still be finishing a step. the pass is its own function now, queue_due_retries, and checks stop per candidate.

[backfill] drop an orphaned queue entry with remove_pending

the deleted branch checked exclusion and then did the same two writes either way, which is exactly what remove_pending does now.

[backfill] move the lifecycle gauge when a task drops its queue entry

remove_pending took the key off without touching the gauge, so a repo excluded between the task's start and its first step, or parked for an unsafe pds, stayed counted as pending for good. it now moves the gauge to synced, which only happens while the key is still the repo's current one, so an orphan or a key already taken off leaves it alone.

[backfill] let a sparse backfill's pds slot go before its steps

a full backfill drops its pds admission once the car is read, but a sparse one kept it through every step on the shard, which can wait a while behind live traffic, so other repos on the same pds got turned away as busy meanwhile. it now drops it before the steps. the test pds answers getLatestCommit so a sparse backfill can run end to end.

[backfill] only remove our own leftover spills at start

the start removed every .car file in the spill dir, so with HYDRANT_BACKFILL_SPILL_DIR pointed at a shared dir it deleted someone else's car exports. it now only removes files named the way a spill names them, {pid}-{n}.car.

[backfill] check spill headroom across every spill

each spill looked at the free space only after its own 64 MiB, so four at once could each write 64 MiB past a passing check and take the whole 256 MiB headroom, leaving the database to hit a full disk. the bytes are now counted across every spill, so all of them together write at most 64 MiB between checks. the BACKFILL_SPILLS docs say so, and also that a slot is held until the last step and that a backfill waiting for one keeps its fetch timeout running.