[docs-site] one list for prose pages and configuration, steady home banner

[docs-site] full text fallback in the address bar, shorter crumbs on phones

[nix] build the docs site with bun, drop verbiage

[docs-site] redirect the old .md urls, add a 404 page

[docs-site] configuration as list + inspector, settings in search

[docs-site] scaffold the astro site: window, address bar, home grid, doc pages

[docs] fence the example startup error so its inner backticks render

[docs] drop the link to tuning.md, which was never written

[ingest] say mst inversion also runs under signature verification

[docs] show VERIFY_MST's effective default, merge the do-nothing settings into compatibility settings

[firehose] test that requestCrawl restarts a source only after it gave up

[firehose] wait out the startup jitter when requestCrawl restarts a known source

[api] resolve a minidoc for dids hydrant doesn't index

resolveMiniDoc refused any did not yet synced (or never indexed) with a 400, but slingshot's contract is a minidoc for any did, and bobbin and pocket lean on that. on RepoNotFound and NotSynced it now falls back to the cached identity resolver, verifies the handle the same way as the indexed path, and writes nothing to the repo db.

[net] use system trust with reqwest and jacquard 0.13

[tests] fix http-nu mock response metadata

[deps] bump rustls-webpki to 0.103.16 for the emSign blocklist

[backfill] say the first retry waits two minutes, which it always has

next_backoff's comment promised 1m first, but the caller counts the failure before asking, so the first retry is 2m and backoff_grows_across_requeues_until_a_fetch_succeeds pins that.

[indexer] drain a buffered commit the relay already put on the row

the relay puts every commit it forwards on the repo row, including one the indexer only buffers because the repo is backfilling. the backfill's persist keeps that newer root, since a root never goes back, while its records and backfilled_rev stay at the car. the drain then asked whether the buffered commit follows the row's root, which is the commit itself, so it failed with "doesn't follow rev" on the same rev, dropped the buffer and left the records missing until the refetch two minutes later. on a fresh account that's the commit right after the first one, so the records written while hydrant backfills it showed up minutes late. only a commit past the root, which a car older than it took back, is checked against it now. one at or below it came in during the fetch and is applied, and apply_commit already leaves a root that's ahead alone. a chain break the relay let through queues another backfill on its own.

note relay builds seed bsky.network, and uncapped jetstream replay

A relay build following RELAY_HOSTS also seeds a firehose per PDS from SEED_HOSTS unless it is set empty. A single consumer that must catch up fast can leave jetstream out of THROTTLED_STREAMS.

take the signing key from the #atproto verification method

jacquard-common's atproto_public_key returns the first Multikey in verificationMethod whatever its id. The DID spec (https://atproto.com/specs/did) says the signing key is the entry whose id ends in #atproto, typed Multikey, controlled by the DID, and that any others are ignored. did:web:offbeatwit.ch lists #atproto_old first, so every commit of theirs failed signature verification here while the bsky relays took them.

[net] add HYDRANT_ALLOW_PRIVATE_NETWORKS for local dev stacks

a stack that serves every name from loopback can't be reached through the public egress check, which drops any dns answer that isn't a public address. the switch turns that check off for the whole process and defaults to off.

[beads] update beads

[relay] answer a subscribeRepos cursor past the last seq with FutureCursor

the event stream spec says a cursor higher than the current seq gets an error and a close. hydrant replayed from it, found nothing, and skipped every live event below it, so a consumer holding a cursor from another relay or from before a reset waited with no error until the seqs caught up, maybe forever. it gets a FutureCursor error frame now and the stream ends. indigo's relay goes live instead, which hides the gap, while the spec's way lets the consumer decide, and indigo's and hydrant's own consumers already handle FutureCursor from a pds. RelayStreamError has a FutureCursor variant for embedders.

[relay] a dropped queue's newest #account still lands

a relay shard drops a did's held messages at a cap or its deadline and marks them done for the cursor. for commits that's survivable, the next one shows the gap. an #account among them was lost for good though: a takedown is usually the last thing an account sends, so nothing came after it, and relay builds have no crawler to read the status back from listRepos. hydrant kept serving and forwarding the account. the newest #account of a dropped queue now runs anyway, as if the did's lookup had failed, the way held messages replay after a failed lookup. one from a pds still has to pass the host check against the stored pds. a new account's queue has no row to land on, so its messages are dropped as before.

[indexer] a buffered commit that doesn't follow the car fails the drain

the drain applied what was buffered during a fetch without asking whether it follows the car. each commit was checked when it came in, against the root back then, but the car can be older than that root, and the firehose can skip a commit between the two. the commit that triggered the backfill is buffered too, so after a stale car it went straight onto the old records. a buffered commit whose since or prevData doesn't match the root it would apply onto now fails the drain, which costs one refetch the way a buffered commit that can't apply does. a missing prevData isn't held against it here, the check it went through on arrival saw that.

[backfill] a car behind the stored root takes the root back too

a pds is the authority on its repo, so a getRepo answered with a car older than what hydrant had (a replica behind, a restore) replaces the records, as it did before. the root kept the newer rev though, since rebased never moves a root back, so the next commit chained onto that root and landed on the car's records, and the commits in between stayed missing for good. when the car's rev is below the root the backfill started from, the root now goes back with the records. the next commit doesn't follow it, the repo backfills again, and by then the pds usually serves its current repo. rebased keeps its rule, the relay overlay needs it.

[backlinks] find links in string arrays and under any key

jacquard only types a string that's an object's value, guessing from the key, so an at-uri in an array (a threadgate's hiddenReplies, a postgate's detachedEmbeddingUris) stayed a plain string and extract_links skipped it, and a did under `handle` came out as an at-identifier and was skipped too. the four links tests with a string or an array at the root failed on main for the same reason. the string decides now: one starting at:// that parses as an at-uri, or did: that parses as a did, is a link wherever it sits. records indexed before this keep their missing links until they're written again.

[backlinks] don't list an inactive account's records as links

the same account/en.mdx rule as the record endpoints: an inactive account's content shouldn't be redistributed. getBacklinks answers with the uri and cid of each record that links to the subject, which is that record's content. before the stack a deleted account's backlinks went in the same txn as its heads, so only takendown, suspended and deactivated accounts' links showed. since the wipe runs in steps, a deleted account's links also stay listed until its last step. getBacklinks now leaves out links whose record belongs to a repo in one of those four statuses, reading each linking repo's row once per page. a page can come back shorter than its limit, and the cursor still points past what was scanned. the status check moves from the xrpc module to crate::api so both can use it. getBacklinksCount still counts them: it returns a number and no record, and per-link repo reads would turn its key count into a scan. the test indexes one like by a repo in each of the four statuses, with no wipe run, and checks that getBacklinks lists it only for the active one.

[api] don't serve an inactive account's records

account/en.mdx: "If an account is not active, the service should not redistribute content for that account (repositories, individual records, blobs, etc)." getRepo and getLatestCommit already refused takendown, suspended, deactivated and deleted repos, but getRecord, listRecords, the two getRecordByUri spellings, countRecords and both describeRepos read the records without looking at the repo row. before the stack that only leaked the inactive accounts whose records stay stored, but since a deleted account's heads move to history in steps ("[backfill] erase and wipe repos in steps on their indexer shard, with job markers"), a deleted account's records also stayed readable until the last wipe step, after hydrant had already announced the deletion. each of those handlers now reads the repo row first and refuses those four statuses, like getRepo. the check sits in the xrpc handlers, not in RepoHandle, so embedders keep their own policy. the errors stay inside each lexicon: getRecord and the getRecordByUri pair answer 404 RecordNotFound, the only error their lexicon has and what the reference pds answers for a takendown record, with the reason in the message. listRecords, countRecords and the describeRepos define no errors, so they answer 400 InvalidRequest with the reason, like a pds does for a repo it won't serve. the test seeds one record under a repo row in each of the four statuses, with no wipe run, and checks that every gated endpoint refuses it, and that an active repo's record is still served.

[ingest] a broken identity takes the stored key off the row

the sync spec says to ignore #commit and #sync for an account without a working identity: no signing key, no pds service, or a tombstoned did. since the key lives on the row, a background refresh that found the doc broken was only logged in apply_answer, so the stored key kept verifying and forwarding commits, across restarts, with a lookup each minute that changed nothing. a refresh answer that says the identity itself is broken now takes the signing key off the row. the did's next commit or sync then takes the key path, waits for a lookup, and is dropped while that fails, which backs the did off for 10 min. broken means a fetched doc without a pds (MissingPds), without a key (NoSigningKey), or a tombstone. the resolver couldn't tell a 410 from a 404, so DocumentNotFound now keeps which it was and a 410 is ResolveFailure::Tombstoned (backed off like MissingPds). a plain 404 keeps the key and only backs off, since a plc mirror can be behind, and plc.wtf is the filtered-mode default.

[ingest] an #identity lets a backed off did ask again, once a minute

the sync spec says downstream services should update cached identity on #identity, and indigo purges its cache, error entries too, on every one. with the lookup pool's negative cache, an #identity for a backed off did changed nothing: a pds's was forwarded without a re-resolve, and a new account's was dropped with its frame marked done. that #identity is what an account sends once it fixed its doc, so it sat out a backoff of up to 10 min or 1 h. an #identity now clears the did's backoff (failed_until and the background refresh hold) so its next lookup goes out. the backoff is an abuse shield, so: - one from a relay hydrant follows clears before the message runs. one from a pds clears only in handle_identity, once the host check took it, so a pds can't make hydrant look up a did it doesn't host - at most one clear per did per 60 s (IDENTITY_CLEAR_COOLDOWN, the doubling backoff's first step and the longest first step), stamped in the negative cache, which is bounded at 65536 dids, and only when there was something to clear. a did spammed with #identity costs at most one lookup a minute - the doubling step stays, so a did whose lookups keep failing still backs off further - the sig failure cooldown and the wrong host cache aren't touched

[resolver] a rate limited handle lookup is no verdict, and only answers are cached

the handle spec says to mark a handle invalid once it's confirmed to no longer resolve. handle_verdict turned a 429 from a handle source into false and verify_handle kept every false for 60 s, so a rate limit read as handle.invalid (describeRepo handleIsCorrect false, the mini doc's handle.invalid) for a minute without anything being confirmed. the same went for an outage: every source unreachable was cached as a negative verdict. a rate limit is an error from verify_handle again, as before the stack, and isn't kept. a lookup where no source answered (every one unreachable or timed out) still answers false, so callers keep showing handle.invalid for it like they did before the stack, but isn't kept, so the next ask goes out again. only a source that answered (another did, or no mapping) gives a negative verdict that's kept for 60 s. the source loop says which happened: HandleResolutionExhausted only when some source answered, Transport when none could.

[indexer] work on a shard hands the jobs it started back, no ResumeJob message

a crawler page's deleted listing starts a wipe in work it runs on the repo's owner shard, but that work couldn't queue the wipe's steps, so the crawler sent a ResumeJob message after the page's replies came back. until it arrived the marker was on disk but the repo wasn't busy: live commits and a track went through on a repo whose wipe was about to start. work on a shard now calls ShardState::mark for each job it started, and the shard picks them up from their markers right after that work, before its next message. the shard helpers hand work the shard mutably for it, ResumeJob and its shard arm go, and handle_live returns the one repo its own txn started. the erase/wipe test that sent ResumeJob marks the repo instead. behaviour changes only by closing that window, which the new test pins: with every bulk permit held so the wipe can't run, the repo is busy on the shard's next message.

[indexer] announce an account deletion on /stream

handle_account's deleted branch started the wipe and returned before the push of the account event onto the stream outbox, so /stream never heard of a deletion. jetstream got its account event, which is pushed before the branch, and every other transition (inactive but not deleted, reactivation) was announced. the wipe stages no record deletes either, so a /stream reader had no way to learn the repo was gone. main has the same early return. the account event is now pushed, when the event changed anything, before the branches, so the deleted case announces it like the rest. no per-record delete events: the ephemeral account event is the signal, and a reader that was offline checks GET /repos.

[control] hand out the shards' sender before they start

Hydrant::run spawned the shard threads before it put their sender in state.indexer. a repo write that found no sender ran in place, holding the slot so the sender couldn't go in meanwhile, but the shards were already running: at their start they resume the erases and wipes their markers name, so such a write could overlap a resumed erase or wipe step on the same repo, the two-writer case the owner shards exist to rule out. the sender goes in first now, so a write in place finishes before any shard starts, and once it's in, writes go to the shards' channels and wait there for them. the new test holds the slot like a write in place and checks no shard starts until it lets go. behaviour only changes in that startup window.

[indexer] one helper decides between the owner shard and in place

the choice between running a repo write on its owner shard or in place, while no shard takes work yet, was spelled four times: IndexerTx::on_owner plus repos::on_owner_blocking (each building the OnShard and its reply by hand), repos::in_place for the crawler, and an inline lock in on_owner_shards. the multi-repo callers (track/untrack/resync and the crawler's page) also each split their dids by shard into batches of 256 on their own. now `place` makes the choice once, holding the sender off while it writes in place, and on_owner, on_owner_blocking and a new on_owners (split by shard, 256 to a message, an answer per message) build on it. OnShard::answering is the one place that wraps work with its reply, and IndexerTx keeps on_owner and gains on_owner_blocking as the sending half. the crawler's listings and unknown dids go through on_owners as one list, so a shard gets both in one message and applies listings first like before. REPOS_PER_MESSAGE lives in one place. no behaviour changes, with two small differences in races: the async on_owner sends without parking a blocking pool thread, and a crawler page that found no shards but saw them start before it wrote now hands its writes to them instead of failing with "try again".

[types] drop the exclude flag from the erase job marker

BulkJobV1::Erase carried whether the erase also excluded the repo, but nothing read it. every reader matched Erase { .. }, because start_erase writes the exclude into the filter in the same txn as the marker, so a resumed erase finds it there. nothing with the marker has been pushed, so the stored shape changes without a migration: the golden fixture moves to the unit variant. no behaviour changes.

[tests] relay hand-off tests run on the pipeline harness

worker.rs had a harness of its own for eight tests: a gated plc stub, a pds that hangs up and a relay worker whose forwarded messages the test applied to a served indexer shard by hand. pipeline_tests.rs already has all of it except the last part, so it gets Indexer::Manual (the relay's messages come to the test through next_forwarded, apply hands one to a real shard and waits until it ran), and the eight tests move there unchanged in what they set up and assert. the gated doc is Plc::gate, the wait for it is plc hits, and the raw row writes stay raw. no behaviour change.

[ingest] the relay sinks stage the repo row

worker.rs::commit and context.rs had five cfg blocks for one per-mode step: relay and bare builds put the staged row on the txn, indexer builds handed it to the overlay after the commit. that's what the per-mode sinks are for, so it moves behind them as sink.stage_row(state, txn, did, loaded, staged, firehose). the relay and bare sinks put the row through one shared put_row in sink.rs. the indexer sink keeps it and, once the commit is through, stages it on the overlay and hands it to the frame's events, so the order is what it was, and a failed commit still discards it. stage_repo_row and take_row leave WorkerContext. the indexer sink still commits the relay's txn though it's empty in indexer builds. skipping it would save about nothing (fjall returns early on an empty batch), and a txn dropped unread would silently lose the first outbox entry or pds account a later change puts there. no behaviour change.

[state] relay builds keep no firehose dispatch times

only the crawler reads firehose_dispatched, and the crawler's wait is indexer only, so relay builds kept a map per relay source and bumped it on every watermark move for nobody. the map and note_dispatched are indexer only now, and control/firehose.rs goes through track_dispatched and forget_dispatched, which are no-ops in other builds, instead of touching the map. the test that a source which gave up stops counting is indexer only with it. no behaviour change in indexer builds.

[ingest] lookup counters compile away outside tests

the lookup pool bumped nine atomics on every submit, park and drop, but the only reader is the cfg(test) stats(). they're a zero-size Counter outside tests now, so the hot path does nothing for them. i didn't move them into firehose-diagnostics: the relay shard stats there are per shard and the pool is shared, so they'd need a snapshot section of their own, which is more than this cleanup is for. no behaviour change.

[firehose] count-only stats for doc refreshes and key fetches

the relay shard no longer waits on these, a lookup does, so both calls recorded Duration::ZERO and fetch_key_micros, refresh_doc_micros and their max fields always read 0 while looking like latencies. record_fetch_key and record_refresh_doc only count now, the four duration fields leave RelayShardStatsSnapshot, and docs/api/firehose.md points at resolve_doc_micros for how long lookups take. no behaviour change outside the diagnostics output.

[backfill] build Spills in one place

open and the test-only never each spelled out the whole Inner literal, so a new field had to be added twice with the same cfg(test) dance. both call one constructor now. no behaviour change.

[backfill] share the mock pds plumbing between the backfill tests

the process, sparse and task tests each built their own client that reaches a local server as pds.example, two of them had their own serve, and two their own sh.tangled.repo record. they live in backfill::test_util now, so a change to how the backfill client is built lands in one place. tests only, no behaviour change.

[backfill] move the records view out of process.rs

RepoRecords, the whole car's blocks, the first pass and the record and stored key types lived in process.rs, so apply.rs imported them back from the module that calls it, and process.rs mixed the fetch with how both passes read a repo. they move to records.rs as they were, with the test that covers them, and only the visibility process.rs needs is added. no behaviour change.

[backfill] one outcome enum for a backfill

how a backfill ended went through three enums: apply's Written, process's Fetched and task's Handoff, each mapped into the next, and Fetched::Dropped hid three different reasons. apply.rs's Written also shared its name with an unrelated enum on the shard side. there's one Outcome now, made in apply and returned by process_did as is, so did_task matches what really happened. no behaviour change.

[backfill] keep the spill check interval a const outside tests

the interval between spill headroom checks sat in an atomic that every appended block loaded, only so a test could lower it. production now reads the const and only test builds keep the atomic. no behaviour change.

[docs] say STREAM_SEND_TIMEOUT covers a whole websocket batch

the websocket sends up to 64 frames or 64 KiB in one write when they're ready together, and STREAM_SEND_TIMEOUT bounds that whole write. the reference said "a websocket write that doesn't finish", which read like one frame. docs only.

[stream] say what the unchecked collection and rkey really risk

the SAFETY comment on replay's Nsid::unchecked and Rkey::unchecked said hydrant checked both when it wrote the event, while the test right below says rows written before ingest checked them can hold a collection no ingest would take. it now says what holds: nothing unsound rests on them, the db is trusted so a replay doesn't check again, and Nsid::name panics on one without a dot. the test's doc named the old commit that started checking, it now says the same without the hash. Decoded::frame_bytes estimated a frame for a form that only goes to in-process readers, which are never capped, so nothing ever paid for it. it returns 0 and says why. no behaviour changes.

[throttle] drop the egress pool size every caller passed as 1

Throttler::new took a pool size that multiplied per-pds concurrency and every rate tier window, meant for spreading backfill over proxy egress IPs. that code predates the stack, and backfill proxies have long been refused in indexer builds and do nothing in relay ones, so every caller passed 1 and nothing in config could set anything else. the parameter, the field on the handle, and the scaling in limits_for go, and DEFAULT_PER_PDS_CONCURRENCY_PER_IP loses its per ip. the two tests that only checked the scaling go with it. no behaviour changes: with a pool of 1 every limit and semaphore size is what it was.