--- title: relay, seeding & crawler sources --- ## multiple relay support `hydrant` supports connecting to multiple relays simultaneously for firehose ingestion. when `RELAY_HOSTS` is configured with multiple URLs: - one independent firehose stream loop is spawned per relay - each relay maintains its own firehose cursor state - all ingestion loops share the same worker pool and database commit events are de-duplicated according to the repo `rev`. account / identity events are de-duplicated using the `time` field. a relay build also seeds sources from `SEED_HOSTS` (`https://bsky.network` by default, see [firehose seeding](#firehose-seeding)), which adds a firehose per PDS on top of `RELAY_HOSTS`. to follow only the relays you listed, set `HYDRANT_SEED_HOSTS=` (empty). ## direct PDS connections a firehose source can also be a direct connection to a PDS rather than a relay. prefix the URL with `pds::` to mark it as such: ``` HYDRANT_RELAY_HOSTS=wss://bsky.network,pds::wss://pds.example.com ``` only when a source is marked as a direct PDS (`is_pds: true`), hydrant enforces host authority. relays (`is_pds: false`, the default) are exempt from this check, since they forward commits from many PDSes by design. ## firehose seeding in relay mode, `RELAY_HOSTS` defaults to empty. set `SEED_HOSTS` to one or more relay base URLs and hydrant will call `com.atproto.sync.listHosts` on each at startup and every 30 minutes thereafter, adding every returned PDS as a firehose source: ``` HYDRANT_SEED_HOSTS=https://bsky.network ``` seeding runs as a background task so the main firehose loop is not blocked. seed URLs are fetched concurrently (up to four at a time) and the full `listHosts` pagination is consumed for each. if a request fails partway through, the hosts collected so far are still added and the failure is logged. each discovered host is added as a persistent PDS firehose source (`is_pds: true`), equivalent to calling `POST /firehose/sources`. banned hosts (`status: "banned"`) are skipped. all other statuses are included since the firehose ingestor retries on disconnect and transiently-unavailable hosts will reconnect on their own. a host we have no cursor for starts from the seq the seed relay reported for it, or from the live head when it reported none, so its retained history is not replayed. hosts that already have a cursor keep it. in indexer mode that live tail would miss repos that already lived on a new PDS, so a host that is new to the network also gets one `listRepos` walk, its repos going through the same signal check as crawler sources. a seed's hosts only count as new once its `listHosts` has been read to the end once, so a fresh node doesn't walk every PDS it seeds. a PDS admitted through `com.atproto.sync.requestCrawl` always gets the walk. it runs while the crawler is enabled, resumes from its last page after a restart, and is dropped if the host is banned or removed as a source. seeding runs from latest cursor on restart so new PDS' added to the upstream relay since the last start are picked up automatically (if they haven't through firehose). sources that are already running are detected and skipped, so re-seeding is idempotent. ## requestCrawl `com.atproto.sync.requestCrawl` asks hydrant to follow a host. the `hostname` is a bare hostname, since one with a port is refused, so hydrant only follows and walks a host on the default port (`wss` for the firehose, `https` for `listRepos`). what it does depends on what hydrant already knows about it: - a banned host gets `HostBanned`. - a host hydrant doesn't follow is added as a PDS firehose source, the same as `POST /firehose/sources`. it counts against `NEW_HOST_LIMIT`, and once that's used up for the UTC day the request gets `RateLimitExceeded`. in indexer mode it also gets the `listRepos` walk [above](#firehose-seeding). - a host hydrant follows whose firehose is connected or still retrying is left as it is. - a host whose firehose hydrant gave up on is restarted right away, after the usual startup jitter, with its failure count reset, instead of waiting for `OFFLINE_HOST_RETRY_INTERVAL`. a PDS that was down for an upgrade can ask to be picked up again this way. in indexer mode a host hydrant follows also gets walked again when it asks, connected or not. a rewalk adds the repos the host lists that we've never seen, like the first walk. it also resyncs the ones whose doc points at the host and that we hold desynchronized or with a failed backfill waiting on its retry, and the ones the host lists at a newer rev than ours. the spec doesn't define that last one (the IETF drafts don't cover crawling, and the lexicon only says to "persistently crawl hosted repos"). it's how hydrant lets a host without a firehose say it changed. a newer rev costs the repo one of its refetches, see [below](#refetch-budgets). the stuck ones are the same resync `POST /repos/resync` does, so a repo keeps its retry count. a host's rewalks may read `REWALK_MB` (8 by default) of `listRepos` responses per `REWALK_WINDOW` (30 minutes by default), counted from the first page one reads and kept across restarts, so a host can ask as often as it likes while it stays inside that. a walk charges each page's size once it has read it, so it can overshoot by a page, and stops before the next one when the bytes are spent. it keeps its cursor and goes on when the window ends, also after a restart. a request while its walk is queued, running or waiting, or once the bytes are spent, gets the usual `200` and does nothing. every `listRepos` walk, a new host's included, refuses a page over 512 KiB or with more repos than the 1000 we asked for, and ends with an error when a page returns a cursor an earlier page of the walk already returned. a rewalk doesn't count against `NEW_HOST_LIMIT`, isn't given to a banned host, and leaves the host's backoff alone: it's paced and throttled like any other walk. ## refetch budgets a full refetch is the cheapest thing a flaky or hostile peer can ask of hydrant: a `#sync` at a fresh rev, a chain break, a `listRepos` listing at a newer rev, a crawl's resync. each one costs the repo a refetch out of `REPO_REFETCH_LIMIT` per `REPO_REFETCH_WINDOW` (6 an hour), counted from the repo's first refetch in the window. the repos of one host also share `HOST_REFETCH_LIMIT` (60) in the same window. hosts on the built-in `trusted` tier are exempt from that one, because an operator who trusts a host already accepts its volume, but their repos keep their own limit. a repo that's out of refetches isn't refetched. it goes `throttled` (an inactive account keeps its status) with a resync entry of kind `ratelimited` whose `next_retry` is the end of the window, and it's refetched then. commits for it that arrive meanwhile are dropped because the refetch replaces what they change. a `#sync` that doesn't move a repo's rev forward is ignored, and a listing dated more than `REV_CLOCK_SKEW` past hydrant's clock isn't taken for news. retries of a refetch that was admitted and failed don't cost another, and neither does `POST /repos/resync`, because the repo's backoff or an operator bounds those. `/stats` counts the refetches denied since start as `counts.refetch_denied`. ## crawler sources the crawler is configured separately from the firehose via `CRAWLER_URLS`. each source is a `[mode::]url` entry where the mode prefix is optional and defaults to `by_collection` in filter mode or `list_repos` in full-network mode. - `list_repos`: enumerates the relay via `com.atproto.sync.listRepos`. relay account status is reconciled for repos already known to hydrant, but a relay listing an active repo as inactive only queues a fetch from its pds (once per listed status), and the account goes inactive only if the pds refuses it, so a relay-level takedown of a repo its pds still serves leaves it synced. the same goes the other way: a relay listing an inactive repo as active at the revision hydrant has only queues a fetch, and the account comes back only if the pds serves it; filtered indexes check unknown repos with `describeRepo` before enqueueing them. a listing whose revision is newer than the one hydrant stored queues a backfill only once hydrant's own firehose of that relay has caught up to when the page was read (the slowest relay firehose for a host hydrant doesn't subscribe to), since a commit still on its way through hydrant would look the same. the crawl keeps reading pages meanwhile, with up to 32 of them waiting, and a page's cursor only moves once it and every page before it are in. if the firehose doesn't get there within 10 minutes, those listings are skipped until the next pass, with one log line. - `relay_first`: an explicit coverage-over-moderation opt-in. it completes the relay's authoritative `listRepos` pass first, then calls the same relay's `listHosts` and crawls `listRepos` on each safe, non-banned PDS. relay-listed inactive repos are retained as status tombstones, so deep pages can only add DIDs the relay did not list and never update relay-authoritative status or revision state. this may discover inactive or other repositories omitted by the relay, which also means it deliberately bypasses relay-level repository moderation for those unknown DIDs. local PDS bans and filter excludes still win. - `by_collection`: queries `com.atproto.sync.listReposByCollection` for each configured signal. more efficient for filtered indexing since it only surfaces repos that have matching records. cursors are stored per collection. note that it won't crawl anything if no signals are specified. ``` CRAWLER_URLS=by_collection::https://lightrail.microcosm.blue,list_repos::wss://bsky.network ``` for deep discovery in a full-network indexer: ``` CRAWLER_URLS=relay_first::https://relay.fire.hose.cam ``` `relay_first` processes at most four PDSes concurrently. each PDS request uses the shared per-PDS concurrency gate and rate-tier pacing, holds its permit through the bounded response read, and feeds 429/hard failures back into the normal PDS throttler. unsafe/private DNS targets, locally or upstream-banned hosts, offline hosts, and PDSes without `listRepos` support are skipped. relay-pass, `listHosts`, and per-PDS cursors are durable; a restart resumes the last committed page, and `/crawler/cursors` clears all of them for the source. each source maintains its own cursor so restarts resume mid-pass. sources can also be added and removed at runtime via the `/crawler/sources` API (see [crawler management](../api/crawler.md)). dynamically added sources are persisted to the database and survive restarts. `CRAWLER_URLS` sources are startup-only: they are not written to the database and will always reappear after a restart regardless of runtime changes (unless you change the config of course).