[docs] describe what requestcrawl does for each kind of host
81 unmodified lines8283848586868788899080 unmodified lines81 unmodified lines| `SEED_POLL_INTERVAL` | `30min` | humantime interval between completed `listHosts` polls. set to `none` (any case) to seed once at startup. `0s` is an error || `ENABLE_FIREHOSE` | `true` | whether to ingest relay subscriptions. `false` keeps the sources from connecting, but their tasks, the shard threads, seeding and the offline retry loop all still run. can also be toggled at runtime through `/ingestion`. || `FIREHOSE_WORKERS` | `8` (`24` full network) | number of firehose shards. messages go to a shard by a hash of their repo's did, so one repo's messages stay in order, and each shard is its own thread with a queue of 64 messages. an indexer runs two sets of shards, one to validate and one to write records, so `2n` threads, and a relay runs `n`. a shard doesn't wait on identity lookups: it holds a repo's messages while a shared lookup pool resolves it and goes on with the others. every commit that emits events still goes through one shared lock, so more shards don't add write throughput past that. backfill results also drain through the shards, so they run even with the firehose off. at least `1` || `FIREHOSE_MAX_FAILURES` | `15` | consecutive connect or stream failures before hydrant gives up on a source. the wait between tries doubles from 15 seconds and is capped at a minute, so `15` is about 14 minutes of retrying. failures only reset after a minute connected. a PDS that hits it is marked offline. given-up sources come back through `OFFLINE_HOST_RETRY_INTERVAL`. `0` and `1` both give up after the first failure || `OFFLINE_HOST_RETRY_INTERVAL` | `30min` | how often sources that were given up on get restarted, which also resets their failure count. `none` (any case) disables it. `0s` is an error, it would restart dead hosts in a hot loop || `OFFLINE_HOST_RETRY_INTERVAL` | `30min` | how often sources that were given up on get restarted, which also resets their failure count. in an indexer it's also the window a followed host's 3 requested rewalks count in. `none` (any case) disables it. `0s` is an error, it would restart dead hosts in a hot loop || `CURSOR_SAVE_INTERVAL` | `3sec` | how often a background thread saves every firehose cursor, checkpoints counts and writes the journal buffer out to the os. it's a write to the os, not an fsync, so it protects against a crash of hydrant but not a power loss. after a crash hydrant re-reads up to this much of the firehose, which is cheap since already applied commits are skipped. every source's cursor is rewritten each tick, changed or not, which adds up on a relay with thousands of seeded sources. `0s` is an error, it would save in a hot loop |
## identity & verification
80 unmodified lines42 unmodified lines4344454647484950515253545556575859606119 unmodified lines42 unmodified linesin indexer mode that live tail would miss repos that already lived on a new PDS, so a host that is new to the network also gets one `listRepos` walk, its repos going through the same signal check as crawler sources. a seed's hosts only count as new once its `listHosts` has been read to the end once, so a fresh node doesn't walk every PDS it seeds. a PDS admitted through `com.atproto.sync.requestCrawl` always gets the walk. it runs while the crawler is enabled, resumes from its last page after a restart, and is dropped if the host is banned or removed as a source.
seeding runs from latest cursor on restart so new PDS' added to the upstream relay since the last start are picked up automatically (if they haven't through firehose). sources that are already running are detected and skipped, so re-seeding is idempotent.
## requestCrawl
`com.atproto.sync.requestCrawl` asks hydrant to follow a host. what it does depends on what hydrant already knows about it:
- a banned host gets `HostBanned`.- a host hydrant doesn't follow is added as a PDS firehose source, the same as `POST /firehose/sources`. it counts against `NEW_HOST_LIMIT`, and once that's used up for the UTC day the request gets `RateLimitExceeded`. in indexer mode it also gets the `listRepos` walk [above](#firehose-seeding).- a host hydrant follows whose firehose is connected or still retrying is left as it is.- a host whose firehose hydrant gave up on is restarted right away, after the usual startup jitter, with its failure count reset, instead of waiting for `OFFLINE_HOST_RETRY_INTERVAL`. a PDS that was down for an upgrade can ask to be picked up again this way.
in indexer mode a host hydrant follows also gets walked again when it asks, connected or not. a rewalk adds the repos the host lists that we've never seen, like the first walk, and resyncs the ones whose doc points at the host and that we hold desynchronized or with a failed backfill waiting on its retry. a repo we hold as synced is left to the firehose, whatever rev the listing has. the resync is the same one `POST /repos/resync` does, so a repo keeps its retry count. a host gets 3 rewalks per `OFFLINE_HOST_RETRY_INTERVAL` (30 minutes when that's off), counted from its first one and kept across restarts. a request while its walk is queued or running, or once the 3 are used, gets the usual `200` and does nothing. a rewalk doesn't count against `NEW_HOST_LIMIT`, isn't given to a banned host, and leaves the host's backoff alone: it's paced and throttled like any other walk.
## crawler sources
the crawler is configured separately from the firehose via `CRAWLER_URLS`. each source is a `[mode::]url` entry where the mode prefix is optional and defaults to `by_collection` in filter mode or `list_repos` in full-network mode.
19 unmodified lines12 unmodified lines13141516171718192012 unmodified lines- `com.atproto.sync.listHosts`- `com.atproto.sync.getRepoStatus`- `com.atproto.sync.listRepos`- `com.atproto.sync.getLatestCommit`- `com.atproto.sync.requestCrawl` (adds the host to firehose sources)- `com.atproto.sync.requestCrawl` (adds the host to firehose sources, and walks a host it already follows again, see [the relay docs](../concepts/relay.md))- `com.atproto.sync.subscribeRepos` (WebSocket firehose stream, requires `relay` feature; slow consumers may receive a `ConsumerTooSlow` error frame before the connection closes, and a `cursor` past the last seq gets a `FutureCursor` error frame and a close, as the spec says)
the record endpoints (`getRecord`, `getRecordByUri`, `listRecords`, both `describeRepo`s, `systems.gaze.hydrant.countRecords`), `getRepo` and `getLatestCommit` don't serve a repo whose account is takendown, suspended, deactivated or deleted, even while its records are still stored. `getRecord` and `getRecordByUri` answer `RecordNotFound` with the reason in the message, the rest answer with their own error for the status or `InvalidRequest`.