[docs] describe refetch budgets and rewalking newer repos
16 unmodified lines1718192021212223242591 unmodified lines11711811912012112112212312412531 unmodified lines1571581591601611621631641651661677 unmodified lines16 unmodified lines- durations use [humantime](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html) and need a unit: `3s`, `5min`, `2h`, `3d`. a bare `300` is an error, only a bare `0` works without one.- the fixed-choice ones (`DATA_COMPRESSION`, `VERIFY_SIGNATURES`, `BACKFILL_STRATEGY`, ...) only take the values listed for them, lowercase.- the ones that take `none` take it in any case.- lists are comma separated, entries trimmed, empty entries dropped. a bad entry in `RELAY_HOSTS`, `SEED_HOSTS`, `CRAWLER_URLS`, `RATE_TIERS` or `TIER_RULES` is dropped with a warning in the log, and the rest of the list is used. `PLC_URL` and `FILTER_EXCLUDES` are stricter, a bad entry there is an error.- values that parse but would crash hydrant or spin a loop are errors too: `FIREHOSE_WORKERS`, `DB_WORKER_THREADS`, `BACKFILL_CONCURRENCY_LIMIT` and `BACKFILL_SPILLS` (indexer), `DB_EVENTS_MEMTABLE_SIZE_MB` and `DB_RECORDS_MEMTABLE_SIZE_MB` of `0`, a zero `CURSOR_SAVE_INTERVAL`, `SEED_POLL_INTERVAL` or `OFFLINE_HOST_RETRY_INTERVAL`, `DB_MAX_JOURNALING_SIZE_MB` below `64` and `DB_REPOS_MEMTABLE_SIZE_MB` below `2`.- values that parse but would crash hydrant or spin a loop are errors too: `FIREHOSE_WORKERS`, `DB_WORKER_THREADS`, `BACKFILL_CONCURRENCY_LIMIT` and `BACKFILL_SPILLS` (indexer), `DB_EVENTS_MEMTABLE_SIZE_MB` and `DB_RECORDS_MEMTABLE_SIZE_MB` of `0`, a zero `CURSOR_SAVE_INTERVAL`, `SEED_POLL_INTERVAL` or `OFFLINE_HOST_RETRY_INTERVAL`, `REPO_REFETCH_LIMIT` or `HOST_REFETCH_LIMIT` of `0`, a zero `REPO_REFETCH_WINDOW`, `DB_MAX_JOURNALING_SIZE_MB` below `64` and `DB_REPOS_MEMTABLE_SIZE_MB` below `2`.- `BACKFILL_PROXIES` with anything in it is an error in an indexer.
## fixed when the database is created
91 unmodified lines
| variable | default | description || :--- | :--- | :--- || `CRAWLER_URLS` | relay hosts as `list_repos` (full network), `by_collection::https://lightrail.microcosm.blue` (filter) | comma-separated list of `[mode::]url` crawler sources. mode is `list_repos`, `relay_first`, or `by_collection` (dashes work too), bare URLs use the default mode (`list_repos` with full network, `by_collection` otherwise). `list_repos` pages a relay's repo list, `by_collection` asks for repos holding each `FILTER_SIGNALS` collection and does nothing without signals. `relay_first` is an explicit coverage-over-moderation opt-in. an entry with an unknown mode or bad URL is dropped with a warning. set to empty string to disable the configured sources. sources added through `/crawler/sources` are saved in the database and still start || `ENABLE_CRAWLER` | `true` if full network or sources configured | whether to actively query the network for unknown repositories. since `CRAWLER_URLS` has a default in every mode, unset means on unless `CRAWLER_URLS` is empty. leave it unset (or empty) for that default. with `false` the crawler tasks still exist and wait. can also be toggled at runtime through `/ingestion` || `ENABLE_CRAWLER` | `true` if full network or sources configured | whether to actively query the network for unknown repositories. since `CRAWLER_URLS` has a default in every mode, unset means on unless `CRAWLER_URLS` is empty. leave it unset (or empty) for that default. with `false` the crawler tasks still exist and wait. that includes the `listRepos` walks `com.atproto.sync.requestCrawl` queues (a new host's first walk and a followed host's rewalks): they wait while it's off and go on once it's on. can also be toggled at runtime through `/ingestion` || `CRAWLER_MAX_PENDING_REPOS` | `2000` (`25000` full network) | persistent discovery lookahead retained in the pending queue before the crawler pauses. this does not increase active backfill task concurrency. it's checked before each page of discovered repos is queued, so the queue can go over by one page (up to 2000 repos), and while paused the crawler rechecks every 5 seconds || `CRAWLER_RESUME_PENDING_REPOS` | `1000` (`12500` full network) | pending-repo count at which discovery resumes. not checked against the max, and a value at or above it just resumes as soon as the queue is back under the max |
## stream31 unmodified lines
| variable | default | description || :--- | :--- | :--- || `NEW_HOST_LIMIT` | `50` | how many new hosts can be added via `com.atproto.sync.requestCrawl` per UTC day. `none` (any case) means unlimited. seeded hosts and ones added through the management API don't count || `REPO_REFETCH_LIMIT` | `6` | how many full refetches one repo may use per `REPO_REFETCH_WINDOW`, whatever asks for them (a `#sync`, a chain break, a newer listing, a crawl). `none` (any case) means unlimited. `0` is an error, the repo could never be refetched again. see [refetch budgets](concepts/relay.md#refetch-budgets) || `HOST_REFETCH_LIMIT` | `60` | how many full refetches the repos of one PDS may use together per `REPO_REFETCH_WINDOW`. hosts on the built-in `trusted` tier are exempt. `none` (any case) means unlimited, `0` is an error. a repo with no known PDS only counts against its own limit || `REPO_REFETCH_WINDOW` | `1h` | how long a refetch allowance lasts, counted from the first refetch it covers (a humantime duration, `0s` is an error) || `RATE_TIERS` | | comma-separated list of named rate tier definitions in `name:base/mul/hourly/daily[/account_limit]` format (e.g. `trusted:5000/10.0/18000000/432000000/10000000`). the per second limit is `max(base, accounts × mul)`, with hourly and daily caps on top. it paces firehose events from `pds::` sources and backfill requests to that PDS. the optional account limit prevents new accounts from being created on a PDS once reached, and leaving it off (or giving one that doesn't parse) means no account limit at all. built-in tiers (`default`, `trusted`) are always present and can be overridden. in indexer builds `default` has the same limits as `trusted` (5000/s), so pacing rarely kicks in for backfill there. in relay builds it's 50/s with a 100 account limit || `TIER_RULES` | | comma-separated ordered list of glob rules in `pattern:tier_name` format (e.g. `*.bsky.network:trusted`). rules are evaluated in order, first match wins. explicit API assignments via `PUT /pds/tiers` take precedence, and the `default` tier is the final fallback (defaults to `trusted` in indexer mode). a rule naming a tier that doesn't exist falls back to `default`. uses standard glob wildcards (`*`, `?`) matched against the PDS hostname |
## compatibility settings7 unmodified lines22 unmodified lines2324252627272829303130 unmodified lines22 unmodified linesboth endpoints say whether hydrant still has work queued for the repository: `pending` is `true` while a backfill of it is queued or running, and a `resync` object is there while it sits in the resync queue. a failed backfill waiting for its retry shows only `resync`, and it moves back to `pending` once the retry is due:
| `state` | fields | meaning || :--- | :--- | :--- || `error` | `kind`, `retry_count`, `next_retry`, `host_failures` | the last backfill failed and is retried at `next_retry`. `retry_count` is how many backfills failed in a row, across retries, until one succeeds, and the wait doubles with it from about 2 minutes up to an hour (a skip because the PDS is throttled waits a minute and doesn't count). `kind` is `ratelimited` (the PDS asked to slow down, or hydrant skipped it because its PDS is throttled), `transport` (the PDS couldn't be reached) or `generic` (anything else). `host_failures` is how many transport failures in a row the repository's PDS has had since hydrant started, so a `ratelimited` repo on a PDS that keeps failing is waiting on a dead host rather than a rate limit || `error` | `kind`, `retry_count`, `next_retry`, `host_failures` | the last backfill failed and is retried at `next_retry`. `retry_count` is how many backfills failed in a row, across retries, until one succeeds, and the wait doubles with it from about 2 minutes up to an hour (a skip because the PDS is throttled waits a minute and doesn't count). `kind` is `ratelimited` (the PDS asked to slow down, or hydrant skipped it because its PDS is throttled), `transport` (the PDS couldn't be reached) or `generic` (anything else). a repo that used up its refetches (see [refetch budgets](../concepts/relay.md#refetch-budgets)) shows `ratelimited` too, with `retry_count` unchanged, `next_retry` at the end of its window and the repository's status `throttled`. `host_failures` is how many transport failures in a row the repository's PDS has had since hydrant started, so a `ratelimited` repo on a PDS that keeps failing is waiting on a dead host rather than a rate limit || `gone` | `status` | the account is inactive (deactivated, takendown, ...) and gets backfilled again once it comes back |
while a repository is `pending` or has a `resync` entry, don't read its records as a snapshot: a backfill writes them in steps, and one that fails partway leaves the steps it got through, so they can be a mix of the old repo and the new one until both are gone. `getRepo` answers `RepoNotFound` while it's pending, and while a failed backfill waits for its retry with records that don't match the repo's root.
30 unmodified lines44 unmodified lines4546474849495051525354555656575859606162636465666719 unmodified lines44 unmodified linesseeding runs from latest cursor on restart so new PDS' added to the upstream relay since the last start are picked up automatically (if they haven't through firehose). sources that are already running are detected and skipped, so re-seeding is idempotent.
## requestCrawl
`com.atproto.sync.requestCrawl` asks hydrant to follow a host. what it does depends on what hydrant already knows about it:`com.atproto.sync.requestCrawl` asks hydrant to follow a host. the `hostname` is a bare hostname, since one with a port is refused, so hydrant only follows and walks a host on the default port (`wss` for the firehose, `https` for `listRepos`). what it does depends on what hydrant already knows about it:
- a banned host gets `HostBanned`.- a host hydrant doesn't follow is added as a PDS firehose source, the same as `POST /firehose/sources`. it counts against `NEW_HOST_LIMIT`, and once that's used up for the UTC day the request gets `RateLimitExceeded`. in indexer mode it also gets the `listRepos` walk [above](#firehose-seeding).- a host hydrant follows whose firehose is connected or still retrying is left as it is.- a host whose firehose hydrant gave up on is restarted right away, after the usual startup jitter, with its failure count reset, instead of waiting for `OFFLINE_HOST_RETRY_INTERVAL`. a PDS that was down for an upgrade can ask to be picked up again this way.
in indexer mode a host hydrant follows also gets walked again when it asks, connected or not. a rewalk adds the repos the host lists that we've never seen, like the first walk, and resyncs the ones whose doc points at the host and that we hold desynchronized or with a failed backfill waiting on its retry. a repo we hold as synced is left to the firehose, whatever rev the listing has. the resync is the same one `POST /repos/resync` does, so a repo keeps its retry count. a host gets 3 rewalks per `OFFLINE_HOST_RETRY_INTERVAL` (30 minutes when that's off), counted from its first one and kept across restarts. a request while its walk is queued or running, or once the 3 are used, gets the usual `200` and does nothing. a rewalk doesn't count against `NEW_HOST_LIMIT`, isn't given to a banned host, and leaves the host's backoff alone: it's paced and throttled like any other walk.in indexer mode a host hydrant follows also gets walked again when it asks, connected or not. a rewalk adds the repos the host lists that we've never seen, like the first walk. it also resyncs the ones whose doc points at the host and that we hold desynchronized or with a failed backfill waiting on its retry, and the ones the host lists at a newer rev than ours. the spec doesn't define that last one (the IETF drafts don't cover crawling, and the lexicon only says to "persistently crawl hosted repos"). it's how hydrant lets a host without a firehose say it changed. a newer rev costs the repo one of its refetches, see [below](#refetch-budgets). the stuck ones are the same resync `POST /repos/resync` does, so a repo keeps its retry count. a host gets 3 rewalks per `OFFLINE_HOST_RETRY_INTERVAL` (30 minutes when that's off), counted from its first one and kept across restarts. a request while its walk is queued or running, or once the 3 are used, gets the usual `200` and does nothing. a rewalk doesn't count against `NEW_HOST_LIMIT`, isn't given to a banned host, and leaves the host's backoff alone: it's paced and throttled like any other walk.
## refetch budgets
a full refetch is the cheapest thing a flaky or hostile peer can ask of hydrant: a `#sync` at a fresh rev, a chain break, a `listRepos` listing at a newer rev, a crawl's resync. each one costs the repo a refetch out of `REPO_REFETCH_LIMIT` per `REPO_REFETCH_WINDOW` (6 an hour), counted from the repo's first refetch in the window. the repos of one host also share `HOST_REFETCH_LIMIT` (60) in the same window. hosts on the built-in `trusted` tier are exempt from that one, because an operator who trusts a host already accepts its volume, but their repos keep their own limit.
a repo that's out of refetches isn't refetched. it goes `throttled` (an inactive account keeps its status) with a resync entry of kind `ratelimited` whose `next_retry` is the end of the window, and it's refetched then. commits for it that arrive meanwhile are dropped because the refetch replaces what they change. a `#sync` that doesn't move a repo's rev forward is ignored, and a listing dated more than `REV_CLOCK_SKEW` past hydrant's clock isn't taken for news. retries of a refetch that was admitted and failed don't cost another, and neither does `POST /repos/resync`, because the repo's backoff or an operator bounds those. `/stats` counts the refetches denied since start as `counts.refetch_denied`.
## crawler sources
the crawler is configured separately from the firehose via `CRAWLER_URLS`. each source is a `[mode::]url` entry where the mode prefix is optional and defaults to `by_collection` in filter mode or `list_repos` in full-network mode.
19 unmodified lines12 unmodified lines13141516171718192012 unmodified lines- `com.atproto.sync.listHosts`- `com.atproto.sync.getRepoStatus`- `com.atproto.sync.listRepos`- `com.atproto.sync.getLatestCommit`- `com.atproto.sync.requestCrawl` (adds the host to firehose sources, and walks a host it already follows again, see [the relay docs](../concepts/relay.md))- `com.atproto.sync.requestCrawl` (adds the host to firehose sources, and walks a host it already follows again to resync its stuck repos and the ones it lists at a newer rev, see [the relay docs](../concepts/relay.md))- `com.atproto.sync.subscribeRepos` (WebSocket firehose stream, requires `relay` feature; slow consumers may receive a `ConsumerTooSlow` error frame before the connection closes, and a `cursor` past the last seq gets a `FutureCursor` error frame and a close, as the spec says)
the record endpoints (`getRecord`, `getRecordByUri`, `listRecords`, both `describeRepo`s, `systems.gaze.hydrant.countRecords`), `getRepo` and `getLatestCommit` don't serve a repo whose account is takendown, suspended, deactivated or deleted, even while its records are still stored. `getRecord` and `getRecordByUri` answer `RecordNotFound` with the reason in the message, the rest answer with their own error for the status or `InvalidRequest`.