[config] add REWALK_MB
16 unmodified lines17181920212122232425135 unmodified linesExpand all1611621631641651661671681697 unmodified lines16 unmodified lines- durations use [humantime](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html) and need a unit: `3s`, `5min`, `2h`, `3d`. a bare `300` is an error, only a bare `0` works without one.- the fixed-choice ones (`DATA_COMPRESSION`, `VERIFY_SIGNATURES`, `BACKFILL_STRATEGY`, ...) only take the values listed for them, lowercase.- the ones that take `none` take it in any case.- lists are comma separated, entries trimmed, empty entries dropped. a bad entry in `RELAY_HOSTS`, `SEED_HOSTS`, `CRAWLER_URLS`, `RATE_TIERS` or `TIER_RULES` is dropped with a warning in the log, and the rest of the list is used. `PLC_URL` and `FILTER_EXCLUDES` are stricter, a bad entry there is an error.- values that parse but would crash hydrant or spin a loop are errors too: `FIREHOSE_WORKERS`, `DB_WORKER_THREADS`, `BACKFILL_CONCURRENCY_LIMIT` and `BACKFILL_SPILLS` (indexer), `DB_EVENTS_MEMTABLE_SIZE_MB` and `DB_RECORDS_MEMTABLE_SIZE_MB` of `0`, a zero `CURSOR_SAVE_INTERVAL`, `SEED_POLL_INTERVAL` or `OFFLINE_HOST_RETRY_INTERVAL`, `REPO_REFETCH_LIMIT` or `HOST_REFETCH_LIMIT` of `0`, a zero `REPO_REFETCH_WINDOW` or `REWALK_WINDOW`, `DB_MAX_JOURNALING_SIZE_MB` below `64` and `DB_REPOS_MEMTABLE_SIZE_MB` below `2`.- values that parse but would crash hydrant or spin a loop are errors too: `FIREHOSE_WORKERS`, `DB_WORKER_THREADS`, `BACKFILL_CONCURRENCY_LIMIT` and `BACKFILL_SPILLS` (indexer), `DB_EVENTS_MEMTABLE_SIZE_MB` and `DB_RECORDS_MEMTABLE_SIZE_MB` of `0`, a zero `CURSOR_SAVE_INTERVAL`, `SEED_POLL_INTERVAL` or `OFFLINE_HOST_RETRY_INTERVAL`, `REPO_REFETCH_LIMIT` or `HOST_REFETCH_LIMIT` of `0`, a zero `REPO_REFETCH_WINDOW` or `REWALK_WINDOW`, `REWALK_MB` of `0`, `DB_MAX_JOURNALING_SIZE_MB` below `64` and `DB_REPOS_MEMTABLE_SIZE_MB` below `2`.- `BACKFILL_PROXIES` with anything in it is an error in an indexer.
## fixed when the database is created
135 unmodified linesExpand all| `REPO_REFETCH_LIMIT` | `6` | how many full refetches one repo may use per `REPO_REFETCH_WINDOW`, whatever asks for them (a `#sync`, a chain break, a newer listing, a crawl). `none` (any case) means unlimited. `0` is an error, the repo could never be refetched again. see [refetch budgets](concepts/relay.md#refetch-budgets) || `HOST_REFETCH_LIMIT` | `60` | how many full refetches the repos of one PDS may use together per `REPO_REFETCH_WINDOW`. hosts on the built-in `trusted` tier are exempt. `none` (any case) means unlimited, `0` is an error. a repo with no known PDS only counts against its own limit || `REPO_REFETCH_WINDOW` | `1h` | how long a refetch allowance lasts, counted from the first refetch it covers (a humantime duration, `0s` is an error) || `REWALK_WINDOW` | `30min` | in an indexer, how long a followed host's `listRepos` rewalk allowance lasts, counted from the first page a rewalk reads (a humantime duration, `0s` is an error). see [`requestCrawl`](concepts/relay.md) || `REWALK_MB` | `8` | in an indexer, how many MiB of `listRepos` responses a followed host's rewalks may read per `REWALK_WINDOW`. a rewalk that spends it stops and resumes when the window ends. `0` is an error, no rewalk could read a page. see [`requestCrawl`](concepts/relay.md) || `RATE_TIERS` | | comma-separated list of named rate tier definitions in `name:base/mul/hourly/daily[/account_limit]` format (e.g. `trusted:5000/10.0/18000000/432000000/10000000`). the per second limit is `max(base, accounts × mul)`, with hourly and daily caps on top. it paces firehose events from `pds::` sources and backfill requests to that PDS. the optional account limit prevents new accounts from being created on a PDS once reached, and leaving it off (or giving one that doesn't parse) means no account limit at all. built-in tiers (`default`, `trusted`) are always present and can be overridden. in indexer builds `default` has the same limits as `trusted` (5000/s), so pacing rarely kicks in for backfill there. in relay builds it's 50/s with a 100 account limit || `TIER_RULES` | | comma-separated ordered list of glob rules in `pattern:tier_name` format (e.g. `*.bsky.network:trusted`). rules are evaluated in order, first match wins. explicit API assignments via `PUT /pds/tiers` take precedence, and the `default` tier is the final fallback (defaults to `trusted` in indexer mode). a rule naming a tier that doesn't exist falls back to `default`. uses standard glob wildcards (`*`, `?`) matched against the PDS hostname |
## compatibility settings7 unmodified lines262 unmodified linesExpand all263264265266267268269270271272273274275169 unmodified linesExpand all445446447448449450451452453147 unmodified linesExpand all601602603604605606607608609610611612613362 unmodified linesExpand all262 unmodified linesExpand all /// reads. the window starts there, it doesn't slide. /// set via `HYDRANT_REWALK_WINDOW` (humantime duration, e.g. `30min`). pub rewalk_window: Duration,
/// how many bytes of `listRepos` responses a followed host's rewalks may read per /// `rewalk_window`, in MiB. a walk that spends it stops and resumes when the window ends. /// set via `HYDRANT_REWALK_MB`. pub rewalk_mb: u32,
/// how often offline firehose sources are automatically retried. /// set via `HYDRANT_OFFLINE_HOST_RETRY_INTERVAL` (humantime duration, e.g. `30min`). /// set to `none` to disable automatic retries. pub offline_host_retry_interval: Option<Duration>,169 unmodified linesExpand all repo_refetch_limit: Some(6), host_refetch_limit: Some(60), repo_refetch_window: Duration::from_secs(60 * 60), rewalk_window: Duration::from_secs(30 * 60), rewalk_mb: 8, offline_host_retry_interval: Some(Duration::from_secs(30 * 60)), tier_rules: vec![], tier_policy: { let mut tiers = HashMap::new();147 unmodified linesExpand all miette::bail!( "invalid HYDRANT_REWALK_WINDOW `0s`: no rewalk could be counted, want a duration like 30min" ); } if self.rewalk_mb == 0 { miette::bail!( "invalid HYDRANT_REWALK_MB `0`: no rewalk could read a page, want at least 1" ); } if self.db_worker_threads == 0 { miette::bail!( "invalid HYDRANT_DB_WORKER_THREADS `0`: the database needs at least 1 thread to flush" );362 unmodified linesExpand all432 unmodified linesExpand all433434435436437438439440441147 unmodified linesExpand all589590591592593594595596597271 unmodified linesExpand all86987087187287387487587687719 unmodified lines89789889990090190290390490512 unmodified lines91891992092192292392492592692728 unmodified lines95695795895996096196296396496524 unmodified lines432 unmodified linesExpand all defaults.host_refetch_limit, )?; let repo_refetch_window = cfg!(lookup, "REPO_REFETCH_WINDOW", defaults.repo_refetch_window); let rewalk_window = cfg!(lookup, "REWALK_WINDOW", defaults.rewalk_window); let rewalk_mb = cfg!(lookup, "REWALK_MB", defaults.rewalk_mb); let seed_poll_interval = read_or_none( &lookup, "HYDRANT_SEED_POLL_INTERVAL", defaults.seed_poll_interval,147 unmodified linesExpand all repo_refetch_limit, host_refetch_limit, repo_refetch_window, rewalk_window, rewalk_mb, offline_host_retry_interval: offline_retry_interval, tier_policy, tier_rules, cache_size,271 unmodified linesExpand all ("HYDRANT_HOST_REFETCH_LIMIT", "500"), ("HYDRANT_REPO_REFETCH_LIMIT", "2"), ("HYDRANT_REPO_REFETCH_WINDOW", "10s"), ("HYDRANT_REWALK_WINDOW", "10s"), ("HYDRANT_REWALK_MB", "1"), ("HYDRANT_DATA_COMPRESSION", "lz4"), ("HYDRANT_JOURNAL_COMPRESSION", "none"), ("HYDRANT_VERIFY_SIGNATURES", "backfill-only"), ("HYDRANT_BACKFILL_STRATEGY", "sparse-filter"),19 unmodified lines ("HYDRANT_REPO_REFETCH_LIMIT", "2"), ("HYDRANT_HOST_REFETCH_LIMIT", "9"), ("HYDRANT_REPO_REFETCH_WINDOW", "10s"), ("HYDRANT_REWALK_WINDOW", "10s"), ("HYDRANT_REWALK_MB", "3"), ("HYDRANT_SEED_POLL_INTERVAL", "NONE"), ("HYDRANT_MAINTENANCE_TIMEOUT", "0"), ]) .unwrap();12 unmodified lines assert_eq!(defaults.host_refetch_limit, Some(60)); assert_eq!(defaults.repo_refetch_window, Duration::from_secs(3600)); assert_eq!(config.rewalk_window, Duration::from_secs(10)); assert_eq!(defaults.rewalk_window, Duration::from_secs(30 * 60)); assert_eq!(config.rewalk_mb, 3); assert_eq!(defaults.rewalk_mb, 8); assert_eq!(config.seed_poll_interval, None); assert_eq!(config.maintenance_timeout, Duration::ZERO); }
28 unmodified lines ("HYDRANT_REPO_REFETCH_WINDOW", "0s"), ("HYDRANT_REPO_REFETCH_WINDOW", "never"), ("HYDRANT_REWALK_WINDOW", "0s"), ("HYDRANT_REWALK_WINDOW", "never"), ("HYDRANT_REWALK_MB", "0"), ("HYDRANT_REWALK_MB", "8MiB"), ("HYDRANT_DATA_COMPRESSION", "gzip"), ("HYDRANT_JOURNAL_COMPRESSION", "LZ4"), ("HYDRANT_VERIFY_SIGNATURES", "yes"), ("HYDRANT_BACKFILL_STRATEGY", "fast"),24 unmodified lines