# knot-static scale bench: findings (layer 1, end to end)

harness: `benches/scale/` (`cargo bench --bench scale`, env in its header). raw rows and tables:

- `macos-small.{jsonl,md}`: this mac (m-series, apfs), small preset, nginx 1.30.6, release binary.
- `linux-big.{jsonl,md}`: chernobog (12 cores, ext4, nixos), big preset, `nix build .#knot-static`, nginx 1.30.5.
- `linux-kernel.{jsonl,md}`: chernobog, torvalds/linux master (12M objects, 6.6 GB pack).

chernobog is a desktop: both chernobog runs were taken inside
`systemd-run --user --scope -p MemoryMax=6G -p MemorySwapMax=0 -p CPUWeight=20 -p IOWeight=20 -- nice -n 19`,
so their wall times include yielding to the desktop; cpu and rss don't. an earlier unfenced kernel
push swapped the box and isn't used for any number here.

every axis is varied alone from the base point (100 commits, 4 refs, no tickets, 1 repo). sequence
per point: import, cold generate, activate, `nginx -t`, a ref moved in the hosted repo then
`generate --repo` + activate + reload, a push over git-ssh, a `createRecord` ticket, a `putRecord`
on ticket 1. seeding is outside the measurement: history and refs by one `git fast-import`, cobs
signed in-process with knot2's `CobStore` through `tests/common/cobs.rs` (5000 tickets in 14 s).

how to read the tags: [measured] = numbers from these runs; [read] = from reading the code;
[timers] = from temporary `[trace]` timers inside `write_smart_artifacts` (not committed, run on
the mac commits axis), added because the smart phase is one opaque 10-step span.

bench artefacts, not knot-static costs: `activate` and the owner `write` cli poll
`https://<hostname>` for up to 10 s before they return (crawl::announce), and a push waits up to
2 s in `spindle::tell` (wait_served, even with no spindle). the bench knot never serves its own
hostname, so those are fixed 10 s / 2 s floors here: writes are timed until they answer
(`answered s`) and stopped there; activation's own clock is in the jsonl (`phases.activation_s`).

## hot spots, worst scaling first

### 1. canned fetch replies are rebuilt whole on every generation, by a packer 10x slower than git
- [measured] push wall at 1e3 / 1e4 / 1e5 commits (chernobog): 0.16 / 0.98 / 17.7 s of export, of
  which smart artifacts 0.13 / 0.92 / 17.2 s. bytes written per push 1.25 / 13 / 177 MB: the full
  `upload-pack` + `fetch-done` + `shallow` + `shallow-head` are rewritten (changed) every time,
  2.6x the repo's pack per push.
- [measured] with cobs in the repo it gets much worse: cob-put at 1e4 commits 2.6 s / 16 cpu-s,
  at 1e5 commits 45 s, 292 cpu-s, **1.33 GB rss** (cob-create 1.38 GB; a push before any cob:
  20 s, 463 MB).
- [timers] at 1e4 commits (mac): git's full v0 and v2 packs 0.11 s each; knot-pack's `budget`
  full pack 1.1 s (2.0 s once cobs exist) and it is only used for its length; each `have-N` /
  `have-N-done` reply 20 ms without cobs, ~0.95 s with cobs (the cob tips aren't reachable from a
  base, so knot-pack walks all history per reply); 3 bases x 2 replies.
- [read] `git_export::write_fetch_replies`: `budget = reply(full want, done).len()` then 2 knot-pack
  runs per candidate base (up to 8), all from scratch; `write_smart_artifacts` clears `smart/` first.
- fix: size the budget from git's `fetch-done` (already built) instead of a knot-pack run; build
  `have-N` with `git upload-pack` (same request bytes) or fix knot-pack's reachability walk to stop
  at the haves; key every canned reply by a hash of its request and the refs it reads and hardlink
  it from the previous release when unchanged (cob-put already leaves `upload-pack` identical,
  4.9 MB rewritten for nothing). per push: O(history) knot-pack walks x (1 + 2 bases) ->
  one O(history) git pack (or O(new objects) with pack reuse) + O(new objects) per base.

### 2. a ref fetch per 100 refs: export is O(refs^2)
- [measured] cold generate fetch phase vs refs (chernobog): 1e2 0.04 s, 1e3 0.61 s, 1e4 28.6 s
  (cold generate 51 s, 21 s of it smart artifacts).
  cob refs count too: 5000 tickets -> 9.7 s cold fetch, and every push/write at 5000 tickets
  spends ~7.5 s of untraced export in the delta fetch (git_export 8.8 s, traced parts 1.3 s).
- [read] `git_export::fetch_refs` runs one `git fetch` per 100 refspecs; each one makes the
  source advertise all R refs and negotiate: R/100 x O(R).
- fix: one `git fetch --stdin` with every refspec (or fetch `refs/*` and drop what isn't exported).
  O(R^2/100) -> O(R).

### 3. a new cob (or any new ref) rebuilds the export from scratch
- [measured] cob-create vs cob-put: createRecord takes the cache-miss path (fetch all, prune,
  fsck) and writes a whole new objects pack each time: at 1e5 commits 62 s and 270 MB written vs
  45 s / 177 MB for putRecord; at 1e4 commits +6.3 MB new pack per ticket.
- [read] `refs_fast_forwarded` returns false when a ref is missing from the prior export, so an
  added ref (every new ticket, comment, pull, branch, tag) can never take the delta path.
- fix: treat added refs as fast-forwards from nothing (fetch only their objects onto the hardlinked
  prior entry); keep the full path for deletions and rewrites. O(history) -> O(new objects).

### 4. every write re-reads every cob and rewrites every record file
- [measured] the build's unreported remainder ("atproto" column: cob state, chain settle, record
  and blob files) per push/write: tickets 100 / 1000 / 5000 -> 0.07 / 0.60 / 3.0 s; versions 10 /
  100 (10 tickets) -> 0.03 / 0.21 s; repos 50 / 200 -> 0.07 / 0.26 s. a push at 5000 tickets
  is 14.6 s in all. a git-only push at 1000
  tickets rewrites 1.35 MB of byte-identical atproto files (lists 0.84, versions 0.30, records
  0.21 MB) and changes the 1.23 MB records map and the 0.47 MB `.complete.json`; at 5000 tickets
  6.8 MB identical, records map 6.2 MB, `.complete.json` 2.3 MB. each `have-N` reply also carries
  every cob (0.56 MB at 1000 tickets, 2.8 MB at 5000), since cob tips are never in a base.
- [read] `atproto::cob_state` replays `changes_of` for every social entry of the captured repo;
  `read_back`/`whole_tree` walks the whole chain of every repo, carried ones included; `emit`
  writes every repo's car, record, list and version file into the new generation without linking
  the unchanged ones; `.complete.json` inventories every file.
- fix: cache each cob's state by its ref oid in the repo store and settle only changed keys;
  emit content-addressed record/list/version files and hardlink them from the previous generation;
  carry a carried repo's atproto dir like its git mount. per write O(tickets x versions + all
  repos' records) -> O(changed cobs).

### 5. old generations, releases and export entries are never removed
- [measured] base knot: 0.8 MB after the first generate, +0.6 MB per write, 5.8 MB after 9
  generations; 1e4 commits 25.9 -> 85.7 MB after 4 operations; 1e5 commits 363 MB -> 1.17 GB;
  5000 tickets 258 -> 538 MB. linux kernel: see below.
- [read] nothing under `generations/`, `repos/*/releases` or the export store is ever deleted
  (README says so).
- fix: after activation, keep the active generation, the previous K for rollback, and the
  releases/store entries they reference; delete the rest. disk O(generations x repo) -> O(K x repo).

### 6. import scans every small blob with its own git process
- [measured] import of a bare repo: 1e3 / 1e4 / 1e5 commits = 1.5 / 15 / 154 s on chernobog,
  10 / 102 s on the mac, cpu almost all in git spawns. the kernel (millions of small blobs) would
  take hours, so the linux run hardlinks the clone's pack into a created repo instead.
- [read] `provision::detect_lfs_pointers`: `cat-file --batch-all-objects` then one
  `git cat-file blob <oid>` per blob <= 4 KiB (the worktree grep runs first but falls through when
  it finds nothing).
- fix: one `git cat-file --batch` fed the candidate oids (or `--batch-all-objects --batch` with a
  size filter). O(blobs) processes -> 1.

### 7. a depth-1 regex that nginx can't parse past ~95 refs (correctness)
- [measured] any repo with >= 100 public refs fails activation: `nginx -t` says "too long parameter"
  for the `shallow` row of `$ks_git_reply` (knot-static.conf:69). the refs axis stops at activate
  for 100, 1000 and 10000 refs, on both machines.
- [read] `nginx::fetch_map` puts every public tip in one alternation `(?:tip1|tip2|...)\n` inside a
  single map key; nginx caps a config token at 4 KiB, which is ~95 sha1 tips.
- [read] the `have-N` rows have the same shape (one lookahead per tip of a base in one key), so a
  base with ~80+ independent branch tips would hit the same limit.
- fix: leave a row out when its key passes nginx's token limit (that fetch falls back to the full
  pack, which is still right), or split the tip test across chained maps. O(tips) token -> bounded.

### 8. the linux kernel: a cold generate doesn't fit in 6 GiB (canned replies buffered in memory)
- [measured] fenced at 6 GiB with no swap, the cold generate of linux master spent 358 s in
  `export: fetch public refs (cache miss)`, writing a second 6.7 GB copy of the pack into the
  export (fetch over `file://` re-packs and re-indexes every object instead of linking), then
  stalled at the cap in `export: smart artifacts`: the scope held 6.3 GB anon, knot-static 3.7 GB
  and git pack-objects 2.5 GB, during the first full v0 `upload-pack`, reclaiming instead of progressing; i stopped it
  after 564 s wall, 645 cpu-s, peak single-process rss 6.2 GiB. (unfenced, the same step reached
  knot-static 4.4 GiB + pack-objects ~5 GiB and swapped 10 GiB.)
- [read] `git_export::upload_pack` runs git with `Command::output()`, so each canned reply is
  collected in a Vec before it is written: the full v0 pack (6.6 GB for linux) is held in
  knot-static's memory, and `write_smart_artifacts` keeps that Vec alive while
  `write_v2_artifacts` builds `fetch-done`, a second full pack. peak ~2x the repo's pack.
- fix: stream each reply straight into its file (stdout = the staged file), and seed the export
  from the hosted repo's packs by hardlink (`git clone --local`-style or `objects/info/alternates`
  resolved at export time) rather than a `file://` fetch. memory O(pack) -> O(1), cold export
  disk and time O(pack) of copying -> O(1) links + the fsck walk.
- the import path couldn't be measured on the kernel at all (#6), and nothing after the cold
  generate was reached.

## suspects that the numbers refute or bound
- index rebuilt per cob write: [measured] `bootstrap: index rebuild` is 1-2 ms, 16 ms at 200 repos.
- nginx map size vs `nginx -t` / reload: [measured] 6 MB conf (5000 tickets) -> `nginx -t` 0.06-0.12 s,
  reload (SIGHUP until the new rev is served) 0.056 s; 200 repos 952 KB conf -> 0.03 s. linear but small.
  activation itself (tombstone, -t, symlink, records) is <= 0.11 s everywhere.
- repos per knot: [measured] an incremental generate for one of 200 repos is 0.45 s (carried
  repos aren't re-exported); what grows is #4 (atproto 0.26 s) and the conf.
- generate memory: [measured] cold generate peak rss 27 / 86 / 330 MB at 1e3 / 1e4 / 1e5 commits;
  the peaks come from the smart phase (#1, #8): 1.38 GB for a cob-create at 1e5 commits, and
  the kernel doesn't fit in 6 GiB.