11 kB 148 lines
ed964db8
add scale bench results for the mac, chernobog and the linux kernel
dawn
⋯
bench/results/findings.md
sourcerendered
download raw history
at main Markdown

knot-static scale bench: findings (layer 1, end to end) #

harness: benches/scale/ (cargo bench --bench scale, env in its header). raw rows and tables:

  • macos-small.{jsonl,md}: this mac (m-series, apfs), small preset, nginx 1.30.6, release binary.
  • linux-big.{jsonl,md}: chernobog (12 cores, ext4, nixos), big preset, nix build .#knot-static, nginx 1.30.5.
  • linux-kernel.{jsonl,md}: chernobog, torvalds/linux master (12M objects, 6.6 GB pack).

chernobog is a desktop: both chernobog runs were taken inside systemd-run --user --scope -p MemoryMax=6G -p MemorySwapMax=0 -p CPUWeight=20 -p IOWeight=20 -- nice -n 19, so their wall times include yielding to the desktop; cpu and rss don't. an earlier unfenced kernel push swapped the box and isn't used for any number here.

every axis is varied alone from the base point (100 commits, 4 refs, no tickets, 1 repo). sequence per point: import, cold generate, activate, nginx -t, a ref moved in the hosted repo then generate --repo + activate + reload, a push over git-ssh, a createRecord ticket, a putRecord on ticket 1. seeding is outside the measurement: history and refs by one git fast-import, cobs signed in-process with knot2's CobStore through tests/common/cobs.rs (5000 tickets in 14 s).

how to read the tags: [measured] = numbers from these runs; [read] = from reading the code; [timers] = from temporary [trace] timers inside write_smart_artifacts (not committed, run on the mac commits axis), added because the smart phase is one opaque 10-step span.

bench artefacts, not knot-static costs: activate and the owner write cli poll https://<hostname> for up to 10 s before they return (crawl::announce), and a push waits up to 2 s in spindle::tell (wait_served, even with no spindle). the bench knot never serves its own hostname, so those are fixed 10 s / 2 s floors here: writes are timed until they answer (answered s) and stopped there; activation's own clock is in the jsonl (phases.activation_s).

hot spots, worst scaling first #

1. canned fetch replies are rebuilt whole on every generation, by a packer 10x slower than git #

  • [measured] push wall at 1e3 / 1e4 / 1e5 commits (chernobog): 0.16 / 0.98 / 17.7 s of export, of which smart artifacts 0.13 / 0.92 / 17.2 s. bytes written per push 1.25 / 13 / 177 MB: the full upload-pack + fetch-done + shallow + shallow-head are rewritten (changed) every time, 2.6x the repo's pack per push.
  • [measured] with cobs in the repo it gets much worse: cob-put at 1e4 commits 2.6 s / 16 cpu-s, at 1e5 commits 45 s, 292 cpu-s, 1.33 GB rss (cob-create 1.38 GB; a push before any cob: 20 s, 463 MB).
  • [timers] at 1e4 commits (mac): git's full v0 and v2 packs 0.11 s each; knot-pack's budget full pack 1.1 s (2.0 s once cobs exist) and it is only used for its length; each have-N / have-N-done reply 20 ms without cobs, ~0.95 s with cobs (the cob tips aren't reachable from a base, so knot-pack walks all history per reply); 3 bases x 2 replies.
  • [read] git_export::write_fetch_replies: budget = reply(full want, done).len() then 2 knot-pack runs per candidate base (up to 8), all from scratch; write_smart_artifacts clears smart/ first.
  • fix: size the budget from git's fetch-done (already built) instead of a knot-pack run; build have-N with git upload-pack (same request bytes) or fix knot-pack's reachability walk to stop at the haves; key every canned reply by a hash of its request and the refs it reads and hardlink it from the previous release when unchanged (cob-put already leaves upload-pack identical, 4.9 MB rewritten for nothing). per push: O(history) knot-pack walks x (1 + 2 bases) -> one O(history) git pack (or O(new objects) with pack reuse) + O(new objects) per base.

2. a ref fetch per 100 refs: export is O(refs^2) #

  • [measured] cold generate fetch phase vs refs (chernobog): 1e2 0.04 s, 1e3 0.61 s, 1e4 28.6 s (cold generate 51 s, 21 s of it smart artifacts). cob refs count too: 5000 tickets -> 9.7 s cold fetch, and every push/write at 5000 tickets spends ~7.5 s of untraced export in the delta fetch (git_export 8.8 s, traced parts 1.3 s).
  • [read] git_export::fetch_refs runs one git fetch per 100 refspecs; each one makes the source advertise all R refs and negotiate: R/100 x O(R).
  • fix: one git fetch --stdin with every refspec (or fetch refs/* and drop what isn't exported). O(R^2/100) -> O(R).

3. a new cob (or any new ref) rebuilds the export from scratch #

  • [measured] cob-create vs cob-put: createRecord takes the cache-miss path (fetch all, prune, fsck) and writes a whole new objects pack each time: at 1e5 commits 62 s and 270 MB written vs 45 s / 177 MB for putRecord; at 1e4 commits +6.3 MB new pack per ticket.
  • [read] refs_fast_forwarded returns false when a ref is missing from the prior export, so an added ref (every new ticket, comment, pull, branch, tag) can never take the delta path.
  • fix: treat added refs as fast-forwards from nothing (fetch only their objects onto the hardlinked prior entry); keep the full path for deletions and rewrites. O(history) -> O(new objects).

4. every write re-reads every cob and rewrites every record file #

  • [measured] the build's unreported remainder ("atproto" column: cob state, chain settle, record and blob files) per push/write: tickets 100 / 1000 / 5000 -> 0.07 / 0.60 / 3.0 s; versions 10 / 100 (10 tickets) -> 0.03 / 0.21 s; repos 50 / 200 -> 0.07 / 0.26 s. a push at 5000 tickets is 14.6 s in all. a git-only push at 1000 tickets rewrites 1.35 MB of byte-identical atproto files (lists 0.84, versions 0.30, records 0.21 MB) and changes the 1.23 MB records map and the 0.47 MB .complete.json; at 5000 tickets 6.8 MB identical, records map 6.2 MB, .complete.json 2.3 MB. each have-N reply also carries every cob (0.56 MB at 1000 tickets, 2.8 MB at 5000), since cob tips are never in a base.
  • [read] atproto::cob_state replays changes_of for every social entry of the captured repo; read_back/whole_tree walks the whole chain of every repo, carried ones included; emit writes every repo's car, record, list and version file into the new generation without linking the unchanged ones; .complete.json inventories every file.
  • fix: cache each cob's state by its ref oid in the repo store and settle only changed keys; emit content-addressed record/list/version files and hardlink them from the previous generation; carry a carried repo's atproto dir like its git mount. per write O(tickets x versions + all repos' records) -> O(changed cobs).

5. old generations, releases and export entries are never removed #

  • [measured] base knot: 0.8 MB after the first generate, +0.6 MB per write, 5.8 MB after 9 generations; 1e4 commits 25.9 -> 85.7 MB after 4 operations; 1e5 commits 363 MB -> 1.17 GB; 5000 tickets 258 -> 538 MB. linux kernel: see below.
  • [read] nothing under generations/, repos/*/releases or the export store is ever deleted (README says so).
  • fix: after activation, keep the active generation, the previous K for rollback, and the releases/store entries they reference; delete the rest. disk O(generations x repo) -> O(K x repo).

6. import scans every small blob with its own git process #

  • [measured] import of a bare repo: 1e3 / 1e4 / 1e5 commits = 1.5 / 15 / 154 s on chernobog, 10 / 102 s on the mac, cpu almost all in git spawns. the kernel (millions of small blobs) would take hours, so the linux run hardlinks the clone's pack into a created repo instead.
  • [read] provision::detect_lfs_pointers: cat-file --batch-all-objects then one git cat-file blob <oid> per blob <= 4 KiB (the worktree grep runs first but falls through when it finds nothing).
  • fix: one git cat-file --batch fed the candidate oids (or --batch-all-objects --batch with a size filter). O(blobs) processes -> 1.

7. a depth-1 regex that nginx can't parse past ~95 refs (correctness) #

  • [measured] any repo with >= 100 public refs fails activation: nginx -t says "too long parameter" for the shallow row of $ks_git_reply (knot-static.conf:69). the refs axis stops at activate for 100, 1000 and 10000 refs, on both machines.
  • [read] nginx::fetch_map puts every public tip in one alternation (?:tip1|tip2|...)\n inside a single map key; nginx caps a config token at 4 KiB, which is ~95 sha1 tips.
  • [read] the have-N rows have the same shape (one lookahead per tip of a base in one key), so a base with ~80+ independent branch tips would hit the same limit.
  • fix: leave a row out when its key passes nginx's token limit (that fetch falls back to the full pack, which is still right), or split the tip test across chained maps. O(tips) token -> bounded.

8. the linux kernel: a cold generate doesn't fit in 6 GiB (canned replies buffered in memory) #

  • [measured] fenced at 6 GiB with no swap, the cold generate of linux master spent 358 s in export: fetch public refs (cache miss), writing a second 6.7 GB copy of the pack into the export (fetch over file:// re-packs and re-indexes every object instead of linking), then stalled at the cap in export: smart artifacts: the scope held 6.3 GB anon, knot-static 3.7 GB and git pack-objects 2.5 GB, during the first full v0 upload-pack, reclaiming instead of progressing; i stopped it after 564 s wall, 645 cpu-s, peak single-process rss 6.2 GiB. (unfenced, the same step reached knot-static 4.4 GiB + pack-objects ~5 GiB and swapped 10 GiB.)
  • [read] git_export::upload_pack runs git with Command::output(), so each canned reply is collected in a Vec before it is written: the full v0 pack (6.6 GB for linux) is held in knot-static's memory, and write_smart_artifacts keeps that Vec alive while write_v2_artifacts builds fetch-done, a second full pack. peak ~2x the repo's pack.
  • fix: stream each reply straight into its file (stdout = the staged file), and seed the export from the hosted repo's packs by hardlink (git clone --local-style or objects/info/alternates resolved at export time) rather than a file:// fetch. memory O(pack) -> O(1), cold export disk and time O(pack) of copying -> O(1) links + the fsck walk.
  • the import path couldn't be measured on the kernel at all (#6), and nothing after the cold generate was reached.

suspects that the numbers refute or bound #

  • index rebuilt per cob write: [measured] bootstrap: index rebuild is 1-2 ms, 16 ms at 200 repos.
  • nginx map size vs nginx -t / reload: [measured] 6 MB conf (5000 tickets) -> nginx -t 0.06-0.12 s, reload (SIGHUP until the new rev is served) 0.056 s; 200 repos 952 KB conf -> 0.03 s. linear but small. activation itself (tombstone, -t, symlink, records) is <= 0.11 s everywhere.
  • repos per knot: [measured] an incremental generate for one of 200 repos is 0.45 s (carried repos aren't re-exported); what grows is #4 (atproto 0.26 s) and the conf.
  • generate memory: [measured] cold generate peak rss 27 / 86 / 330 MB at 1e3 / 1e4 / 1e5 commits; the peaks come from the smart phase (#1, #8): 1.38 GB for a cob-create at 1e5 commits, and the kernel doesn't fit in 6 GiB.