# PR performance benchmarks

Each push to an open, vouched PR starts an informational Prime Agent benchmark, including PRs targeting
another branch in a stack. Multiple commits in one push produce one run for the final head. Draft PRs
are included. A two-second debounce and per-PR cancellation avoid finishing obsolete runs.
`workflow_dispatch` reruns an open PR by number.
Identical automatic requests reuse the completed comment when both SHAs, harness, and configuration
match. Manual dispatch and GitHub reruns force fresh measurements. Unvouched authors
receive a pending-trust comment; a maintainer can rerun after vouching.

The controller resolves current `main` and the PR head to full SHAs, builds both in separate Prime
sandboxes, and alternates their measurements. Both use the same trusted harness revision, image
digest and resource allocation. Performance changes are informational, not a regression gate. Failed or
incomplete measurements make the benchmark command fail, even when some metrics succeeded.
The baseline is always current `main`, not the PR's target branch or merge base.

## Enable in GitHub

Add the repository secret `PRIME_SANDBOX_API_KEY`: a dedicated key with Sandbox permissions for
provisioning and cleanup. Optional `PRIME_BENCHMARK_TEAM_ID` selects the sandbox billing workspace.
No inference key, model configuration, or login is required.

The workflows must first land on `main`: both `pull_request_target` and the completion listener run
trusted default-branch code. Trigger `Prime Agent benchmarks` manually with an open PR number after
configuring the sandbox secret. The first rollout should include a main-versus-main calibration.

The request checks out the event's `GITHUB_SHA`, which GitHub resolves to the default-branch commit for
[`pull_request_target`](https://github.blog/changelog/2025-11-07-actions-pull_request_target-and-environment-branch-protections-changes/),
including stacked PRs. The benchmark job uses that exact harness SHA. Manual dispatch is allowed only
from `main`.

No credentials are injected into the sandboxes. The sandbox control key and GitHub token stay on the
trusted controller; a separate publisher owns GitHub comment write permission. No PR checkout,
build script, installer, executable, or archive is executed/extracted on a privileged GitHub runner.
The existing Vouch gate controls who can trigger compute usage.

## Measurements

The default configuration uses two Linux x64 containers, each with 4 vCPU, 8 GB RAM, and 20 GB disk.
`config.json` pins the image and sampling policy. Harness dependencies are locked with a seven-day
release cutoff. Provisioning, harness setup, and source compilation have separate recorded durations
outside the timed installation interval. Interactive runs use the same small committed Git fixture.

The base image supplies Node, npm, Git, curl, and Python. Untimed sandbox setup adds Python venv
support, Cairo/Pango/JPEG/GIF/SVG development libraries, ripgrep, and fd. These prerequisites are
shared within each sandbox; the normal installer still prepares each user's stock Python environment
and other missing tools. Their setup time and disk usage are outside the installation metrics.

- **Cold startup:** process launch until the editor visibly echoes a typed marker, after stopping all
  processes owned by the benchmark user. OS filesystem caches are not flushed.
- **Warm startup:** the same input-ready measurement while retaining the daemon and stopping its
  previous active sessions. Each TUI launch opens a fresh conversation.
- **Installation:** the normal installer and Python/tool bootstrap in three new user homes, each
  with empty npm and uv caches. Unpublished candidate release tarballs are served over loopback;
  npm/Python dependencies use the real network. This does not measure public release-CDN latency.
- **Compressed artifacts:** total bytes of the four npm tarballs and, for revisions with compiled
  release support, the Linux x64 archive. Both sides use the same synthetic version and download
  origin. This includes the npm fallback packages, but excludes other platforms and external registry
  dependencies. The result records which formats were built.
- **Installed footprint:** apparent bytes added after first use in the first fresh home, including stock Python,
  runtime, and tool assets; excluding download caches, session history, and logs. Shared system
  dependencies supplied by the base image and the fixture repository are excluded.
- **Idle memory:** summed RSS across the benchmark user's entire process tree after input readiness
  and a one-second settle. Raw results include each process and PSS when Linux permits reading it.
  The controller, PTY harness, artifact server, and build user are excluded. RSS can double-count
  shared pages.

Startup and memory use 10 trials per revision. Installation uses three; sizes are measured once. UI interactions use 3 trials per revision (`ui_trials`).
Compiled revisions provision pinned Bun tooling and build their Linux x64 archive during untimed
setup. The installer selects its normal default from the available artifacts; the harness does not
force Node or compiled mode. Loopback downloads use the installer's explicit test exception, while
external downloads retain normal HTTPS checks. A compiled candidate that falls back to Node is
reported as a failed installation, rather than measuring the wrong runtime. Harness changes must land on `main` before CI uses
them, including when benchmarking the Bun migration stack.
Stock tools, skills, daemon, persistence, and Python bootstrap remain enabled. Homes contain no
credentials, extensions, MCP servers, or personal skills. The onboarding splash is marked as already
shown before timing, so startup measures the editor rather than waiting for a person to sign in.
The editor runs without a selected model; no prompts are submitted. Readiness requires a typed probe
to appear in the terminal and be removed without Enter. Process creation and terminal input mode are
not proof of editor readiness. Navigation labels, placeholders, and model names are not readiness
signals. Both revisions use the same probe and launch-to-render timing; probe cleanup is untimed.
Dropped input is retried every 20 ms, adding up to one retry interval plus terminal polling/rendering
delay when the input handler starts after terminal raw mode. There is no fixed delay on the ready path.
These measurements cover the local startup path, not authenticated provider discovery or inference.

### Python runtime

A separate probe drives the **installed** `python -m rlm.repl` over its JSONL protocol. It measures the
CPython kernel and stock `rlm.bash` implementation, including IPC, serialization, and output capture.
It does not include the TypeScript kernel manager, TUI rendering, or model execution. The harness
fails visibly if either revision does not support the required runtime protocol; it never substitutes
a plain Python process or omits a failed metric.

Each of the 10 trials starts with all benchmark-user processes stopped and a new kernel:

- **Kernel startup:** process spawn through the protocol-ready handshake. Includes the small
  `runuser` launch overhead; excludes Python installation and host-side agent bootstrap.
- **Python cell round trip:** mean of 50 sequential `pass` cells through the complete request/done
  protocol. Five Python cells and one empty bash command warm the runtime before execution timings.
- **Empty bash command:** mean of five `await bash(':')` calls with successful, empty results.
- **Bash git status:** mean of five `git status --porcelain` calls in the fixed, clean Git fixture.
- **Bash 32 KiB output:** mean of five shell commands producing exactly 32,768 bytes. The runtime
  captures and forwards the entire output, and the probe verifies its byte count.
- **35 cells / 9 shell calls:** total time for a fixed mix of 26 Python no-ops and nine git-status
  calls, alternating one shell call with three Python cells.
- **Interrupt to done:** after an executing cell signals readiness, send an interrupt and wait for
  its KeyboardInterrupt/done acknowledgement. A subsequent cell must still execute successfully.
- **State snapshot:** serialize a 10,000-row, eight-column integer pandas DataFrame and a
  10,000-integer list to disk. Includes first-use serialization imports.
- **State restore:** restore that state into a fresh kernel, including first-use pandas imports but
  excluding kernel startup. Verify the recovered dimensions, sum, and list values after timing.
- **Kernel idle RSS:** memory immediately after the ready handshake, before workload imports.
- **Kernel RSS after pandas workload:** memory after the mixed calls and construction of the same
  DataFrame/list used for snapshotting. Both memory measurements include only the isolated user's
  Python process; raw per-process measurements are retained.

The runtime rows show medians across the 10 independent trials. Batch iteration counts and fixture
sizes are fixed in `kernel.py`; failures and incomplete output remain failed samples. These are
subsystem measurements alongside the normal installed CLI benchmarks, not end-to-end tool latency.

The comment has four columns: metric, main, PR, and signed change with the percentage in parentheses.
A single summary counts regressions, improvements, metrics with no clear change, and incomplete or
unavailable comparisons. Improvements are green with `↓`, regressions red with `↑`, and changes within
the noise threshold remain neutral with `≈`. The entire change value, including the arrow, delta, and
percentage, goes from muted to vivid as the absolute percentage grows, reaching maximum intensity at
100%. An undefined percentage uses the muted shade. Colors use GitHub's native MathJax rendering;
no external badge service is required. Successful/attempted counts and spread appear in the collapsed
methodology.
Arrows require a change larger than the metric's absolute floor, relative floor,
and observed spread. The initial relative floor is 20% for timings and memory, 0.5% for artifact
size, and 1% for disk footprint. A same-revision calibration on separate sandboxes showed roughly
7–18% variation across several timings; the conservative floor avoids labeling that as a code regression.
Smaller signed differences remain visible. Revisit these floors after collecting more control runs. This is a practical noise filter, not a statistical significance test. Inspect
raw trials before acting on small changes; sandbox scheduling and filesystem caches still introduce noise.

### UI interactions

A separate probe drives the interactive TUI over a PTY with the same `pexpect`/`pyte` harness the
startup metrics use, against a deterministic on-disk fixture set. Every UI trial regenerates the
sessions directory, session artifacts, and spawn ledger from scratch, so trials cannot inherit state.

The fixture set is fixed and mirrors a long-lived install: 194 top-level sessions (3 large —
one of them a very large ~40 MB transcript — 150 medium, 40 small fan-out children of the chain
root, and the root itself) plus a chain of 6 nested subagent sessions. Large sessions hold 1,999
user/assistant/toolResult triples (~5 MB of JSONL each), the very large one 15,999 (~40 MB),
medium sessions 119, and each subagent 399. Content is synthetic but shaped like real transcripts
(thinking, tool calls, tool output). The subagent chain and the fan-out are linked through the
same spawn-ledger records the daemon replays (~46 edges), so roster hydration, catalog scanning,
and saved-session paths are exercised at a scale where regressions are visible. No inference is
called at any point.

Each trial stops all benchmark-user processes first, then measures:

- **Resume large session (cold):** `prime-agent --resume <id>` from process spawn until the resumed
  transcript tail is rendered and the editor echoes a typed marker. This is the "switch into a large
  session" path including daemon and session-worker startup.
- **Switch into large session:** in a running TUI, `/resume <id>` keystroke to rendered tail plus
  echoed marker. Measures the in-place session switch with a ~40 MB transcript.
- **Open agents view from a session:** the left-arrow keystroke to the rendered agents-view splash.
- **Full agents roster, many sessions:** after the agents-view splash is rendered, until the
  roster lists the fixture's saved sessions and its count stops growing. Saved sessions stream in, so this isolates
  catalog scanning, spawn-ledger replay, and hydration across 200 session files.
- **Open another session from agents view:** typing the target's unique session-id prefix,
  right-arrow to open, until the target transcript renders and echoes. This is the full
  "session → agents view → another session" round trip with many sessions on disk.
- **Reopen resident large session:** return to agents view and select the large session just opened.
  Time right-arrow through transcript tail and editor echo, excluding search setup and probe cleanup.
  This exercises attach-snapshot reuse after model-catalog refresh (#2296), separately from cold
  worker creation. Readiness retries the editor probe every 20 ms.
- **Open chain parent from agents view:** back to the agents view, search the subagent chain's
  root session by id prefix, right-arrow to open. The parent becomes live.
- **Open subagent session at depth 6:** back to the agents view, then drill into the live chain:
  filter by the deepest subagent's id prefix, clear the search (expansion needs an empty query),
  expand the parent, refilter past the fan-out siblings, and repeat clear-expand-step for each of
  the six levels until the deepest subagent row is selected, then right-arrow to open. This is the
  exact manual flow the product requires today, so the metric includes its real cost.
- **CPU per interaction:** summed `utime+stime` across the whole benchmark-user process tree
  (TUI client, daemon, session workers, Python kernels) between two instants around each interaction.
- **UI memory after interactions:** summed RSS across the same tree after the scenario settles.
  Per-process snapshots with PSS and CPU are saved as raw artifacts, as for idle memory.

Readiness is never a spinner or a status line: a trial only counts when the target session's
final transcript line is visible and the editor echoes a marker, so half-rendered states fail
loudly instead of measuring fast. The roster wait likewise only settles once the fixture's saved
sessions are listed, so a stalled or empty catalog scan fails the trial instead of recording a
fast sample. Raw results also record PTY bytes per interaction, a proxy for
how much the renderer redraws. UI trials default to 3 per revision (`ui_trials` in `config.json`)
to bound sandbox cost; the scenario takes roughly 60–90 s per trial.

### Scheduled catalog and cold workers

After the navigation scenario, the UI trial stops its benchmark-user processes and regenerates
an independent 2,300-session fixture: 2,298 children linked through real ledger edges, their root,
and an unrelated two-message cold target. Each non-target transcript contains 64 assistant usage
entries (147,136 in total), exceeding the 100,000-entry metadata-cache budget. Thirteen children
own scheduled-job artifacts. All schedules are paused, have no next run, and must retain runCount=0.

The installed daemon runs on a dedicated socket. The probe requires protocol 7 and the negotiated
heartbeat_catalog capability; unsupported revisions fail visibly. It records:

- **Scheduled catalog, first request** and **repeated request:** complete global heartbeats_list
  round trips, with process-tree CPU. The daemon may already have scanned schedules during startup;
  the first request does not claim a cold filesystem or empty metadata cache.
- **Cold worker with three catalog scans:** pipeline three global catalog requests, allow 20 ms
  for their handlers to enter, then create the unrelated saved session. Time the create request
  through a matching ready worker summary. The target must not already be resident. This isolates
  worker readiness under scan traffic (#2299), without transcript rendering or PTY delays.

Every catalog reply must contain exactly the expected 13 jobs, with owner names and first-message
metadata. Competing scans must also complete successfully. Missing jobs, dropped metadata, failed
scans, and wrong/not-ready workers cannot count as improvements. Raw results record how many scan
replies arrived before worker readiness; this is an observed response order, not proof of which
internal scan was executing at each instant. CPU for the cold-worker interval includes scan work.
No schedules execute and no prompts are submitted. Scans remain independent of the ordinary UI
fixture so the added history cannot change the original navigation measurements. Allow additional
fixture-generation and five scan round trips per UI trial; full Linux calibration is still required.

## Lifecycle and costs

The main workflow posts a single marked comment and updates it in place. A new push replaces the
previous table with a short running notice and a link to the latest run. The completion workflow
fills that same comment with the latest results or a cancellation/failure notice. Previous results
remain available in workflow artifacts. Publishers serialize by PR and check the
current head, latest run ID, attempt, and existing comment generation before writing. A rerun of an
older commit cannot overwrite a newer result, including when the newer run has the same head SHA.

Results and logs are retained as GitHub artifacts for 14 days. They include exact source and harness
SHAs, environment details, installed dependency inventory, transcripts, raw observations, failures,
and estimated sandbox costs. The publisher validates schema, identity, finite numbers, and report
size and escapes sandbox-provided text.

The controller stops scheduling work at 25 minutes or its $1 estimated budget target. Every sandbox
has a 30-minute TTL. Teardown runs in `finally`; the independent completion workflow deletes any
remaining sandboxes with the exact repository/run/attempt labels, including after cancellation.
Cleanup enumerates all pages before deleting so pagination cannot skip a sandbox.

By default, two consecutive identical failures stop further trials for that revision and phase
(`failure_limit` in the configuration). Other independent phases can continue. A failed cold readiness
probe skips warm startup because there is no verified cold session to retain; a failed first installation
skips dependent interactive and runtime trials.
Skipped trials are not measurements or successful samples. Failure causes and terminal transcripts
remain in the result artifacts, which are uploaded even when the benchmark command fails.
Log-collection and deferred sandbox-cleanup problems appear as operational warnings. They do not
mark complete measurements as partial; missing or failed measurements still fail the command.

At the configured list rates, two sandboxes cost about $0.01/minute together. A 10-minute run costs
about $0.10 in sandbox compute. There are no inference charges. UI-interaction trials add roughly
3–6 minutes per run at the default 3 trials per revision. Full sandbox lifetimes are recorded,
including provisioning, setup, building, measurements, and cleanup. The $1 target is an estimate,
not a hard billing cap; scheduling and deletion latency can increase the final bill.

## Local development

The existing runtime Python CI job runs the harness checks. To run them locally from this directory:

```sh
uv sync --locked
uv run --locked ruff check .
uv run --locked ruff format --check .
uv run --locked python -m unittest discover -s tests -v
```

Tests use fake GitHub/SDK responses and terminal streams; they never invoke inference or provision
sandboxes. A live run is separate and requires only the `PRIME_SANDBOX_API_KEY` environment variable:

```sh
uv run --locked cli.py local --base FULL_MAIN_SHA --head FULL_HEAD_SHA --results results/local
```

Use identical SHAs to calibrate main against itself. `--config /path/to/config.json` can reduce trials
for a smoke run; the report records the effective configuration. Local mode never posts GitHub
comments. `results/`, `.venv/`, and `.ruff_cache/` are ignored. Run the repository's `npm run check`
after code changes as well.