4.9 kB 54 lines
413b7bd7
Land the uncommitted tree: the cut campaign, the work under it, and the daemon's window
dawn
⋯
evidence/m3-service-lifecycle-ownerlock.md
sourcerendered
download raw history
at main Markdown

Production Defect Diagnosis: Runner Task Leaks Past Service Quiescence #

Date: 2026-09-11
Investigation: service_lifecycle::service_quiesce_preserves_waiting_deadline_and_resumes intermittent failure
Target: klbr-daemon --test service_lifecycle
Status: Confirmed Production Lifecycle Defect

1. Executive Summary #

The store owner lock failure (WouldBlock at klbr-runtime/src/store.rs:220) is only the most visible symptom. The defect is that a retired runner outlives quiescence, which contradicts the runtime-v2 handoff requirement to await accepted host tasks before exiting the service.

When a session turn completes normally or is interrupted, runner_task is dropped without being joined. Under CPU saturation across multiple cores, the un-awaited task continues running asynchronously on a Tokio worker thread while still holding clones of SessionRegistry and SessionRuntime. When the test (or service restart caller) re-opens the database, the lingering background task still holds the _owner: Arc<File> lock, causing File::try_lock() to fail immediately with WouldBlock.

2. Minimal Reproduction #

CARGO_TARGET_DIR=/tmp/klbr-sub-target cargo test --workspace --locked --no-fail-fast
  • When run alone (cargo test -p klbr-daemon --test service_lifecycle), the suite passes 30/30 runs because on an idle system, Tokio worker threads reap the un-awaited task stack in microseconds before start_service re-opens the database.
  • Under full workspace parallelism (saturating all 10 cores), thread scheduling delays task cleanup, causing deterministic-looking contention on the second Store::open.
  • Reproduced across both service_quiesce_preserves_waiting_deadline_and_resumes and explicit_operator_stop_survives_service_quiesce_and_restart.

3. Mechanism & Exact File:Line #

  1. Task Spawn: klbr-daemon/src/daemon.rs:920 (trigger_session_turn_with_soul)
    • Spawns runner_task = tokio::spawn(async move { ... }).
    • Captures reg_clone: Arc<SessionRegistry> (which holds Store) and entry_clone: Arc<SessionEntry> (which holds entry.runtime.store).
    • Attaches runner_task's JoinHandle into entry.active_runner via entry.attach_runner_task(...).
  2. Un-Awaited Drop on Retirement: klbr-daemon/src/session_registry.rs:248 (retire_runner)
    • When the turn finishes, entry_clone.retire_runner(&op_id).await runs from inside runner_task.
    • retire_runner executes *guard = None, dropping RunnerRegistration and its contained JoinHandle<()> without awaiting it.
    • Similarly, in abort_runner (session_registry.rs:300), guard.take() drops the registration and handle without awaiting task exit.
  3. Quiescence Bypass: klbr-daemon/src/session_registry.rs:779 (shutdown_all) / session_registry.rs:387 (quiesce)
    • quiesce() takes self.active_runner.lock().await.take().
    • Because retire_runner already set guard = None, quiesce observes taken = None and does not await runner_task.
    • quiesce_and_join() completes and returns authority to caller while runner_task is still alive on a worker thread.
  4. Contention on Reopen:
    • First Opener: klbr-daemon/tests/service_lifecycle.rs:146 via start_service at :226.
    • Second Opener: klbr-daemon/tests/service_lifecycle.rs:146 via start_service at :288.
    • Line :288 calls Store::open(db_path) which invokes File::try_lock() on <db>.owner.lock at klbr-runtime/src/store.rs:220.
    • Since runner_task has not finished unwinding and dropping reg_clone / entry_clone, _owner: Arc<File> has strong count $\ge 1$ and the OS file lock remains held. try_lock() returns WouldBlock.

4. Ruled-Out Hypotheses #

  • Hypothesis (B) Cross-Binary Contention: Ruled out. The test database path is allocated inside a fresh private tempfile::tempdir() (temp.path().join("quiesce.sqlite")), so no other test binary or daemon process has access to or knowledge of the file.
  • Release::publish Contention: Ruled out. Inspected Release::publish (klbr-runtime/src/behavior.rs:112); it performs only directory synchronization and manifest parsing. It never opens a SQLite database or .owner.lock.
  • Working Tree Drift: Ruled out. Neither klbr-runtime/src/store.rs nor klbr-daemon/tests/service_lifecycle.rs has unstaged or staged modifications (git diff clean vs HEAD).

5. Architectural Recommendation #

The honest fix is in (iii) the caller (SessionRegistry / SessionEntry task tracking):

  • The daemon must retain and await the spawned runner_task until full exit so no runner outlives service quiescence.
  • Production try_lock() locking semantics at klbr-runtime/src/store.rs:220 must not be weakened to blocking locks; fail-fast single-owner verification is a load-bearing production safety invariant.