# Production Defect Diagnosis: Runner Task Leaks Past Service Quiescence

**Date:** 2026-09-11  
**Investigation:** `service_lifecycle::service_quiesce_preserves_waiting_deadline_and_resumes` intermittent failure  
**Target:** `klbr-daemon --test service_lifecycle`  
**Status:** Confirmed Production Lifecycle Defect  

## 1. Executive Summary

The store owner lock failure (`WouldBlock` at `klbr-runtime/src/store.rs:220`) is only the most visible **symptom**. The defect is that a retired runner outlives quiescence, which contradicts the runtime-v2 handoff requirement to await accepted host tasks before exiting the service.

When a session turn completes normally or is interrupted, `runner_task` is dropped without being joined. Under CPU saturation across multiple cores, the un-awaited task continues running asynchronously on a Tokio worker thread while still holding clones of `SessionRegistry` and `SessionRuntime`. When the test (or service restart caller) re-opens the database, the lingering background task still holds the `_owner: Arc<File>` lock, causing `File::try_lock()` to fail immediately with `WouldBlock`.

## 2. Minimal Reproduction

```bash
CARGO_TARGET_DIR=/tmp/klbr-sub-target cargo test --workspace --locked --no-fail-fast
```

- When run alone (`cargo test -p klbr-daemon --test service_lifecycle`), the suite passes 30/30 runs because on an idle system, Tokio worker threads reap the un-awaited task stack in microseconds before `start_service` re-opens the database.
- Under full workspace parallelism (saturating all 10 cores), thread scheduling delays task cleanup, causing deterministic-looking contention on the second `Store::open`.
- Reproduced across both `service_quiesce_preserves_waiting_deadline_and_resumes` and `explicit_operator_stop_survives_service_quiesce_and_restart`.

## 3. Mechanism & Exact File:Line

1. **Task Spawn:** `klbr-daemon/src/daemon.rs:920` (`trigger_session_turn_with_soul`)
   - Spawns `runner_task = tokio::spawn(async move { ... })`.
   - Captures `reg_clone: Arc<SessionRegistry>` (which holds `Store`) and `entry_clone: Arc<SessionEntry>` (which holds `entry.runtime.store`).
   - Attaches `runner_task`'s `JoinHandle` into `entry.active_runner` via `entry.attach_runner_task(...)`.
2. **Un-Awaited Drop on Retirement:** `klbr-daemon/src/session_registry.rs:248` (`retire_runner`)
   - When the turn finishes, `entry_clone.retire_runner(&op_id).await` runs from *inside* `runner_task`.
   - `retire_runner` executes `*guard = None`, dropping `RunnerRegistration` and its contained `JoinHandle<()>` without awaiting it.
   - Similarly, in `abort_runner` (`session_registry.rs:300`), `guard.take()` drops the registration and handle without awaiting task exit.
3. **Quiescence Bypass:** `klbr-daemon/src/session_registry.rs:779` (`shutdown_all`) / `session_registry.rs:387` (`quiesce`)
   - `quiesce()` takes `self.active_runner.lock().await.take()`.
   - Because `retire_runner` already set `guard = None`, `quiesce` observes `taken = None` and does **not** await `runner_task`.
   - `quiesce_and_join()` completes and returns authority to caller while `runner_task` is still alive on a worker thread.
4. **Contention on Reopen:**
   - **First Opener:** `klbr-daemon/tests/service_lifecycle.rs:146` via `start_service` at `:226`.
   - **Second Opener:** `klbr-daemon/tests/service_lifecycle.rs:146` via `start_service` at `:288`.
   - Line `:288` calls `Store::open(db_path)` which invokes `File::try_lock()` on `<db>.owner.lock` at `klbr-runtime/src/store.rs:220`.
   - Since `runner_task` has not finished unwinding and dropping `reg_clone` / `entry_clone`, `_owner: Arc<File>` has strong count $\ge 1$ and the OS file lock remains held. `try_lock()` returns `WouldBlock`.

## 4. Ruled-Out Hypotheses

- **Hypothesis (B) Cross-Binary Contention:** Ruled out. The test database path is allocated inside a fresh private `tempfile::tempdir()` (`temp.path().join("quiesce.sqlite")`), so no other test binary or daemon process has access to or knowledge of the file.
- **`Release::publish` Contention:** Ruled out. Inspected `Release::publish` (`klbr-runtime/src/behavior.rs:112`); it performs only directory synchronization and manifest parsing. It never opens a SQLite database or `.owner.lock`.
- **Working Tree Drift:** Ruled out. Neither `klbr-runtime/src/store.rs` nor `klbr-daemon/tests/service_lifecycle.rs` has unstaged or staged modifications (`git diff` clean vs HEAD).

## 5. Architectural Recommendation

The honest fix is in **(iii) the caller** (`SessionRegistry` / `SessionEntry` task tracking):
- The daemon must retain and await the spawned `runner_task` until full exit so no runner outlives service quiescence.
- Production `try_lock()` locking semantics at `klbr-runtime/src/store.rs:220` must **not** be weakened to blocking locks; fail-fast single-owner verification is a load-bearing production safety invariant.