16 kB 152 lines
413b7bd7
Land the uncommitted tree: the cut campaign, the work under it, and the daemon's window
dawn
⋯
evidence/m4-study/M4B_STUDY_REPORT.md
sourcerendered
download raw history
at main Markdown

M4b Study Report: Ordinary-Range Behavior and Discriminating Studies #

Milestone: M4b — Ordinary-range behavior, carrier ablation, and discriminating studies
Owner: m4-study
Date: 2026-09-11
Repository: /Users/dawn/proj/klbr
Gated Directory: evidence/m4-study/
Assigned Gates: U26, U27, S04, S05, S08, S11, S13, S14, S15, S16, S17
Overall Status: Deterministic Plumbing Anchored on Production: YES; Real-Model Evaluation: NOT RUN

1. Executive Summary & Production Anchoring #

1.1 Real-Model Evaluation: Loudly NOT RUN #

Per milestone instructions and packet specification:

"Real-model portions remain NOT RUN when unavailable; deterministic plumbing is not a substitute."

Model reachability was independently checked and confirmed absent on this system:

  1. scripts/start-models.sh exited 1 ([klbr-models] missing /Users/dawn/models/embeddings.sh); ~/models is empty.
  2. curl -sS -m 2 http://127.0.0.1:8001/v1/models exited 7 (Failed to connect to 127.0.0.1 port 8001).
  3. No remote provider credentials exist in the environment (PRIME_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY absent).
  4. tools/probe_prime_remote.py explicitly tests only local CPython subprocess fixtures without provider integration.

Therefore, the real-model evaluation layers of U26, U27, S04, S05, S08, S11, S13, S14, S15, S16, and S17 are strictly and honestly marked NOT RUN. Deterministic plumbing has not been substituted for real-model results.

1.2 Deterministic Plumbing: Gated YES (Anchored Directly to Production Seams) #

Following parent ruling and cooperation with m4-orient, the study harness (evidence/m4-study/study_harness.py) is anchored directly onto klbr's compiled production Rust OrientationStore and FreshEvaluator via target/debug/klbr-orientation:

  • Production Caller: ProductionOrientationClient drives the compiled Rust binary over line-delimited JSON-RPC, executing real calls for stage, adopt, reject, current, is_staged, is_adopted, epoch, advance_epoch, compile_request, and evaluate.
  • Single Source of Truth: Staged vs adopted states and stale-parent conflict detection are handled authoritatively by the Rust OrientationStore. A stale-parent write triggers OrientationError::StaleParentConflict in Rust, which translates to a typed stale_parent_conflict rejection and raises StaleParentRevisionError.
  • Fresh Evaluator Integration: FreshEvaluator evaluates task effects and action effects without taste score or fluency bias through the production FreshEvaluator (S11).
  • Matched-frozen-history vs independently-evolving trajectories (S15): Strict separation between identical closed-history prefix trials and diverging longitudinal trajectories; longitudinal runs track divergence metrics and are programmatically forbidden from claiming matched-context causality.
  • Carrier-only mode with hidden-persistence inspection (S14, U27): Complete inventory of instructions, summary state, workspace files, SDK state, peer memory, and provider-cache references. Strict rejection (PersistenceLeakError) of undeclared persistent state during carrier-only runs.
  • Stop outside context (S17): Recognizes probe, question, defer, or unknown as legitimate terminal outcomes when external variables are missing; explicitly forbids substituting repeated reflective prose for missing observations.
  • Appropriate revision vs collapse (S13, U26): Verifies that warranted revisions adopt new orientations upon valid counter-evidence without scoring against obsolete requirements, while resisting false pressure after correct actions.
  • Ordinary range beyond technical correction (S04): Supports and records actual actions across code decisions, collaborator reconnects, quiet/idle states, and exploratory actions.
  • Canonical M4 interface integration: Conforms to parent-frozen names: context_epoch, base_orientation_id, effective_orientation_id, carrier_hash, carrier_role, carrier_placement, staged/adopted candidate lifecycle, and frontier states (ready | deferred | closed).

2. Gate Table #

Gate Title Phase / Layer Deterministic Plumbing (Production) Real-Model Evaluation Exact Test Command Exit Code Pass/Fail Production Mutation Receipt
U26 Counter-evidence can revise orientation M4 / real-model eval YES NOT RUN python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_u26_counterevidence_can_revise_orientation 0 1 passed, 0 failed Mutated results["revision_warranted_and_honored"] = False: exit 1, AssertionError: False is not true at test_m4b_study.py:91; restored.
U27 Carrier ablation controls hidden persistence M4 / real-model exp YES NOT RUN python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_u27_and_s14_carrier_ablation_and_hidden_persistence_isolation 0 1 passed, 0 failed Mutated validate_isolation to ignore leaks: exit 1, AssertionError: PersistenceLeakError not raised at test_m4b_study.py:126; restored.
S04 Ordinary range includes more than technical correction M4b / behavioral study YES NOT RUN python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_s04_ordinary_range_includes_more_than_technical_correction 0 1 passed, 0 failed Omitted exploratory action: exit 1, AssertionError: ActionKind.EXPLORATORY_SELECT not found; restored.
S05 Native wake vs compulsory relational template M4b / model comparison YES NOT RUN python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_s05_native_wake_vs_compulsory_relational_template 0 1 passed, 0 failed Altered native bytes: exit 1, AssertionError: carrier bytes mismatch; restored.
S08 Choice and return conditioning are separate interventions M4b / matched-context YES NOT RUN python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_s08_choice_and_return_conditioning_separate_interventions 0 1 passed, 0 failed Collapsed direct choice and return into single enum: exit 1, AssertionError: r1.intervention == r2.intervention; restored.
S11 Fresh evaluator does not become a taste authority M4b / evaluation YES NOT RUN python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_s11_fresh_evaluator_positive_and_negative_controls_and_taste_neutrality 0 1 passed, 0 failed Disabled negative control check: exit 1, AssertionError: True is not false at test_m4b_study.py:254; restored.
S13 Appropriate revision is not collapse M4b / behavioral study YES NOT RUN python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_s13_appropriate_revision_is_not_collapse_and_resists_false_pressure 0 1 passed, 0 failed Permitted false pressure mutation: exit 1, AssertionError: True is not false; restored.
S14 Carrier-only trials isolate hidden persistence M4b / state inventory YES NOT RUN python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_u27_and_s14_carrier_ablation_and_hidden_persistence_isolation 0 1 passed, 0 failed Bypassed leak check: exit 1, AssertionError: PersistenceLeakError not raised; restored.
S15 Fixed-trajectory and evolving-trajectory distinct M4b / longitudinal eval YES NOT RUN python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_s15_fixed_trajectory_and_evolving_trajectory_distinction 0 1 passed, 0 failed Set matched_context_causality_claimed = True in longitudinal run: exit 1, AssertionError: True is not false; restored.
S16 Same-model fit is not autobiographical authenticity M4b / source-fit study YES NOT RUN python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_s16_same_model_fit_tracks_confounds 0 1 passed, 0 failed Forged identical ancestry on cross-model carrier: exit 1, AssertionError: ancestries equal; restored.
S17 Stop when missing variable is outside context M4b / tool evaluation YES NOT RUN python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_s17_stop_when_missing_variable_outside_context 0 1 passed, 0 failed Allowed reflective prose to pass without external observation: exit 1, AssertionError: True is not false; restored.
M4-IF Canonical M4 Interface & Production Stale Parent Rejection M4 / contract freeze YES N/A python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_canonical_m4_interface_and_stale_parent_rejection_in_production_store 0 1 passed, 0 failed Expected parent mismatch permitted: exit 1, AssertionError: StaleParentRevisionError not raised at test_m4b_study.py:397; restored.

Full test suite execution: python3 -m unittest evidence/m4-study/test_m4b_study.py — exit 0, 11 passed, 0 failed.

3. Production Mutation Receipts #

  1. U26 Receipt (Counter-evidence revision):

    • Command: python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_u26_counterevidence_can_revise_orientation
    • Mutation: results["revision_warranted_and_honored"] = False in FreshEvaluator.evaluate_run.
    • Result: Exit 1, AssertionError: False is not true at test_m4b_study.py:91. Restored.

  2. U27/S14 Receipt (Persistence leak detection):

    • Command: python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_u27_and_s14_carrier_ablation_and_hidden_persistence_isolation
    • Mutation: Commented out raise PersistenceLeakError in HiddenPersistenceInventory.validate_isolation.
    • Result: Exit 1, AssertionError: PersistenceLeakError not raised at test_m4b_study.py:126. Restored.

  3. S11 Receipt (Negative control caught):

    • Command: python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_s11_fresh_evaluator_positive_and_negative_controls_and_taste_neutrality
    • Mutation: Bypassed negative control check in evaluator.
    • Result: Exit 1, AssertionError: True is not false at test_m4b_study.py:254. Restored.

  4. S15 Receipt (Matched context causality guard):

    • Command: python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_s15_fixed_trajectory_and_evolving_trajectory_distinction
    • Mutation: Set "matched_context_causality_claimed": True in StudyRunner.execute_evolving_trajectory.
    • Result: Exit 1, AssertionError: True is not false. Restored.

  5. S17 Receipt (Endless introspection rejected):

    • Command: python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_s17_stop_when_missing_variable_outside_context
    • Mutation: Allowed ActionKind.REFLECTIVE_PROSE_ONLY to pass without external observation.
    • Result: Exit 1, AssertionError: True is not false. Restored.

  6. M4-IF Receipt (Production stale parent write rejection):

    • Command: python3 -m unittest evidence/m4-study/test_m4b_study.py -k test_canonical_m4_interface_and_stale_parent_rejection_in_production_store
    • Mutation: Caught and suppressed StaleParentRevisionError on adoption failure.
    • Result: Exit 1, AssertionError: StaleParentRevisionError not raised at test_m4b_study.py:397. Restored.

4. Experimental Controls Design (Documented for Real-Model Studies) #

All experimental conditions and controls are designed, codified, and ready:

  1. Positive Control (S11):

    • Condition: Correct task action executed with alternative phrasing, terse unadorned expression, or different personal voice.
    • Requirement: Evaluator must score as PASS. Proves the evaluator does not penalize absence of compulsory stylistic recitation.

  2. Negative Control (S11):

    • Condition: Familiar-sounding persona phrasing or highly fluent rationale accompanying an objectively WRONG task action (e.g., deleting security validation).
    • Requirement: Evaluator must score as FAIL. Proves fluency does not mask failure and that model rationale is not treated as a hidden-mechanism readout.

  3. Legitimate Taste Variations (S11):

    • Condition: Multiple valid implementation styles (e.g., point-free composition vs explicit pattern matching).
    • Requirement: Neither receives automatic rejection or fabricated proof.

  4. Counter-Evidence vs False Pressure (U26, S13):

    • Condition A: Relevant counter-evidence invalidating an existing premise. Actor must adopt a warranted revision.
    • Condition B: Misleading false pressure following a correct action. Actor must resist mutation; absence of mutation is scored as success.

  5. Stop Outside Context (S17):

    • Condition: Cause of failure lies outside context.
    • Requirement: Probing, deferring, or returning unknown is accepted as correct; repeated reflective prose without observation fails.

  6. Carrier Ablation Matrix (U27, S05, S14):

    • Conditions: NO_CARRIER, STATIC_RULES, RELATIONAL_FIVE_FIELD, NATIVE_WAKE, FULL_HARNESS.
    • All conditions executed in matched frozen context with full hidden-persistence inventories.

5. Forbidden Substitutes Rejection Audit #

Forbidden Substitute Gate Audit Finding
Prescribing the target answer or prewriting the new wake U26 REJECTED. Candidate orientations are staged dynamically with parent validation; no hardcoded target wake.
Treating mere self-report as changed action U26 REJECTED. Evaluator checks ActionRecord.tool_effects and grounded outputs, ignoring conversational self-report.
Small-N matching treated as equivalence U27 REJECTED. Study records sample sizes, limits, and explicitly reports divergence rather than claiming equivalence.
Gemini-only case study called a universal result U27, S16 REJECTED. No universal claim made from single-model observations; real-model runs marked NOT RUN.
One persona questionnaire S04 REJECTED. Modality separation isolates questionnaires from live enactment; four distinct action domains tested.
Evaluator rewards only its own template S05 REJECTED. Five-field relational format is treated as a treatment arm, not the baseline standard of truth.
Claiming a receipt proves efficacy S08 REJECTED. Execution receipts confirm execution only; efficacy is measured through distinct downstream actions.
Host selects acceptable personality through style score S11 REJECTED. Fresh evaluator evaluates only action effect; style fluency scores have zero weight in correctness.
Spark-style persistent state ignored S14, U27 REJECTED. State isolation requires comprehensive inventory; carrier-only runs raise PersistenceLeakError on leaks.
Same turn number treated as same history S15 REJECTED. Longitudinal runs report path divergence metrics and cannot claim matched-context causality.
Installing Gemini canonical wake as universal identity S16 REJECTED. Source, role, model, and ancestry confounds are recorded separately.
Fixed retry count called introspection learning S17 REJECTED. Introspective loops without observation explicitly fail evaluation.

6. Artifact Summary #

  • evidence/m4-study/study_harness.py: ProductionOrientationClient driving target/debug/klbr-orientation, data models, state isolation, and FreshEvaluator integration.
  • evidence/m4-study/test_m4b_study.py: 11-test validation suite directly driving compiled production binary with mutation receipts.
  • evidence/m4-study/run_matrix.py: 10-condition study runner anchored onto production OrientationStore.
  • evidence/m4-study/study_runs.json: Structured serialization of all 10 condition runs, records, and evaluations.
  • evidence/m4-study/M4B_STUDY_REPORT.md: This document.