harden memory architecture
4 unmodified lines567899101112131415168 unmodified lines25262728262930293132333435364 unmodified lines
outputs = inp: inp.parts.lib.mkFlake { inputs = inp; } { systems = [ "x86_64-linux" ]; systems = [ "x86_64-linux" "aarch64-darwin" ]; imports = [ inp.nci.flakeModule ]; perSystem = { pkgs,8 unmodified lines (old.packages or [ ]) ++ (with pkgs; [ cargo-outdated clang wild bun typescript-language-server ]); ]) ++ pkgs.lib.optionals pkgs.stdenv.isLinux [ pkgs.wild ]; }); }; };}21 unmodified lines2223242526272829303139 unmodified lines21 unmodified lines memory direction after the 2026-06-29 review. use this for retrieval and architecture decisions. main follow-up: `klbr-u0q`.- [`memory-implementation-status.md`](./memory-implementation-status.md) - current implementation map, verification commands, and known open work.- [`memory-storage-ownership.md`](./memory-storage-ownership.md) - canonical source tables versus rebuildable retrieval projections and migration rules.- [`folgezettel-memory.md`](./folgezettel-memory.md) - current agent-facing memory tool ontology: sourced zettel tree, titles as api handles, bodies as attractors, and trail-aware retrieval.- [`memory-benches.md`](./memory-benches.md) - current benchmark protocol and39 unmodified lines26 unmodified lines272829303132333132333435363738394041424344454647484946 unmodified lines969798991001011021031041051068 unmodified lines11511611711811912012112212312412530 unmodified lines156157158159142160161162145146163164148149165166167168169170231 unmodified linesExpand all26 unmodified lines - fts rows through `promptable_text` - unicode-token fts rows in `promptable_text_fts` - trigram fts rows in `promptable_text_trigram` for no-space, substring, identifier, and exact-ish candidate generation- `MemoryGarden` can write markdown note files and sync them back into sqlite.- legacy `memory_edges` now mirror into canonical `edges`, `sync_reference_indexes` backfills old rows, and memory-id provenance APIs read canonical ref edges.- memory status changes now project onto canonical refs and rebuild fts visibility.- `MemoryGarden` reconciles files with sqlite using persisted file/store hashes. create, update, rename, delete, and conflict outcomes are explicit; deletion tombstones an unchanged store note and two-sided edits overwrite neither side.- session ingestion is resumable and idempotent through `session_ingests`. turn, note, and memory projection writes use local transactions, and provenance failures are no longer discarded.- [`memory-storage-ownership.md`](./memory-storage-ownership.md) defines source content tables and canonical refs/edges versus rebuildable projections. episode cards are note-owned instead of dual-written to legacy L1 memory.- legacy `memory_edges` mirror into canonical `edges`; explicit projection repair backfills old rows, while normal retrieval performs no global repair.- memory status changes update only the affected canonical refs and FTS rows.- async pipeline storage jobs use a bounded blocking gate. retrieval opens independent query-only WAL connections while writes retain one ordered transactional connection.- note, episode, and chunk refs can resolve benchmark session ids from ref metadata/frontmatter.- dense retrieval now searches canonical refs through `embedding_items`, with bounded local embedding text.- fts retrieval stays fts-native in the production pipeline; the old46 unmodified lines- collect-mode retrieval is wired through `OpPlan::collect_all` for aggregate, order, and update-resolution questions. packet planning widens candidate and packet limits, traces fact-group coverage/gap state, and favors distinct session/ref fact groups instead of letting one session monopolize context. collect mode now consumes a bounded second pass, follows query-aligned target gaps, merges only new refs, and reports explicit sufficiency/degradation stop reasons.- the structured query planner parser is tolerant of common model label shapes (`op`, `operation`, `query_op`, `count`, `latest`, `average`, etc.) while preserving the typed `QueryOp` interface. longmemeval category hints can directly supply `update_resolution`, `order_or_rank`, or8 unmodified lines keep typed numeric value mentions (`number`, `currency`, `percent`, `duration`, `date`, `time`, `ordinal`) so aggregate synthesis does not have to trust the first number in a body. rendered `<memory_packets>` include `<timeline>` and `<facts>` blocks before bodies. rows also carry query-derived target identity, extraction/polarity provenance, and separate `observed_at`/`valid_at` axes. reducers require a consistent target and defer ambiguous preference rows to the grounded reader.- `MemoryPipeline::answer` has an initial planned synthesis path for aggregate count/sum/avg, chronological ordering, previous/latest update resolution, and preference recommendations with personal-support guards. ordinary `lookup` questions still fall through to the reader model.30 unmodified linesrtk cargo test -p klbr-benchrtk cargo run -p klbr-bench -- continuous-loop /tmp/klbr-continuous-loop```
current result:current local result on aarch64-darwin through `nix develop`:
```textcargo check -p klbr-core: 0 errors, 1 pre-existing warningcargo check -p klbr-bench: 0 errors, 1 pre-existing warningcargo check --workspace: passedfocused folgezettel tests: mk surface 2 passed; trail-support packet test 1 passedklbr-core tests: 164 passed, 1 ignoredklbr-bench tests: 19 passedklbr-core tests: 183 passed; 1 local-embedder integration test skippedklbr-bench tests: 21 passedcontinuous-loop smoke: passed with the pre-existing agent.rs dead-code warningsession-first/fts-only trace smoke: evaluated 1, CandidateSessionRecallAny@5 1.0000```
231 unmodified linesExpand all123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081# memory storage ownership
status: current architecture decisiondecided: 2026-07-10tracked by: `klbr-1jn.3`
## decision
klbr has one canonical identity and lifecycle substrate: `refs`, aliases, and`edges`. content remains canonical in the source table for its entity family.search indexes and rendered retrieval material are projections and must neverbe repaired as a side effect of reading.
| concern | canonical owner | derived / compatibility structures ||---|---|---|| immutable conversation source | `turns`, `turn_chunks` | `promptable_text`, fts, embeddings || markdown, zettel, profile, procedural, and episode-card content | `markdown_notes`, `markdown_note_chunks` | `promptable_text`, fts, embeddings || legacy L1 memory and versions | `memories`, `memory_versions` | `vec_memories`, `promptable_text`, fts || stable identity, aliases, lifecycle | `refs`, `ref_aliases` | none || retrieval lane/kind annotations | `ref_metadata` | lane fields copied into fts rows || provenance and topology | `edges` | `memory_edges` is migration compatibility only || dense vectors for canonical refs | `embedding_items` | `vec_memories` is legacy-memory compatibility only || lexical search | none | `promptable_text`, `promptable_text_fts`, `promptable_text_trigram` |
episode event cards are markdown-note entities. production ingest must not alsocreate a semantically identical `memories` row. dense retrieval embeds theepisode note ref through `embedding_items` like every other canonical ref.
## invariants
- a logical artifact has one source content row and one canonical ref.- mutations update the source row, ref lifecycle, and affected projections in one local transaction.- readers never call global backfill or index rebuild functions.- projection rows may be deleted and rebuilt from canonical owners.- lifecycle filtering comes from `refs.status`; projections cannot revive an inactive ref.- `edges` is the only graph written by new code. legacy graph rows may be read or migrated, but are not parallel authority.
## migration
1. stop dual-writing episode cards to `memories`.2. incrementally maintain fts and embedding projections on mutation.3. add an explicit integrity audit and repair command for old databases.4. backfill canonical episode-note embeddings, then retire episode-shaped L1 duplicates after verifying their source refs and edges were preserved.5. remove legacy graph/vector compatibility only after deployed databases have crossed the migration and rollback window.
rollback keeps the source tables and refs untouched. projections can be rebuilt;compatibility tables remain readable until their later removal issue lands.
## garden reconciliation
garden sync records the last file hash and store hash for every path/ref pair.one-sided file changes import normally. renames preserve the note ref. deletiontombstones the note only when the store still matches the last sync. if bothsides changed, sync reports a conflict and overwrites neither side.
automatic store-to-file export is intentionally unsupported for now: agent/dbchanges are protected as conflicts until a person exports or reconciles them.this is preferable to silently replacing a manually edited note or reviving adeleted one.
## sqlite execution
the canonical store keeps one ordered writer connection for transactions.async memory-pipeline operations acquire a bounded permit and run sqlite workon Tokio's blocking pool. retrieval jobs open independent query-only WALconnections, so concurrent reads do not wait on or share the mutable writer.dropping a caller cancels interest in the result, not an already-running sqlitecall; the bounded gate supplies backpressure and prevents unbounded blockingjobs. transaction commit remains the cancellation boundary for writes.
## rejected alternatives
keeping every table authoritative preserves the current dual-write and repairfailure modes. making `memories` own notes would erase the file-backed noteidentity and folgezettel semantics. making `promptable_text` canonical wouldturn a tokenizer-specific cache into durable user data.1234556789246 unmodified linesExpand all25625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130225 unmodified lines3283293303313063323333343353363 unmodified lines34034134234331834434534634734819 unmodified lines36836937037137237337437537685 unmodified lines46246346446543946646746846947047144344444544644744844947247347447547647747847948048148248348428 unmodified lines51351451551651751851952052112 unmodified lines53453553653753853954054154254354454554654754854955055155255355455555655755855922 unmodified lines58258358458558658758858959059159259372 unmodified lines66666766866962567067162762862963067267367467567667767867968068168268368468539 unmodified lines72572672772867872973073173273373473573673773873974074174274374474574698 unmodified lines845846847848849850851852853854855856857858859382 unmodified linesExpand all12421243124412451246124712481249125012511252125312541255125612571258125912601261126212631264126512661267126812691270127112721273127412751276127712781279128012811282128312841285376 unmodified linesExpand all16621663166416651666166716681669167016711672167316741675167616776 unmodified lines16841685168616871688168916901691169216931 unmodified line169516961697169815881699170017011702170324 unmodified lines172817291730173117321733173417351736173717381739174017411742174317441745174641 unmodified lines17881789179017911792179317941795179617971798179918001801180218035 unmodified lines180918101811181218131814181518161817181845 unmodified lines18641865186618671868186918701871187218731874187518761877187818795 unmodified lines188518861887188818891890189118921893189431 unmodified lines1926192719281929193019311932193319341935193619371938193919401941303 unmodified linesExpand alluse std::collections::{HashMap, HashSet};use std::sync::OnceLock;
use anyhow::Result;use chrono::{SecondsFormat, TimeZone, Utc};use chrono::{NaiveDate, SecondsFormat, TimeZone, Utc};use regex::Regex;
use crate::memory::{to_base36, MemoryLane, MemoryStore, ResolvedRef};
246 unmodified linesExpand all pub ref_id: String, pub entity_type: String, pub session_id: Option<String>, pub timestamp: Option<i64>, #[serde(default, skip_serializing_if = "Option::is_none")] pub observed_at: Option<i64>, #[serde(default, skip_serializing_if = "Option::is_none")] pub valid_at: Option<i64>, #[serde(default, skip_serializing_if = "Option::is_none")] pub valid_until: Option<i64>, pub ordinal: usize, #[serde(default, skip_serializing_if = "Option::is_none")] pub target_key: Option<String>, #[serde(default = "unspecified_polarity")] pub polarity: String, #[serde(default)] pub extraction_confidence: f32, #[serde(default = "default_extraction_method")] pub extraction_method: String, pub value_text: Option<String>, pub value_number: Option<f64>, #[serde(default)] pub values: Vec<EvidenceValueMention>, pub text: String,}
fn unspecified_polarity() -> String { "unspecified".to_string()}
fn default_extraction_method() -> String { "legacy_unspecified".to_string()}
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]pub struct EvidenceTimelineEvent { pub ref_id: String, pub session_id: Option<String>, pub timestamp: i64, #[serde(default, skip_serializing_if = "Option::is_none")] pub observed_at: Option<i64>, #[serde(default, skip_serializing_if = "Option::is_none")] pub valid_at: Option<i64>, pub ordinal: usize, pub text: String,}
25 unmodified lines }
pub fn build_packets_with_collect_mode( &self, _query: &str, query: &str, candidates: &[EvidenceAtom], limit: usize, packet_token_budget: usize, collect_mode: bool,3 unmodified lines let mut omissions = Vec::new();
for (index, candidate) in candidates.iter().enumerate() { let expanded = self.expand_candidate_packet(candidate, index + 1, packet_token_budget)?; self.expand_candidate_packet(query, candidate, index + 1, packet_token_budget)?; let packet_id = expanded.packet.packet_id.clone(); omissions.extend(expanded.omissions); if let Some(position) = packet_positions.get(&packet_id).copied() { merge_packet_signals(&mut packets[position], expanded.packet);19 unmodified lines }
fn expand_candidate_packet( &self, query: &str, candidate: &EvidenceAtom, rank: usize, packet_token_budget: usize, ) -> Result<ExpandedPacket> {85 unmodified lines refs.join(",") ); let packet_id = format!("pkt_{}", stable_base36(&packet_seed)); let fact_rows = self.fact_rows_for_packet(&refs, &bodies, candidate.session_id.as_deref())?; self.fact_rows_for_packet(query, &refs, &bodies, candidate.session_id.as_deref())?; let timeline = fact_rows .iter() .filter_map(|row| { row.valid_at .or(row.observed_at) row.timestamp.map(|timestamp| EvidenceTimelineEvent { ref_id: row.ref_id.clone(), session_id: row.session_id.clone(), timestamp, ordinal: row.ordinal, text: row.text.clone(), }) .map(|timestamp| EvidenceTimelineEvent { ref_id: row.ref_id.clone(), session_id: row.session_id.clone(), timestamp, observed_at: row.observed_at, valid_at: row.valid_at, ordinal: row.ordinal, text: row.text.clone(), }) }) .collect(); let omissions = pending_omissions .into_iter()28 unmodified lines }
fn fact_rows_for_packet( &self, query: &str, refs: &[String], bodies: &[String], fallback_session_id: Option<&str>, ) -> Result<Vec<EvidenceFactRow>> {12 unmodified lines .or_else(|| fallback_session_id.map(str::to_string)); let timestamp = self.memory.ref_timestamp(ref_id)?; let values = numeric_value_mentions(body); let (value_text, value_number) = primary_numeric_value(&values); let (target_key, extraction_confidence) = query_target_key(query, body); let valid_at = explicit_valid_at(&values); let text = truncate_chars(body, 180); let row_seed = format!("{ref_id}:{ordinal}:{}", text); Ok(EvidenceFactRow { row_id: format!("fact_{}", stable_base36(&row_seed)), ref_id: ref_id.clone(), entity_type, session_id, timestamp, observed_at: timestamp, valid_at, valid_until: None, ordinal, target_key, polarity: "unspecified".to_string(), extraction_confidence, extraction_method: "query_token_overlap_v1".to_string(), value_text, value_number, values, text,22 unmodified lines
#[derive(Debug, Clone, Default, serde::Serialize, serde::Deserialize)]pub struct EvidenceCoverageTrace { pub collect_mode: bool, pub passes: usize, pub stop_reason: String, #[serde(default, skip_serializing_if = "Vec::is_empty")] pub followup_queries: Vec<String>, pub available_fact_groups: usize, pub selected_fact_groups: usize, pub fact_row_count: usize, pub gap_count: usize,72 unmodified lines "ref=\"{}\" entity_type=\"{}\" ord=\"{}\" session=\"{}\"", xml_escape(&fact.ref_id), xml_escape(&fact.entity_type), fact.ordinal, xml_escape(fact.session_id.as_deref().unwrap_or("")) xml_escape(fact.session_id.as_deref().unwrap_or("")), ); if let Some(timestamp) = fact.timestamp { attrs.push_str(&format!(" t=\"{}\"", timestamp)); if let Some(date) = format_timestamp_utc(timestamp) { attrs.push_str(&format!(" date=\"{}\"", xml_escape(&date))); if let Some(target_key) = &fact.target_key { attrs.push_str(&format!(" target=\"{}\"", xml_escape(target_key))); } if let Some(observed_at) = fact.observed_at { attrs.push_str(&format!(" observed_at=\"{}\"", observed_at)); } if let Some(valid_at) = fact.valid_at { attrs.push_str(&format!(" valid_at=\"{}\"", valid_at)); if let Some(date) = format_timestamp_utc(valid_at) { attrs.push_str(&format!(" valid_date=\"{}\"", xml_escape(&date))); } } if let Some(value) = &fact.value_text { attrs.push_str(&format!(" value=\"{}\"", xml_escape(value)));39 unmodified lines .take(8) .filter_map(|index| packet.timeline.get(index)) { let line = format!( " <event ref=\"{}\" t=\"{}\"{} ord=\"{}\" session=\"{}\">{}</event>\n", " <event ref=\"{}\" t=\"{}\"{}{}{} ord=\"{}\" session=\"{}\">{}</event>\n", xml_escape(&event.ref_id), event.timestamp, format_timestamp_utc(event.timestamp) .map(|date| format!(" date=\"{}\"", xml_escape(&date))) .unwrap_or_default(), event .observed_at .map(|timestamp| format!(" observed_at=\"{timestamp}\"")) .unwrap_or_default(), event .valid_at .map(|timestamp| format!(" valid_at=\"{timestamp}\"")) .unwrap_or_default(), event.ordinal, xml_escape(event.session_id.as_deref().unwrap_or("")), xml_escape(&truncate_chars(&event.text, 96)), );98 unmodified lines .map(|packet| packet.fact_rows.len()) .sum(); EvidenceCoverageTrace { collect_mode, passes: 1, stop_reason: if collect_mode { "initial_pass".to_string() } else { "single_pass".to_string() }, followup_queries: vec![], available_fact_groups: available.len(), selected_fact_groups: selected_count, fact_row_count, gap_count,382 unmodified linesExpand all .map(|value| (Some(value.text.clone()), value.number)) .unwrap_or((None, None))}
fn query_target_key(query: &str, body: &str) -> (Option<String>, f32) { let query_terms = normalized_terms(query); if query_terms.is_empty() { return (None, 0.0); } let body_terms = normalized_terms(body).into_iter().collect::<HashSet<_>>(); let matching = query_terms .iter() .filter(|term| body_terms.contains(*term)) .collect::<Vec<_>>(); let confidence = matching.len() as f32 / query_terms.len() as f32; (matching.last().map(|term| (*term).clone()), confidence)}
fn explicit_valid_at(values: &[EvidenceValueMention]) -> Option<i64> { values .iter() .filter(|value| value.kind == EvidenceValueKind::Date) .find_map(|value| { ["%Y-%m-%d", "%Y/%m/%d", "%m/%d/%Y"] .into_iter() .find_map(|format| NaiveDate::parse_from_str(&value.text, format).ok()) }) .and_then(|date| date.and_hms_opt(0, 0, 0)) .map(|datetime| datetime.and_utc().timestamp())}
fn normalized_terms(text: &str) -> Vec<String> { text.split(|ch: char| !ch.is_alphanumeric() && ch != '_') .map(str::trim) .filter(|term| term.chars().count() >= 3) .filter(|term| !term.chars().all(|ch| ch.is_numeric())) .map(|term| term.to_lowercase()) .collect()}
fn numeric_value_mentions(text: &str) -> Vec<EvidenceValueMention> { numeric_token_regex() .find_iter(text) .filter(|mat| !token_embedded_in_word(text, mat.start(), mat.end()))376 unmodified linesExpand all ref_id: "ref_fact".to_string(), entity_type: "turn_chunk".to_string(), session_id: Some("s1".to_string()), timestamp: Some(42), observed_at: Some(42), valid_at: None, valid_until: None, ordinal: 0, target_key: Some("personal_best".to_string()), polarity: "asserted".to_string(), extraction_confidence: 1.0, extraction_method: "test_fixture".to_string(), value_text: Some("27:45".to_string()), value_number: None, values: vec![EvidenceValueMention { text: "27:45".to_string(),6 unmodified lines packet.timeline = vec![EvidenceTimelineEvent { ref_id: "ref_fact".to_string(), session_id: Some("s1".to_string()), timestamp: 42, observed_at: Some(42), valid_at: None, ordinal: 0, text: "personal best was 27:45".to_string(), }];
1 unmodified line
assert!(rendered.contains("<timeline>")); assert!(rendered.contains("<facts>")); assert!(rendered.contains("ref=\"ref_fact\"")); assert!(rendered.contains("date=\"1970-01-01T00:00:42Z\"")); assert!(rendered.contains("observed_at=\"42\"")); assert!(rendered.contains("value=\"27:45\"")); assert!(rendered.contains("values=\"time:27:45\"")); assert!(rendered.contains("<body ref=\"ref_fact\"")); }24 unmodified lines ); }
#[test] fn explicit_iso_date_becomes_valid_time_without_replacing_observed_time() { let values = numeric_value_mentions("the event happened on 2019-04-03"); let valid_at = explicit_valid_at(&values).expect("ISO date should parse");
assert_eq!( format_timestamp_utc(valid_at).as_deref(), Some("2019-04-03T00:00:00Z") ); }
#[test] fn rendered_packet_with_refs_tracks_only_emitted_evidence_refs() { let mut packet = test_packet_with_refs( "pkt_rendered_refs", &["ref_anchor", "ref_omitted"],41 unmodified lines ref_id: ref_id.clone(), entity_type: "turn_chunk".to_string(), session_id: Some("s1".to_string()), timestamp: Some(42 + ordinal as i64), observed_at: Some(42 + ordinal as i64), valid_at: None, valid_until: None, ordinal, target_key: Some("target".to_string()), polarity: "asserted".to_string(), extraction_confidence: 1.0, extraction_method: "test_fixture".to_string(), value_text: Some(format!("{ordinal}")), value_number: Some(ordinal as f64), values: vec![], text: "fact row with enough text to consume visible packet budget".repeat(3),5 unmodified lines .map(|fact| EvidenceTimelineEvent { ref_id: fact.ref_id.clone(), session_id: fact.session_id.clone(), timestamp: fact.timestamp.unwrap_or_default(), observed_at: fact.observed_at, valid_at: fact.valid_at, ordinal: fact.ordinal, text: fact.text.clone(), }) .collect();45 unmodified lines ref_id: ref_id.clone(), entity_type: "turn_chunk".to_string(), session_id: Some("s1".to_string()), timestamp: Some(42 + ordinal as i64), observed_at: Some(42 + ordinal as i64), valid_at: None, valid_until: None, ordinal, target_key: Some("target".to_string()), polarity: "asserted".to_string(), extraction_confidence: 1.0, extraction_method: "test_fixture".to_string(), value_text: None, value_number: None, values: vec![], text: packet.bodies[ordinal].clone(),5 unmodified lines .map(|fact| EvidenceTimelineEvent { ref_id: fact.ref_id.clone(), session_id: fact.session_id.clone(), timestamp: fact.timestamp.unwrap_or_default(), observed_at: fact.observed_at, valid_at: fact.valid_at, ordinal: fact.ordinal, text: fact.text.clone(), }) .collect();31 unmodified lines ref_id: ref_id.clone(), entity_type: "turn_chunk".to_string(), session_id: Some("s1".to_string()), timestamp: Some(42 + ordinal as i64), observed_at: Some(42 + ordinal as i64), valid_at: None, valid_until: None, ordinal, target_key: Some("target".to_string()), polarity: "asserted".to_string(), extraction_confidence: 1.0, extraction_method: "test_fixture".to_string(), value_text: None, value_number: None, values: vec![], text: packet.bodies[ordinal].clone(),303 unmodified linesExpand all12345678891011121314151617181920212223242526272829303132333435363733 unmodified lines71727374527576777879578081828384858687888990919293949596971 unmodified line991001011026710310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119269193194195196197196 unmodified linesExpand all394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477use std::collections::{HashMap, HashSet};use std::fs;use std::path::{Path, PathBuf};
use anyhow::{Context, Result};use chrono::{SecondsFormat, Utc};
use crate::memory::{ generated_note_ref_for_garden, MarkdownNoteInput, MarkdownNoteRecord, MemoryLane, MemoryStore, generated_note_ref_for_garden, simple_hash, GardenSyncState, MarkdownNoteInput, MemoryLane, MemoryStore,};
#[derive(Debug, Clone, PartialEq, Eq)]pub struct GardenSyncOutcome { pub path: String, pub note_ref: String, pub action: String, pub message: Option<String>,}
#[derive(Debug, Clone, Default, PartialEq, Eq)]pub struct GardenSyncReport { pub outcomes: Vec<GardenSyncOutcome>,}
impl GardenSyncReport { pub fn has_conflicts(&self) -> bool { self.outcomes .iter() .any(|outcome| outcome.action == "conflict") }}
#[derive(Debug, Clone)]pub struct MemoryGarden { root: PathBuf,}33 unmodified lines .with_context(|| format!("failed to write {}", path.display()))?; Ok(path) }
pub fn sync_to_store(&self, memory: &MemoryStore) -> Result<Vec<MarkdownNoteRecord>> { pub fn sync_to_store(&self, memory: &MemoryStore) -> Result<GardenSyncReport> { let mut paths = Vec::new(); collect_markdown_files(&self.root, &mut paths)?; paths.sort();
let mut records = Vec::new(); let states = memory.garden_sync_states()?; let by_path = states .iter() .cloned() .map(|state| (state.path.clone(), state)) .collect::<HashMap<_, _>>(); let by_ref = states .iter() .cloned() .map(|state| (state.note_ref.clone(), state)) .collect::<HashMap<_, _>>(); let mut seen_paths = HashSet::new(); let mut seen_refs = HashSet::new(); let mut report = GardenSyncReport::default(); for path in paths { let contents = fs::read_to_string(&path) .with_context(|| format!("failed to read {}", path.display()))?; let relative = path1 unmodified line .unwrap_or(&path) .to_string_lossy() .replace('\\', "/"); let input = parse_note_file(&contents, relative)?; records.push(memory.upsert_markdown_note(&input)?); let relative = input.path.clone().unwrap_or_default(); let note_ref = input.note_ref.clone().unwrap_or_default(); let file_hash = simple_hash(&contents); let previous = by_path.get(&relative).or_else(|| by_ref.get(¬e_ref)); let store_hash = memory.markdown_note_hash(¬e_ref)?; let file_changed = previous.is_none_or(|state| state.file_hash != file_hash); let store_changed = previous.is_some_and(|state| { store_hash .as_deref() .is_some_and(|hash| hash != state.store_hash) }); seen_paths.insert(relative.clone()); seen_refs.insert(note_ref.clone());
if file_changed && store_changed { report.outcomes.push(GardenSyncOutcome { path: relative, note_ref, action: "conflict".to_string(), message: Some("file and store both changed since the last sync".to_string()), }); continue; }
let action = if previous.is_none() { "created" } else if previous.is_some_and(|state| state.path != relative) { "renamed" } else if file_changed { "updated" } else if store_changed { report.outcomes.push(GardenSyncOutcome { path: relative, note_ref, action: "conflict".to_string(), message: Some( "store changed; store-to-file export is intentionally manual".to_string(), ), }); continue; } else { "unchanged" }; let record = memory.upsert_markdown_note(&input)?; let store_hash = memory .markdown_note_hash(&record.note_ref)? .ok_or_else(|| { anyhow::anyhow!("synced note '{}' has no store hash", record.note_ref) })?; if let Some(old) = previous.filter(|state| state.path != relative) { memory.remove_garden_sync(&old.path)?; } memory.record_garden_sync(&relative, &record.note_ref, &file_hash, &store_hash)?; report.outcomes.push(GardenSyncOutcome { path: relative, note_ref: record.note_ref, action: action.to_string(), message: None, }); }
for GardenSyncState { path, note_ref, store_hash, .. } in states { if seen_paths.contains(&path) || seen_refs.contains(¬e_ref) { continue; } let current_store_hash = memory.markdown_note_hash(¬e_ref)?; if current_store_hash.as_deref() != Some(store_hash.as_str()) { report.outcomes.push(GardenSyncOutcome { path, note_ref, action: "conflict".to_string(), message: Some("file was deleted after the store changed".to_string()), }); continue; } memory.tombstone_markdown_note(¬e_ref)?; memory.remove_garden_sync(&path)?; report.outcomes.push(GardenSyncOutcome { path, note_ref, action: "deleted".to_string(), message: None, }); } Ok(records) Ok(report) }}
fn render_note_file(input: &MarkdownNoteInput, note_ref: &str) -> String {196 unmodified linesExpand all } }) .collect()}
#[cfg(test)]mod tests { use super::*; use tempfile::{NamedTempFile, TempDir};
fn note(note_ref: &str, body: &str) -> MarkdownNoteInput { MarkdownNoteInput { note_ref: Some(note_ref.to_string()), lane: MemoryLane::Semantic, kind: "zettel_note".to_string(), title: "garden synchronization is explicit".to_string(), path: Some(format!("semantic/{note_ref}.md")), body: body.to_string(), sources: vec![], follow: None, entities: vec![], status: "active".to_string(), frontmatter: serde_json::json!({}), } }
#[test] fn garden_sync_tracks_rename_and_tombstones_deleted_files() -> Result<()> { let root = TempDir::new()?; let db = NamedTempFile::new()?; let store = MemoryStore::open(db.path().to_str().unwrap(), 4)?; let garden = MemoryGarden::new(root.path()); let original = garden.write_note(note("zgarden", "initial body"))?;
let created = garden.sync_to_store(&store)?; assert_eq!(created.outcomes[0].action, "created"); let renamed = root.path().join("semantic/renamed.md"); fs::rename(&original, &renamed)?; let rename_report = garden.sync_to_store(&store)?; assert!(rename_report .outcomes .iter() .any(|outcome| outcome.action == "renamed"));
fs::remove_file(&renamed)?; let delete_report = garden.sync_to_store(&store)?; assert!(delete_report .outcomes .iter() .any(|outcome| outcome.action == "deleted")); assert!(store .search_refs_fts("initial body", &[MemoryLane::Semantic], 5)? .is_empty()); Ok(()) }
#[test] fn garden_sync_reports_two_sided_edit_without_overwriting_store() -> Result<()> { let root = TempDir::new()?; let db = NamedTempFile::new()?; let store = MemoryStore::open(db.path().to_str().unwrap(), 4)?; let garden = MemoryGarden::new(root.path()); let path = garden.write_note(note("zconflict", "initial body"))?; garden.sync_to_store(&store)?;
store.upsert_markdown_note(¬e("zconflict", "store edit"))?; fs::write( &path, render_note_file(¬e("zconflict", "file edit"), "zconflict"), )?; let report = garden.sync_to_store(&store)?;
assert!(report.has_conflicts()); assert!(store .search_refs_fts("store edit", &[MemoryLane::Semantic], 5)? .iter() .any(|entry| entry.alias.as_deref() == Some("zconflict"))); assert!(!store .search_refs_fts("file edit", &[MemoryLane::Semantic], 5)? .iter() .any(|entry| entry.body.contains("file edit"))); Ok(()) }}