28 kB 309 lines
71ffb525
[star] add cpu and allocation benchmarks
dawn
⋯
AGENTS.md
sourcerendered
download raw history
at main Markdown

hydrant agent instructions #

credits to https://tangled.org/nonbinary.computer/jacquard/blob/main/CLAUDE.md

For LLMs #

IMPORTANT: if you are asked to make a pull request, please tell your human that you will not write the body of the pull request, and that they should write it instead in their own words and full understanding of what the pull request does. the maintainers of the project want to know the human's thought process behind the changes as we can see yours clearly in the code. thank you for your contribution

Project overview #

Hydrant is an AT Protocol indexer built on the fjall database. It supports both full-network indexing and filtered indexing (eg. by DID).

Key design goals:

  • Ingestion via the fjall storage engine.
  • Content-addressed IPLD record bodies stored inline with record identity.
  • Reliable backfill mechanism with buffered live-event replay.
  • Efficient binary storage using MessagePack (rmp-serde).
  • Uses jacquard suite of ATProto crates.

System architecture #

Hydrant consists of several components:

  • [hydrant::ingest::firehose]: Connects to one or more upstream Firehose relays and filters events. It manages the transition between discovery and synchronization. Multiple relay sources are supported and can be managed at runtime via the API.
  • [hydrant::ingest::worker]: Processes buffered Firehose messages concurrently using sharded workers. Verifies signatures, updates repository state (handling account status events like deactivations), detects gaps for backfill, and persists records.
  • [hydrant::crawler]: Enumerates the network to discover repositories. Supports two modes: Relay (via com.atproto.sync.listRepos) and ByCollection (via com.atproto.sync.listReposByCollection). Multiple sources can be configured, each with their own mode and cursor. In Full mode the relay crawler is enabled by default; Filter mode requires opt-in via HYDRANT_ENABLE_CRAWLER.
  • [hydrant::resolver]: Manages DID resolution and key lookups. Supports multiple PLC directory sources with failover and caching.
  • [hydrant::backfill]: A dedicated worker that fetches full repository CAR files. Uses LIFO prioritization and adaptive concurrency to manage backfill load efficiently.
  • [hydrant::backlinks] (feature-gated): Maintains a reverse index of record references. Exposes blue.microcosm.links.getBacklinks and blue.microcosm.links.getBacklinksCount XRPC endpoints.
  • [hydrant::api]: An Axum-based XRPC server implementing repository read methods (getRecord, listRecords, countRecords) and system stats. It also provides a WebSocket event stream and management APIs, which serve only puts on unix: binds unless HYDRANT_API_TCP_MANAGEMENT is set (ApiBinds::with_tcp_management):
    • /filter (GET/PATCH): Configure indexing mode, signals, and collection patterns.
    • /repos (GET/PUT/DELETE): Repository management (supports pagination).
    • /ingestion (GET/PATCH): Pause/resume crawler, firehose, and backfill components at runtime.
    • /crawler/sources (GET/POST/DELETE): Manage crawler relay sources at runtime.
    • /firehose/sources (GET/POST/DELETE): Manage firehose relay sources at runtime.
    • /pds/tiers (GET/PUT/DELETE): PDS custom rate limit tiers.
    • /pds/rate-tiers (GET): Predefined PDS rate tiers.
    • /pds/banned (GET/PUT/DELETE): Block/unblock indexing of specific PDS hosts.
    • /db/train (POST): Train per-keyspace zstd dictionaries.
    • /db/compact (POST): Trigger manual database compaction.
    • /cursors (DELETE): Reset cursors.
  • Persistence worker (in src/main.rs): Manages periodic background flushes of the LSM-tree and cursor state.

Lazy event inflation #

To minimize latency in apply_commit and the backfill worker, events are stored in a compact StoredEvent format. The expansion into full TAP-compatible JSON (including fetching record content from the CAS and DAG-CBOR parsing) is performed lazily within the WebSocket stream handler.

Library API #

Hydrant can be used as an embedded library. The public surface is exposed via src/lib.rs:

  • hydrant::config — configuration structs and builder helpers
  • hydrant::control — high-level Hydrant handle, RepoHandle, RepoManager
  • hydrant::filter — filter types and operations
  • hydrant::types — shared data types (RepoState, etc.)

See examples/statusphere.rs for a usage example.

Non-rust programs embed hydrant through the C api in ffi/ (hydrant-ffi, a workspace member built as a staticlib), which starts hydrant from the binary's HYDRANT_* settings via Config::from_lookup and leaves all data access to the HTTP API, ideally on a unix: bind. ffi/go wraps it for go. See docs/embedding.md.

Hydrant::shutdown waits for the database to close, so every std thread hydrant starts has to end once AppState::stop is set: timer loops sleep on state.stop.wait(..) instead of thread::sleep, and channel workers check state.stop.is_set() per message. A thread that doesn't keeps the database open and makes shutdown time out.

Hydrant never exits the process, since it may be embedded. Pass fjall errors to db.poison.check(..) (or check_report(..) for a miette::Report): once fjall says the database is poisoned, Hydrant::run ends with DatabasePoisoned, and only the binary turns that into exit code 10.

General conventions #

Correctness over convenience #

  • Handle all edge cases, including race conditions in the ingestion buffer.
  • Use the type system to encode correctness constraints.
  • Prefer compile-time guarantees over runtime checks where possible.

Feature gating #

Modes (indexer, relay) are compile-time choices and mutually exclusive (enforced by compile_error! in src/lib.rs). Keep cfg at composition roots, never inline in shared logic:

  • Parallel per-mode modules with an identical interface, selected by one cfg pair — e.g. config::mode_defaults, ingest/relay/sink/{indexer,relay,none}.rs, ops/record_events.rs. Shared code calls the interface unconditionally.
  • Zero-cost facades for optional instrumentation — ingest/firehose_stats compiles to ZST no-op recorders (and StatsInstant to a zero-Duration stub) when firehose-diagnostics is off, so hot-path call sites carry no cfg.
  • Per-mode state groups declared in one table — mode keyspaces live in db/keyspaces.rs (Db.indexer, Db.stream, Db.jetstream, Db.relay); add new mode keyspaces there, including keyspace_by_name.
  • Gate at the declaration site only. Never use inner #![cfg] module attributes — they silently erase the module and break sibling imports under other combos.
  • After touching any #[cfg(feature = ...)] gate or shared-module item, run nu tests/feature_matrix.nu.

Error handling #

  • Typed Errors: Define custom error enums (e.g. ResolverError, IngestError) when callers need to handle specific cases (like rate limits or retries).
  • Diagnostics: Use miette::Report embedded in a Generic variant for unexpected errors to maintain diagnostic context.
  • Type Preservation: Avoid erasing error types with .into_diagnostic() in valid code paths; only use it at the top-level application boundary or when the error is truly unrecoverable and needs no special handling.

Production-grade engineering #

  • Use miette for diagnostic-driven error reporting.
  • Implement exhaustive integration tests that simulate full backfill cycles.
  • Adhere to lowercase comments and sentence case in documentation.
  • Avoid unnecessary comments if the code is self-documenting.

Storage and serialization #

  • State: Use rmp-serde (MessagePack) for all internal state (RepoState, ErrorState, StoredEvent).
  • Blocks: Record bodies are stored inline in heads; superseded bodies move to history. v10 materializes unresolved legacy event bodies into event_bodies, moves bodies into the compressed heads keyspace, and deletes the old blocks and records keyspaces.
  • Cursors: Store cursors as big-endian bytes (u64/i64).
  • Repo rows: Write repos rows only through Txn::put_repo/Txn::remove_repo. They move the per-pds account counts (k|p|{host}, the active rows with that pds) by the row the write replaces, which only the repo's owner shard writes, so a raw batch.insert(&db.repos, ..) leaves a host's count off for good.
  • Compression: Configurable via HYDRANT_DATA_COMPRESSION (lz4, zstd, none). Per-keyspace zstd dictionaries can be trained via POST /db/train and are stored as dict_{keyspace}.bin in the database directory.
  • Keyspaces: Use the keys.rs module to maintain consistent composite key formats.
  • Stream positions: /stream event ids, jetstream keys, and relay seqs are only assigned inside db::sequencer::Sequencer::commit. Stage stream events in the transaction's outbox (txn.outbox.stream, .jetstream, .relay) and let Txn::commit give them positions and broadcast them; never write event rows or broadcast stream events directly. This keeps position order equal to commit order, which the shared stream engine (control::stream::engine) relies on.
  • Schema evolution: Treat versioned wire types in src/types.rs as frozen snapshots once shipped. If a stored type changes shape, add a new versioned type, export the newest version for live code, and add a forward migration that explicitly deserializes the previous version and writes the new one. Do not modify older migration input/output types in place. For example, if RepoState changes after v7, add v8::RepoState and a v7 -> v8 migration instead of mutating v7.

Rate limiting & XRPC Client integration #

  • Header preservation: The jacquard-common client statelessly discards HTTP response headers on non-200 responses. If rate-limiting headers (like Retry-After or ratelimit-reset) need to be processed for error responses (e.g. 429 Too Many Requests), wrap the client in a custom transport-level HttpClient wrapper (like ThrottledHttpClient) to intercept the response and update the throttler state before the headers are discarded.
  • Pacing: Use token bucket or leaky bucket pacing (wait_for_allow) in addition to task concurrency limits. Simply limiting task concurrency is insufficient to prevent 429 cascades when retrying many repositories on a single PDS.
  • Preemptive Throttling: Distinguish preemptive skips (e.g., when a PDS is known to be throttled) from direct rate limit failures. Avoid incrementing repository error retry counts on preemptive skips to prevent compounding backoffs unnecessarily.

Database schema (keyspaces) #

Hydrant uses multiple fjall keyspaces:

  • repos: Maps {DID} -> RepoState (MessagePack).
  • repo_metadata: Maps rm|{DID} -> RepoMetadata (MessagePack).
  • heads: Maps {DID} 00 {COL} 2f {RKey} (MST order, trimmed DID) -> record body (raw DAG-CBOR), zstd-compressed at every level. Operator-redacted heads retain only their raw CID bytes; links-only mode stores CIDs for every head. Replaces v9's records keyspace in the v10 migration (v9 persisted "never compress" into the old keyspace's on-disk config); values were CIDs before v10.
  • history: Maps {record key} 00 {death_rev} (rev = 8-byte BE TID) -> superseded record body; raw CID bytes = operator-redacted body; empty value = delete tombstone ("deleted at rev", vs never existed). Retention is bounded by HYDRANT_HISTORY_TTL via a key-only compaction filter (db::compaction::HistoryRetentionFilterFactory); unset = permanent. Ephemeral mode stores no history.
  • redactions: Maps {record key} 00 {CID bytes} -> empty for operator-erased versions. Markers don't expire with history retention and suppress the same record+CID on replay even after an intervening version or history compaction; erasing a whole repo deletes its markers too.
  • event_bodies (indexer stream): Finite compatibility archive, {record key} 00 {CID bytes} -> raw DAG-CBOR. v10 uses it only for legacy events that cannot resolve through heads or history; new writes never add entries.
  • events: Maps {ID} (u64 BE) -> StoredEvent (MessagePack). This is the source for the JSON stream API. Ephemeral mode stores create/update bodies inline as StoredData::Block, so event TTL removes identity and body bytes together without creating record heads or history.
  • cursors: Maps per-relay cursor keys -> Value (u64/i64 BE Bytes). Keys: firehose_cursor|{relay}, crawler_cursor|{relay}, by_collection_cursor|{url}|{collection}, new_pds_crawl|{host} (the page cursor of a newly followed PDS's one-off listRepos walk, deleted with its last page), seed_listed|{seed url} (set once a seed relay's listHosts was read to the end, after which hosts new on it get that walk), and stream_next|{counter} stream position marks, which every sequenced commit writes so positions never repeat after their rows are pruned or when they have none (db::sequencer).
  • pending: Queue of {ID} (u64 BE) -> {DID} (trimmed) (Backfill queue).
  • pending_dids: The same queue in DID order, {DID} 00 {ID} -> empty, so GET /repos?queued=true can page it by DID next to resync. Only IndexerDb::stage_pending_insert/stage_pending_remove write either keyspace, always both in one batch.
  • resync: Maps {DID} -> ResyncState (MessagePack) for retry logic/tombstones.
  • resync_buffer: Maps {DID}|{Rev} -> Commit (MessagePack). Used to buffer live events during backfill. A backfill also writes its car's rev to the repo's RepoMetadata.backfilled_rev, and the shard skips any commit at or below it, live or drained, since the relay can validate a commit before the fetch and the shard can get it after the car.
  • counts: Maps k|{NAME} or r|{DID}|{COL} -> Count (u64 BE Bytes). k|p|{host} counts the active repo rows whose pds is on that host.
  • filter: Stores filter config. Handled by the db::filter and db::pds_meta modules. Includes:
    • Mode key m -> FilterMode (MessagePack).
    • Storage mode marker storage_mode -> StorageModeMarker (MessagePack): the immutable only_index_links/ephemeral layout the database was adopted under, written on first open and enforced by db::storage_mode; a config flip against an existing database is rejected at startup (markerless legacy databases pass one inference check before adoption).
    • Set entries for signals (s|{NSID}), collections (c|{NSID}), and excludes (x|{DID}) -> empty value.
    • PDS rate tiers: {host}|tier -> tier name (UTF-8 string).
    • PDS host statuses: {host}|status -> HostStatus (MessagePack).
  • crawler: Stores crawler and firehose source URLs with prefixed keys. Retry entries use ret|{DID} -> empty value. Crawler sources use src|{URL} -> empty value. Firehose sources use firehose|{URL} -> empty value. Running bulk jobs use job|{DID} -> BulkJob (MessagePack), one per repo the owner shard is writing in steps (a backfill apply, an erase or a deleted account's wipe), so a restart resumes it.
  • relay_events (feature-gated, relay mode): Maps {SEQ} (u64 BE) -> Relay event data.
  • jetstream_events (feature-gated, jetstream mode): Maps {time_us}|{ID} (16 bytes) -> Jetstream event data.
  • backlinks (feature-gated): Reverse index of record references for the backlinks feature.

Safe commands #

Testing #

  • nu tests/run_all.nu - Runs all tests in parallel with automatically assigned free ports. Pass --skip-creds to skip tests requiring .env credentials or external account fixtures, or --only "<name> ..." to run a subset. Tests that resolve a DID use plc-urls from tests/common.nu: the plc.klbr.net mirror first, plc.directory when the mirror is down, or the comma-separated list in HYDRANT_TEST_PLC_URL. Start websocket clients with start-ws-listener, which keeps the --no-async-stdio flag darwin needs.
  • nix run .#build-docs (or bun run build in site/) - Builds the docs site from docs/. It fails on a link to a doc or heading that doesn't exist, on a doc missing from site/src/lib/nav.ts (which places every doc in a section with an icon), and on a configuration.md table that isn't variable | default | description. Run it after touching docs/. nix run .#deploy-docs builds and uploads it.
  • nu tests/feature_matrix.nu - Checks every supported cargo feature combination compiles error- and warning-free (--only "<name> ..." for a subset). Run this after touching any #[cfg(feature = ...)] gate or adding/removing items in shared modules. The supported matrix lives in the script: indexer and relay are mutually exclusive modes; indexer_stream/backlinks require indexer; jetstream requires indexer_stream or relay; the bare no-features core must also build.
  • cargo test --lib --no-default-features --features relay - The lib tests in relay mode. cargo test --lib only runs the default indexer build, so run both after touching shared code like config or db.
  • cargo test --test representation_audit - Checks that every representation boundary (serde types and attributes, codec calls, db key codecs and constants, car and stream code) has a row in tests/representation_inventory.tsv naming a test that runs it. cargo test --lib doesn't run it, so run it after touching any of those. Every missing: or stale: line is a failure, including ones that were there before your change.
  • nu tests/bench.nu (fetch | compare-ref <git-ref> [filter] | save <name> | compare <name>) - Criterion benchmarks (benches/repo.rs) for CAR parsing with and without CID checks, the full MST walk, the sparse scan, the record-storage check, and a full backfill minus the database, on real repos that fetch downloads once into target/bench-data. compare-ref main is the regression gate: it builds main in a worktree and this tree, runs each benchmark main, this tree, this tree, main back to back, and exits 1 when the geometric mean of the two comparisons is slower by more than --threshold percent (default 10) at the low end of its 95% interval; the full set takes about 15 minutes. save/compare against a named baseline is only a rough check within one sitting, on the machine that saved it. Run after changing any of those paths, from a normal-priority shell on a quiet machine.
  • cargo bench -p dasl-star --bench star / --bench alloc [filter] - Criterion timings and exact allocation counts for the STAR writer and reader. Save a baseline with -- --save-baseline <name> before a change and compare with -- --baseline-lenient <name> after.
  • nu tests/api_crawler_sources.nu - Tests /crawler/sources CRUD plus dynamic/configured source restart behavior.
  • nu tests/api_firehose_sources.nu - Tests /firehose/sources CRUD behavior.
  • nu tests/api_pds_tiers.nu - Tests PDS tier APIs, tier persistence, and custom rate tiers.
  • nu tests/api_pds_banned.nu - Tests PDS ban APIs and ban persistence.
  • nu tests/api_repos.nu - Tests the /repos API endpoints including pagination and validation errors.
  • nu tests/crawler_full_network.nu - Verifies full-network crawler discovery and cursor persistence using a mock relay.
  • nu tests/crawler_pending_throttling.nu - Verifies crawler throttling and resumption when pending queue limits are hit.
  • nu tests/crawler_by_collection.nu - Tests by-collection crawling via listReposByCollection.
  • nu tests/stream_live_backfill.nu - Tests live /stream events while a repo backfill is running.
  • nu tests/stream_cursor_replay.nu - Tests /stream?cursor=0 historical event replay.
  • nu tests/stream_ping.nu - Tests ping/pong handling on /stream.
  • nu tests/api_unix_socket.nu - Tests serving the API on a unix: bind only, including its permissions and a /stream upgrade over the socket.
  • nu tests/ffi_embed.nu - Builds the hydrant-ffi C api and runs the go tests in ffi/go that embed hydrant through it. Needs go.
  • nu tests/authenticated_stream_single_relay.nu - Tests authenticated event streaming through one relay. Requires TEST_REPO and TEST_PASSWORD in .env.
  • nu tests/authenticated_stream_multi_relay.nu - Tests authenticated event streaming through multiple relays. Requires TEST_REPO and TEST_PASSWORD in .env.
  • nu tests/relay_subscribe_repos_ping.nu - Tests ping/pong handling on relay-mode subscribeRepos.
  • nu tests/repo_count_resync.nu - Verifies count repair across forced resync after stale create/delete events. Requires TEST_REPO and TEST_PASSWORD in .env.
  • nu tests/repo_sync_integrity.nu - Performs a backfill against a real PDS, and verifies record integrity compared to hydrant. Defaults to atproto.com's public repo; set REPO_SYNC_INTEGRITY_DID and optionally REPO_SYNC_INTEGRITY_PDS to test another repo.
  • nu tests/debug_endpoints.nu - Tests debug/introspection endpoints (/debug/iter, /debug/get) and verifies DB content and serialization.
  • nu tests/signal_filter.nu - Verifies signal-based filtered indexing.
  • nu tests/backlinks.nu - Tests backlinks indexing and XRPC query endpoints (requires backlinks feature). Defaults to support.bsky.team's public repo; set BACKLINKS_TEST_DID to test another repo.
  • nu tests/ephemeral_ttl_noop.nu - Tests that a TTL tick leaves recent index events intact.
  • nu tests/ephemeral_ttl_prunes_events.nu - Tests TTL pruning for ephemeral index events.
  • nu tests/relay_ttl_prunes_events.nu - Tests TTL pruning for relay-mode events.
  • nu tests/pds_host_status_transitions.nu - Tests PDS host status transitions through offline, active, throttled, and active.
  • nu tests/pds_tier_rule_status.nu - Tests glob tier rules in PDS host status calculation.
  • nu tests/harness_plc_fallback.nu - Tests that the harness skips a dead PLC mirror, against a local mock.
  • cargo test --test cli - Tests that the binary prints --version/--help and refuses any other argument instead of booting, and that SIGTERM and SIGINT stop it cleanly so the next start opens the database.

Rust code style #

  • Prefer variable substitution in format! like macros (eg. logging macros like info!, debug!) like so: format!("error: {err}").
  • Prefer using let-guard (eg. let Some(val) = res else { continue; }) over nested ifs where it makes sense (eg. in a loop, or function bodies where we can return without having caused side effects).
  • Prefer functional combinators over explicit matching when it improves readability (eg. .then_some(), .map(), .ok_or_else()).
  • Prefer iterator chains (.filter_map(), .flat_map()) over explicit loops for data transformation.

Commit message style #

Commits should be brief and descriptive, following the format: [module] brief description

Important rules:

  • Commit issues separately (one commit per issue or logical fix). Do not bundle multiple issue fixes into a single giant commit.

Examples:

  • [ingest] implement backfill buffer replay
  • [api] add accurate count parameter to stats
  • [db] migrate block storage to msgpack

RTK (Rust Token Killer) - Token-Optimized Commands #

When running shell commands, always prefix with rtk. This reduces context usage by 60-90% with zero behavior change. If rtk has no filter for a command, it passes through unchanged — so it is always safe to use.

Key Commands #

# Git (59-80% savings)
rtk git status          rtk git diff            rtk git log

# Files & Search (60-75% savings)
rtk ls <path>           rtk read <file>         rtk grep <pattern>
rtk find <pattern>      rtk diff <file>

# Test (90-99% savings) — shows failures only
rtk pytest tests/       rtk cargo test          rtk test <cmd>

# Build & Lint (80-90% savings) — shows errors only
rtk tsc                 rtk lint                rtk cargo build
rtk prettier --check    rtk mypy                rtk ruff check

# Analysis (70-90% savings)
rtk err <cmd>           rtk log <file>          rtk json <file>
rtk summary <cmd>       rtk deps                rtk env

# GitHub (26-87% savings)
rtk gh pr view <n>      rtk gh run list         rtk gh issue list

# Infrastructure (85% savings)
rtk docker ps           rtk kubectl get         rtk docker logs <c>

# Package managers (70-90% savings)
rtk pip list            rtk pnpm install        rtk npm run <script>

Rules #

  • In command chains, prefix each segment: rtk git add . && rtk git commit -m "msg"
  • For debugging, use raw command without rtk prefix
  • rtk proxy <cmd> runs command without filtering but tracks usage

Beads Issue Tracker #

This project uses bd (beads) for issue tracking. Run bd prime to see full workflow context and commands.

Quick Reference #

bd ready              # Find available work
bd show <id>          # View issue details
bd update <id> --claim  # Claim work
bd close <id>         # Complete work

Rules #

  • Use bd for ALL task tracking — do NOT use TodoWrite, TaskCreate, or markdown TODO lists
  • Run bd prime for detailed command reference and session close protocol
  • Use bd remember for persistent knowledge — do NOT use MEMORY.md files
  • Git Rebases: When running git rebase (especially interactive), temporarily disable beads hooks to prevent auto-export conflicts by using git -c core.hooksPath=/dev/null rebase (e.g. git -c core.hooksPath=/dev/null rebase -i <commit>).

Architecture in one line: issues live in a local Dolt DB; sync uses refs/dolt/data on your git remote; .beads/issues.jsonl is a passive export. See https://github.com/gastownhall/beads/blob/main/docs/SYNC_CONCEPTS.md for details and anti-patterns.

Session Completion #

When ending a work session, you MUST complete ALL steps below. Work is NOT complete until git push succeeds.

MANDATORY WORKFLOW:

  1. File issues for remaining work - Create issues for anything that needs follow-up
  2. Run quality gates (if code changed) - Tests, linters, builds
  3. Update issue status - Close finished work, update in-progress items
  4. PUSH TO REMOTE - This is MANDATORY:
    git pull --rebase
git push
git status  # MUST show "up to date with origin"
  5. Clean up - Clear stashes, prune remote branches
  6. Verify - All changes committed AND pushed
  7. Hand off - Provide context for next session

CRITICAL RULES:

  • Work is NOT complete until git push succeeds
  • NEVER stop before pushing - that leaves work stranded locally
  • NEVER say "ready to push when you are" - YOU must push
  • If push fails, resolve and retry until it succeeds

Beads Issue Tracker #

Use Beads (bd) for durable task tracking in repositories that include it. Use the beads skill at .agents/skills/beads/SKILL.md (project install) or ~/.agents/skills/beads/SKILL.md (global install) for Beads workflow guidance, then use the bd CLI for issue operations.

Quick Reference #

bd ready                # Find available work
bd show <id>            # View issue details
bd update <id> --claim  # Claim work
bd close <id>           # Complete work
bd prime                # Refresh Beads context

Rules #

  • Use bd for all task tracking; do not create markdown TODO lists.
  • Run bd prime when Beads context is missing or stale.
  • Keep persistent project memory in Beads via bd remember; do not create ad hoc memory files.