tranquil-store: make it seem a little less 'experimental'
Lewis: May this revision serve well! <did:plc:3fwecdnvtcscjnrx2p4n7alz>
270 unmodified linesExpand all271272273274275275276277278279542 unmodified linesExpand all270 unmodified linesExpand all# Default value: ""#s3_path = ""
# Repository backend: `postgres` by default, or `tranquil-store`, our embedded db.# tranquil-store is EXPERIMENTAL!!!! RISK OF TOTAL DATA LOSS.# `tranquil-store` is our own solution, not a tried-and-tested out-of-box database. If you're not feeling brave, don't choose this.## Can also be specified via environment variable `REPO_BACKEND`.## Default value: "postgres"542 unmodified linesExpand all5 unmodified lines678910101112131461 unmodified lines5 unmodified lines
# Tranquil & the world
A PDS is an extremely important aspect of atproto in general, dare I say the bedrock of the whole thing. Storing data reliably and giving it out at the right time is its bread and butter, and if it fails that even once then it breaks its contract with you.The reference ("ref") PDS uses SQLite for its storage backend, and Tranquil opted to go with PostgreSQL initially (and we have an experimental embedded-db that we sorta made ourselves). We'd also like to implement a SQLite backend too - to that effect we have made a database trait with both postgres and our embedded db implementing the correct functions! All it takes to add a SQLite backend is implementing those same functions - hell wouldn't it be fun to have some fun storage backends too?The reference ("ref") PDS uses SQLite for its storage backend, and Tranquil opted to go with PostgreSQL initially (and we have an embedded-db that we sorta made ourselves). We'd also like to implement a SQLite backend too - to that effect we have made a database trait with both postgres and our embedded db implementing the correct functions! All it takes to add a SQLite backend is implementing those same functions - hell wouldn't it be fun to have some fun storage backends too?Each storage backend has its trade-offs - so at the heart of Tranquil we always want to give users choice and put them in the driving seat - why should we choose your storage backend for you if you have a hankering for MongoDB or something? Go ahead and implement the database functions, let's have it. If I sound sarcastic I'm sorry, I'm actually serious.
Which database should you choose? At time of writing the answer is most definitely postgres - if you choose our embedded db right now please don't complain if you have data loss, the one thing a PDS is not supposed to have - though if you do choose it we'd love the live user testing by fire.
61 unmodified lines1 unmodified line234567678911101113121315161415161817181920212223242526272829303031323233343536363738394041414243424344454546474748495051525253545556571 unmodified line
Welcome, brave one.So you're interested in leaving relational databases behind? Raw performance? Or... perhaps you simply want to run less services on your machine?
Tranquil's embedded DB is experimental.Risk of total data loss.Tranquil's embedded DB is not necessarily experimental now that Mr Oyster has been running his just fine for about half a year, but still, just a *slight* risk of total data loss. Back up your data and we'd be able to nurse borked data back to health with some spit and polish.
## What's the difference?
tranquil-store replaces the entire repository layer. When it is selected the server opens no postgres connection at all. The postgres service, its password secret, and the `database.url` value are all unused. Blob storage is however untouched: filesystem or S3 applies exactly as in the base guide.`tranquil-store` is an alternative repository storage layer. When it's selected in the config, the server won't use postgres at all. However, blob storage won't be affected by this db-level choice, in that filesystem vs S3 will continue as a separate concern exactly as in the base guide.
2 settings select and place the store:Here are our settings that set up the store:
- `repo_backend` under `[storage]`, environment variable `REPO_BACKEND`. Set it to `"tranquil-store"`. The default is `"postgres"`.- `data_dir` under `[tranquil_store]`, environment variable `TRANQUIL_STORE_DATA_DIR`. This is optional. It defaults to `/var/lib/tranquil-pds/store`.- `repo_backend` under `[storage]`, environment variable `REPO_BACKEND`. Set it to `"tranquil-store"`. (Default being `"postgres"`.)- `data_dir` under `[tranquil_store]`, environment variable `TRANQUIL_STORE_DATA_DIR`. Optional if the above is picked! It defaults to `/var/lib/tranquil-pds/store`.
So the minimum config delta is one line:Thus the minimum config is as follows:
```toml[storage]repo_backend = "tranquil-store"```
and voila!
## That being said, here are the facts:
- At time of writing, there's no way to transfer an existing Tranquil instance from PG-backed to embedded or vice-versa. If you have an instance and you want to move to embedded, you'll have to spin it up as a new instance and migrate as you would normally.- You will absolutely want to take backups of all users' CAR files daily of not more frequently. As usual, you *really* should have rotation keys separately stored aside somewhere in case the DB explodes in an unrecoverable way.
## Installing: a patch on the existing guides## Installing: slight differences to existing guides
The procedure is the one in [2_INSTALL_CONTAINERS.md](2_INSTALL_CONTAINERS.md), [2_INSTALL_NIX.md](2_INSTALL_NIX.md), or [2_INSTALL_ALPINE.md](2_INSTALL_ALPINE.md). Follow your chosen guide top to bottom and apply the deltas below, otherwise exactly the same!Choose-your-own-adventure based on your poison as per [2_INSTALL_CONTAINERS.md](2_INSTALL_CONTAINERS.md), [2_INSTALL_NIX.md](2_INSTALL_NIX.md), or [2_INSTALL_ALPINE.md](2_INSTALL_ALPINE.md). Follow your chosen guide top-to-bottom and apply the below changes, otherwise exactly the same!
### Containers
Both base guides assume postgres of course, and the units couple the app to it. Dropping the database means uncoupling that out too.Both base guides assume postgres of course, and the units tie the app to pg running. Therefore, dropping the database means un-coupling those too.
Shared, regardless of init system:
1. In `config.toml`, leave `database.url` unset and add the `[storage]` block shown above.2. Skip the database secret. No need to create `tranquil-pds-db-password`.3. The app unit already mounts the `store` directory, so `data_dir` needs no extra setup. The `postgres` directory in the guide's `mkdir` goes unused.4. Backup section: `pg_dump` does not apply. Back up the `data_dir` instead, which holds the metastore, eventlog, and blockstore. CAR files and rotation keys still belong in your own backup as mentioned above.2. Skip the database secret! No need to create `tranquil-pds-db-password`.3. The app unit already mounts the `store` directory, so `data_dir` doesn't need any extra setup. The `postgres` dir in the guide's `mkdir` won't be used.4. Backup section: `pg_dump` doesn't apply. Back up the `data_dir` instead, which contains the metastore, eventlog, and blockstore. CAR files and rotation keys still belong in your own separate high-level backup as mentioned above.
**Debian (quadlets):** Do not copy `tranquil-pds-db.container`. Drop `tranquil-pds-db` from the `systemctl start` and `systemctl enable` commands. The `After=tranquil-pds-db.service` line in `tranquil-pds-app.container` becomes a no-op with the database gone. Remove it if you want.**Debian (quadlets):** Don't bother copying `tranquil-pds-db.container`. Drop `tranquil-pds-db` from the `systemctl start` and `systemctl enable` commands. The `After=tranquil-pds-db.service` line in `tranquil-pds-app.container` will be a no-op with the database gone. Remove it if you want!
**Alpine (OpenRC):** Do not copy the `tranquil-pds-db` init script. The app script hard-depends on it via `need tranquil-pds-db`, so edit `tranquil-pds-app`'s `depend()` to read `need tranquil-pds-pod` instead. Without this the app needs a service that no longer exists and refuses to start. Drop `tranquil-pds-db` from the `rc-update add` command too.**Alpine (OpenRC):** Don't bother copying the `tranquil-pds-db` init script. The app script hard-depends on it via `need tranquil-pds-db`, so edit `tranquil-pds-app`'s `depend()` to grab `need tranquil-pds-pod` instead. Without this, the app will need a service that doesn't actually exist anymore, and would abort at start. Leave out `tranquil-pds-db` from the `rc-update add` command too.
### Nix
1. Set `services.tranquil-pds.settings.storage.repo_backend = "tranquil-store";`.2. Make sure all the postgres related options are left as their default. Otherwise you might end up with a useless running instance of postgres.2. Make sure all the postgres related options are left as their default. Otherwise you might end up with an unnecessarily-running instance of postgres.
That's it!!
Please report anything wrong to us immediately, so that we can make our DB better, faster, stronger!1233456788 unmodified lines[package]name = "tranquil-store"description = "Embedded storage engine for tranquil-pds (experimental)"description = "Embedded storage engine for tranquil-pds"version.workspace = trueedition.workspace = truelicense.workspace = true
88 unmodified lines773 unmodified linesExpand all7747757767777787787797807817821234 unmodified linesExpand all773 unmodified linesExpand all #[config(env = "S3_PATH", default = "")] pub s3_path: String,
/// Repository backend: `postgres` by default, or `tranquil-store`, our embedded db. /// tranquil-store is EXPERIMENTAL!!!! RISK OF TOTAL DATA LOSS. /// `tranquil-store` is our own solution, not a tried-and-tested out-of-box database. If you're not feeling brave, don't choose this. #[config(env = "REPO_BACKEND", default = "postgres")] pub repo_backend: String,}
1234 unmodified linesExpand all249 unmodified linesExpand all250251252253254254255256257258539 unmodified linesExpand all249 unmodified linesExpand all let cfg = tranquil_config::get();
let mut state = match cfg.storage.repo_backend() { tranquil_config::RepoBackend::TranquilStore => { tracing::info!("tranquil-store repo backend active. EXPERIMENTAL!"); tracing::info!("tranquil-store repo backend active. Thank you so much for believing in us!"); Self::from_store(shutdown).await } tranquil_config::RepoBackend::Postgres => { let database_url = &cfg.database.url;539 unmodified linesExpand all