add tag-triggered release pipeline via tangled ci and komodo
109 unmodified linesExpand all110111112113114114115116117118119120121122109 unmodified linesExpand all "@public/*": ["./apps/main-app/public/*"], "@wispplace/*": ["./packages/@wispplace/*/src"] } }, "exclude": ["docs", "node_modules", "apps/hosting-service/cache", "**/dist", "**/node_modules"] "exclude": [ "docs", "node_modules", "apps/hosting-service/cache", "**/dist", "**/node_modules", "deploy" ]}123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133# Tag-triggered releases
`git tag v1.2.3 && git push --tags` builds every image at version `1.2.3`,rolls the fleet onto them, and verifies it actually moved. If anythingfails the CI pipeline goes red.
## How it flows
```tag v1.2.3 pushed └─ .tangled/workflows/release.yml (spindle, microvm) ├─ reject anything that is not vMAJOR.MINOR.PATCH ├─ reject if the commit has not reached the build mirror └─ POST /execute/RunAction → Komodo └─ Action `wisp-release` (deploy/komodo/wisp-release.ts) ├─ find every build that clones this repo ├─ pin each to 1.2.3 @ <sha>, build them one at a time ├─ find every stack running one of those images ├─ set WISP_TAG on each stack, deploy └─ verify each service by tag, liveness and uptime```
Spindle does not build. It gets a fresh overlay disk per run, so buildingthere means a cold build of everything on every tag, plus emulation for anyarchitecture the runner does not share. The build host already has the warmcache and the emulation set up; CI just drives it.
## No topology in this repo, on purpose
Neither Action names a host, a directory, an env file or a service. Theyask Komodo:
- **builds** — every build whose configured source repo is `SOURCE_REPO`- **images** — each build's registry, organization and image name- **stacks** — every stack whose deployed services run one of those images- **paths** — each stack's own run directory, compose files and env files
So adding a node, moving a service or renaming a stack needs no changehere. The only thing the pipeline is told is which repo it is releasing,set once as `SOURCE_REPO` in the workflow.
The consequence worth knowing: **membership is defined by what is running.**A stack joins the release the moment it runs one of our images, and leaveswhen it stops. That is usually what you want, and it is worth a glance atthe Action's log — it prints every stack it selected — the first few times.
## Setting it up
1. **Repo secrets** on tangled (Settings → Secrets, *not* the workflow's `environment:` block, which is public to anyone who can read the repo): `KOMODO_ADDRESS`, `KOMODO_API_KEY`, `KOMODO_API_SECRET`.
`KOMODO_ADDRESS` takes no trailing slash — the workflow appends `/execute/RunAction` and a double slash 404s.
2. **Create two Komodo Actions**, pasting the file contents from here:
| Action | File | |---|---| | `wisp-release` | `deploy/komodo/wisp-release.ts` | | `wisp-prepare-tag` | `deploy/komodo/wisp-prepare-tag.ts` |
3. **Run `wisp-prepare-tag` once**, by hand, with `{"repo": "<SOURCE_REPO>"}`. It changes nothing without `apply` — it renders the edit beside the real compose files, diffs it, and discards the previews. Read that diff, then re-run with `{"apply": true}`.
It rewrites the `image:` lines that point at our own registry so they read `${WISP_TAG}`, and seeds that variable at each stack's current tag so the first deploy is a no-op.
4. **Check the workflow timeout** on the spindle exceeds a full release. The stock default is 5 minutes, far too short — though the existing flake-check and test workflows already prove it is raised here.
5. **Cut a throwaway tag** and watch it. If nothing fires at all, the spindle may predate the `tag:` trigger field.
## Traps this is shaped around
**Registry tokens here are short lived and expire as 403, not 401.** Dockeronly re-authenticates on a 401, so an expired token is fatal rather thantransparent. Builds run **sequentially** for this reason: parallel buildscontending for one builder stretch each login→push window past the tokenlifetime. A push failure gets one retry against the now-warm cache, whichis the fix — not a config change. Builds configured to push many tags atonce (latest, commit, each semver level) spend longer in that window; fewertags is the cheapest way to shrink the risk.
**A successful deploy call is not evidence of a deploy.** Komodo has beenseen reporting success with only some of a stack's images actually pulled.So the Action trusts none of its own return values and asks the containers:right tag, running, and *recreated within the last 15 minutes*. A containeron the correct tag that was never restarted is precisely the failure mode.
**Image ids and digests are never compared across hosts.** A mixed-archfleet resolves the same multi-arch tag to a different per-arch manifest oneach host, so digest equality would fail on a perfectly healthy fleet.
**Releases never rewrite compose.** Compose files are hand-maintained andcarry per-host detail the release knows nothing about — including whichcredential a given service reads through. The release moves one variable,through the stack environment Komodo already owns. `wisp-prepare-tag` isthe only thing that ever edits compose, it runs once, and it gates on arender diff.
**That render diff never prints values.** A rendered compose file containscredentials. Comparing the render is what catches a service being wired tothe wrong one; printing it is what leaks one. Differences outside `image:`lines are reported by key only, and abort.
**A CI timeout does not cancel the release.** It continues server-side.Treat a timed-out pipeline as *unknown* and read the Update beforere-running, because a blind retry redeploys.
**Tag pushes do not run `test.yml`** — it triggers on `branch: [main]`, anda tag push has no branch. Spindle has no cross-workflow `needs:`, so a tagis assumed to point at a commit that already went green on main.
## Rollback
Not a new tag. Set `WISP_TAG` back on the affected stacks and redeploy —the Action is idempotent, so re-running a previous version is safe. Everycompose file `wisp-prepare-tag` touches is backed up alongside it as`<name>.bak-wisptag-<UTC stamp>`.
## Open
- Sync the Actions via ResourceSync TOML so changes to these files land as PRs instead of a UI paste.- Verification has no error-log gate. A bad release that starts cleanly and only then floods the logs still passes; a post-deploy scan belongs at the end of `wisp-release`.123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990# Release on a v* tag.## This workflow does not build and does not know anything about where the# fleet runs. It validates the tag and hands off to the `wisp-release`# Komodo Action, which discovers the builds and stacks at run time and# reports back. See deploy/README.md.when: - event: ["push"] tag: ["v*"]
engine: microvmimage: nixos
clone: depth: 1
dependencies: - curl - jq
environment: # The mirror the release builds actually clone from. Not a secret — it is # this repo's own public remote — but it is the one thing the release has # to be told, so it lives here rather than inline in two commands. SOURCE_REPO: WaveringAna/wisp.place-monorepo
steps: - name: validate tag command: | export PATH="$HOME/.nix-profile/bin:$PATH" set -euo pipefail
version="${TANGLED_REF_NAME#v}" if ! printf '%s' "$version" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+$'; then echo "refusing to release from tag '$TANGLED_REF_NAME'." >&2 echo "build versions are strictly major.minor.patch — a prerelease" >&2 echo "suffix like -rc1 has nowhere to go." >&2 exit 1 fi
printf '%s' "$version" > /tmp/wisp-version printf '%s' "$TANGLED_SHA" > /tmp/wisp-commit echo "releasing $version from $TANGLED_SHA"
- name: verify commit reached the mirror command: | export PATH="$HOME/.nix-profile/bin:$PATH" set -euo pipefail
# The builds clone $SOURCE_REPO, not the knot. A tag pushed only to # origin would build whatever is on the mirror's default branch # instead — same version number, different tree. sha="$(cat /tmp/wisp-commit)" code="$(curl -s -o /dev/null -w '%{http_code}' \ "https://api.github.com/repos/$SOURCE_REPO/commits/$sha")" if [ "$code" != "200" ]; then echo "commit $sha is not on the mirror (HTTP $code)." >&2 echo "push it there first, then re-run this pipeline." >&2 exit 1 fi echo "commit present on mirror"
- name: run release command: | export PATH="$HOME/.nix-profile/bin:$PATH" set -euo pipefail
payload="$(jq -n \ --arg version "$(cat /tmp/wisp-version)" \ --arg commit "$(cat /tmp/wisp-commit)" \ --arg repo "$SOURCE_REPO" \ '{action: "wisp-release", args: {version: $version, commit: $commit, repo: $repo}}')"
# The connection is held open for the whole release: every image built # in turn, then every stack deployed and verified. # # If THIS step times out the release does not stop — it carries on # server-side. A timeout here means "unknown", not "failed". Read the # Update before re-running, because a blind retry redeploys. resp="$(curl -fsS --max-time 5400 \ -X POST "$KOMODO_ADDRESS/execute/RunAction" \ -H 'Content-Type: application/json' \ -H "X-Api-Key: $KOMODO_API_KEY" \ -H "X-Api-Secret: $KOMODO_API_SECRET" \ -d "$payload")"
echo "$resp" | jq -r '.logs[]? | "── \(.stage)\n\(.stdout // "")\n\(.stderr // "")"'
[ "$(echo "$resp" | jq -r '.success')" = "true" ]123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196// One-time preparation for tag-driven releases.//// File contents of the Komodo Action `wisp-prepare-tag`. Run it once, by// hand, before the first tagged release; `wisp-release` does the rest.//// ARGS = { repo: "owner/name", apply?: true, seed?: "<tag>" }//// Without `apply` it changes nothing: it renders the edit beside the real// files, diffs it, and throws the previews away. Run it that way first.//// What it does: rewrites the `image:` lines that point at our own registry// so they read a variable instead of a hardcoded tag, and seeds that// variable at each stack's current tag so the first deploy is a no-op.//// Like wisp-release, this hardcodes nothing about the fleet — hosts, compose// paths and env files all come back from Komodo at run time.
const TAG_VAR = 'WISP_TAG'
const repo = String(ARGS.repo ?? '').trim()const apply = ARGS.apply === true || ARGS.apply === 'true'const seedArg = String(ARGS.seed ?? '').trim()
if (!repo) { throw new Error('missing repo arg (the source repo the builds clone)')}
const imageRepo = (ref: string): string => { const slash = ref.lastIndexOf('/') const colon = ref.lastIndexOf(':') return colon > slash ? ref.slice(0, colon) : ref}
const imageTag = (ref: string): string => { const slash = ref.lastIndexOf('/') const colon = ref.lastIndexOf(':') return colon > slash ? ref.slice(colon + 1) : ''}
const withVar = (env: string, key: string, value: string): string => { const kept = (env ?? '') .split('\n') .filter((line) => !new RegExp(`^\\s*${key}\\s*=`).test(line)) .join('\n') .trimEnd() return `${kept ? `${kept}\n` : ''}${key}=${value}\n`}
const sh = async (server: string, command: string): Promise<string> => { const lines: string[] = [] let code: string | undefined await komodo.execute_server_terminal( { server, command, init: { command: 'bash' } }, { onLine: (line) => { lines.push(line) }, onFinish: (c) => { code = c }, }, ) const out = lines.join('\n').trim() // onFinish also fires with "Early exit without code" when the stream // ends without a marker — that is a failure, not a pass. if (code !== '0') { throw new Error(`[${server}] exit ${code ?? '?'}\n${out}`) } return out}
// ------------------------------------------------ discover what we build
const builds = (await komodo.read('ListBuilds', {})).filter((b: any) => b.info?.repo === repo)if (!builds.length) { throw new Error(`no Komodo build clones '${repo}'`)}
const images: string[] = []for (const build of builds) { const config = (await komodo.read('GetBuild', { build: build.name })).config const registry = (config.image_registry ?? [])[0] if (!registry?.domain) continue images.push( [registry.domain, registry.organization || registry.account, config.image_name || build.name] .filter(Boolean) .join('/'), )}
const ours = (image: string) => images.includes(imageRepo(image))
const stacks = (await komodo.read('ListStacks', {})).filter((s: any) => (s.info?.services ?? []).some((svc: any) => svc.image && ours(svc.image)),)if (!stacks.length) { throw new Error(`no stack runs any of: ${images.join(', ')}`)}
console.log(`${apply ? 'applying to' : 'dry run over'} ${stacks.length} stacks`)
// -------------------------------------------------------- per stack
for (const stack of stacks) { const server = stack.info?.server_name const config = (await komodo.read('GetStack', { stack: stack.name })).config const dir = config.run_directory const files: string[] = config.file_paths ?? []
console.log(`\n── ${stack.name} (${server})`)
// The seed keeps this stack exactly where it already is. It can only be // inferred when every one of our services here is on the same tag; if // they are drifted, picking one silently rolls the others back. const tags = [ ...new Set( (stack.info?.services ?? []).filter((s: any) => s.image && ours(s.image)).map((s: any) => imageTag(s.image)), ), ] const seed = seedArg || (tags.length === 1 ? (tags[0] as string) : '') if (!seed) { throw new Error( `${stack.name} is drifted across tags (${tags.join(', ')}). ` + 'Seeding from one would roll the others back — pass an explicit seed arg.', ) }
// Every env file Komodo passes, in its order, so the render matches what // actually deploys. Komodo's own env file may not exist until the first // deploy writes it, so only pass the ones that are actually there. const envFiles = [config.env_file_path, ...(config.additional_env_files ?? []).map((a: any) => a.path)] .filter(Boolean) .map((p: string) => (p.startsWith('/') ? p : `${dir}/${p}`))
// A grep-style alternation of our image repos, for sed and for grep. const pattern = images.map((i) => i.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')).join('|')
const script = [ `set -euo pipefail`, `cd '${dir}'`, `envargs=()`, `for f in ${envFiles.map((f) => `'${f}'`).join(' ')}; do [ -f "$f" ] && envargs+=(--env-file "$f"); done`, // a temp env file supplies the variable for the preview render only; // passed last so it wins, and never written into the stack's own files `printf '${TAG_VAR}=%s\\n' '${seed}' > /tmp/wisptag.env`, `before=(); after=(); touched=()`, ...files.map( (f) => `if grep -qE '^[[:space:]]*image:[[:space:]]*(${pattern}):' '${f}'; then sed -E 's|^([[:space:]]*image:[[:space:]]*(${pattern})):[^[:space:]]*|\\1:\\$\\{${TAG_VAR}\\}|' '${f}' > '${f}.wisptag-preview' after+=(-f '${f}.wisptag-preview'); touched+=('${f}')else after+=(-f '${f}')fibefore+=(-f '${f}')`, ), `if [ \${#touched[@]} -eq 0 ]; then echo " nothing to rewrite"; rm -f /tmp/wisptag.env; exit 0; fi`, `echo " rewriting: \${touched[*]}"`, `echo " seed: ${TAG_VAR}=${seed}"`, `docker compose "\${envargs[@]}" "\${before[@]}" config > /tmp/wisptag.before`, `docker compose "\${envargs[@]}" --env-file /tmp/wisptag.env "\${after[@]}" config > /tmp/wisptag.after`, // an empty interpolation still diffs cleanly, so catch it explicitly `if grep -qE "^[[:space:]]*image: '?[^:']*:'?\\$" /tmp/wisptag.after; then`, ` echo " REFUSING: ${TAG_VAR} did not reach the renderer — tags came out empty" >&2; exit 1`, `fi`, // Only image: lines may differ. Everything else is reported by key and // never by value: a rendered compose file contains credentials, and // comparing them is the point while printing them is the hazard. `other="$(diff /tmp/wisptag.before /tmp/wisptag.after | grep -E '^[<>]' | grep -vE '^[<>][[:space:]]*image:' | sed -E 's/^([<>])[[:space:]]*([^:]*):.*/\\1 \\2/' | sort -u || true)"`, `imagediff="$(diff /tmp/wisptag.before /tmp/wisptag.after | grep -E '^[<>][[:space:]]*image:' || true)"`, `[ -n "$imagediff" ] && echo "$imagediff" | sed 's/^/ /' || echo " (identical)"`, `if [ -n "$other" ]; then echo " REFUSING: render differs outside image: lines —" >&2; echo "$other" | sed 's/^/ /' >&2; exit 1; fi`, `echo " gate ok: nothing but image: lines changed"`, `rm -f /tmp/wisptag.env /tmp/wisptag.before /tmp/wisptag.after`, apply ? `stamp=$(date -u +%Y%m%dT%H%M%SZ)for f in "\${touched[@]}"; do cp "$f" "$f.bak-wisptag-$stamp"; mv "$f.wisptag-preview" "$f"; echo " applied $f (backup: $f.bak-wisptag-$stamp)"; done` : `rm -f "\${touched[@]/%/.wisptag-preview}"; echo " dry run — previews discarded"`, ].join('\n')
console.log(await sh(server, script))
if (apply) { await komodo.write('UpdateStack', { id: stack.name, config: { environment: withVar(config.environment ?? '', TAG_VAR, seed) }, }) console.log(` seeded stack environment with ${TAG_VAR}=${seed}`) }}
console.log( apply ? '\nprepared. the next v* tag builds every image at one version and unifies the fleet.' : '\ndry run complete — re-run with apply to commit the change.',)123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209// Fleet release.//// File contents of the Komodo Action `wisp-release`. Kept in git so changes// are reviewable; paste it into the Action after editing here.//// Invoked by .tangled/workflows/release.yml on a v* tag with:// ARGS = { version: "1.2.3", commit: "<40-char sha>", repo: "owner/name" }//// Nothing about the fleet is written down here. Which images exist, which// hosts run them, where their compose lives — all of it is read back from// Komodo at run time, keyed off the one thing this repo legitimately knows// about itself: the repo its builds are built from. Adding or removing a// node or a service needs no change to this file.//// Komodo wraps this in `async function main()` on Deno and provides `ARGS`,// `komodo`, `Types`, `YAML`, `TOML`. Anything thrown fails the Action, which// fails the CI pipeline. Those globals do not exist locally, so `deploy` is// excluded from the root tsconfig and this file is not covered by// `bun check`; Komodo's editor is type-aware and checks it there.
const TAG_VAR = 'WISP_TAG'
// A service already on the right tag but never restarted is the// deploy-reported-success-without-pulling failure. Anything older than this// at verification time means the deploy did not actually happen.const MAX_CONTAINER_AGE_SECONDS = 900
const version = String(ARGS.version ?? '').trim()const commit = String(ARGS.commit ?? '').trim()const repo = String(ARGS.repo ?? '').trim()
if (!/^\d+\.\d+\.\d+$/.test(version)) { throw new Error(`bad version arg: '${version}' (want major.minor.patch)`)}if (!/^[0-9a-f]{40}$/.test(commit)) { throw new Error(`bad commit arg: '${commit}' (want a full 40-char sha)`)}if (!repo) { throw new Error('missing repo arg (the source repo the builds clone)')}
// ---------------------------------------------------------------- helpers
// Strip the tag from an image reference, leaving the repository path.// A colon only introduces a tag when it sits in the last path segment.const imageRepo = (ref: string): string => { const slash = ref.lastIndexOf('/') const colon = ref.lastIndexOf(':') return colon > slash ? ref.slice(0, colon) : ref}
// Replace or append a single KEY=value line, leaving every other line alone.const withVar = (env: string, key: string, value: string): string => { const kept = (env ?? '') .split('\n') .filter((line) => !new RegExp(`^\\s*${key}\\s*=`).test(line)) .join('\n') .trimEnd() return `${kept ? `${kept}\n` : ''}${key}=${value}\n`}
// ------------------------------------------- 1. discover what we build
const builds = (await komodo.read('ListBuilds', {})).filter((b: any) => b.info?.repo === repo)
if (!builds.length) { throw new Error(`no Komodo build clones '${repo}' — nothing to release`)}
// The fully-qualified image each build pushes, so stacks can be matched by// what they actually run rather than by name.const images: string[] = []
for (const build of builds) { const config = (await komodo.read('GetBuild', { build: build.name })).config const registry = (config.image_registry ?? [])[0] if (!registry?.domain) { throw new Error(`build ${build.name} has no image registry configured`) } images.push( [registry.domain, registry.organization || registry.account, config.image_name || build.name] .filter(Boolean) .join('/'), )}
console.log(`releasing ${version} from ${commit.slice(0, 7)} — ${builds.length} images`)
// ------------------------------------------ 2. pin version and commit
// Without the commit pin a build clones whatever is on the tracked branch// right now, which is not necessarily what was tagged.for (const build of builds) { await komodo.write('UpdateBuild', { id: build.name, config: { version: version as any, auto_increment_version: false, commit }, })}
// ------------------------------------------------------ 3. build images
// Sequential, not parallel, on purpose. Registry tokens here are short lived// and an expired one comes back as 403 rather than 401, so Docker gives up// instead of re-authenticating. Builds contending for one builder stretch// each login->push window past the token lifetime; one at a time keeps each// window short.//// One retry, because the correct response to a push 403 is to try again// against the now-warm cache, not to change any config.for (const build of builds) { let ok = false for (let attempt = 1; attempt <= 2 && !ok; attempt++) { const update = await komodo.execute('RunBuild', { build: build.name }) ok = update.success if (!ok) { console.log(`build ${build.name} failed (attempt ${attempt}/2), update ${update.id}`) if (attempt === 2) { throw new Error(`build ${build.name} failed twice — see Komodo update ${update.id}`) } } } console.log(`built ${build.name}:${version}`)}
// -------------------------------------- 4. find the stacks that run them
const ours = (image: string) => images.includes(imageRepo(image))
const stacks = (await komodo.read('ListStacks', {})).filter((s: any) => (s.info?.services ?? []).some((svc: any) => svc.image && ours(svc.image)),)
if (!stacks.length) { throw new Error(`built ${version} but no stack runs any of: ${images.join(', ')}`)}
// ---------------------------------------------------------- 5. deploy
// The tag moves through the stack's own environment, which Komodo owns and// writes out at deploy time. Compose files are hand-maintained and carry// per-host detail this release knows nothing about, so nothing here renders,// templates or rewrites one.for (const stack of stacks) { const config = (await komodo.read('GetStack', { stack: stack.name })).config await komodo.write('UpdateStack', { id: stack.name, config: { environment: withVar(config.environment ?? '', TAG_VAR, version) }, })
const update = await komodo.execute('DeployStack', { stack: stack.name }) if (!update.success) { throw new Error(`DeployStack ${stack.name} failed — see Komodo update ${update.id}`) } console.log(`deployed ${stack.name}`)}
// ---------------------------------------------------------- 6. verify
// A successful execute is not evidence that anything was pulled, so none of// the above is trusted — the containers are asked directly.//// Deliberately not compared across hosts: image ids and digests. A fleet can// legitimately mix architectures, and the same multi-arch tag then resolves// to a different per-arch manifest on each host.const problems: string[] = []const deployed = await komodo.read('ListStacks', {})
for (const stack of stacks) { const services = (deployed.find((s: any) => s.name === stack.name)?.info?.services ?? []) as any[]
for (const svc of services.filter((s) => s.image && ours(s.image))) { const want = `${imageRepo(svc.image)}:${version}` const where = `${stack.name}/${svc.service}`
if (svc.image !== want) { problems.push(`${where}: on ${svc.image}, expected ${want}`) continue }
let state: any try { const inspected: any = await komodo.read('InspectStackContainer', { stack: stack.name, service: svc.service, }) state = (inspected.container ?? inspected).State ?? {} } catch (error) { problems.push(`${where}: not inspectable — ${error}`) continue }
if (!state.Running) { problems.push(`${where}: image is correct but the container is not running`) continue } const age = Math.round((Date.now() - Date.parse(state.StartedAt)) / 1000) if (age > MAX_CONTAINER_AGE_SECONDS) { problems.push(`${where}: tagged ${version} but up for ${age}s — never recreated`) continue } console.log(`ok ${where} ${version} (up ${age}s)`) }}
if (problems.length) { throw new Error(`release ${version} did not land cleanly:\n ${problems.join('\n ')}`)}
console.log(`fleet is on ${version}`)