merge tag-triggered release pipeline
109 unmodified linesExpand all110111112113114114115116117118119120121122109 unmodified linesExpand all "@public/*": ["./apps/main-app/public/*"], "@wispplace/*": ["./packages/@wispplace/*/src"] } }, "exclude": ["docs", "node_modules", "apps/hosting-service/cache", "**/dist", "**/node_modules"] "exclude": [ "docs", "node_modules", "apps/hosting-service/cache", "**/dist", "**/node_modules", "deploy" ]}123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176# Tag-triggered releases
`git tag v1.2.3 && git push --tags` builds every image at version `1.2.3`,rolls the fleet onto them, and verifies it actually moved. If anythingfails the CI pipeline goes red.
## How it flows
```tag v1.2.3 pushed └─ .tangled/workflows/release.yml (spindle, microvm) ├─ reject anything that is not vMAJOR.MINOR.PATCH ├─ reject if the commit has not reached the build mirror └─ POST /execute/RunAction → Komodo └─ Action `wisp-release` (deploy/komodo/wisp-release.ts) ├─ find every build that clones this repo ├─ pin each to 1.2.3 @ <sha>, build them one at a time ├─ find every stack running one of those images ├─ set WISP_TAG on each stack, deploy └─ verify each service by tag, liveness and uptime```
Spindle does not build. It gets a fresh overlay disk per run, so buildingthere means a cold build of everything on every tag, plus emulation for anyarchitecture the runner does not share. The build host already has the warmcache and the emulation set up; CI just drives it.
## No topology in this repo, on purpose
Neither Action names a host, a directory, an env file or a service. Theyask Komodo:
- **builds** — every build whose configured source repo is `SOURCE_REPO`- **images** — each build's registry, organization and image name- **stacks** — every stack whose deployed services run one of those images- **paths** — each stack's own run directory, compose files and env files
So adding a node, moving a service or renaming a stack needs no changehere. The only thing the pipeline is told is which repo it is releasing,set once as `SOURCE_REPO` in the workflow.
The consequence worth knowing: **membership is defined by what is running.**A stack joins the release the moment it runs one of our images, and leaveswhen it stops. That is usually what you want, and it is worth a glance atthe Action's log — it prints every stack it selected — the first few times.
## Setting it up
1. **Repo secrets** on tangled (Settings → Secrets, *not* the workflow's `environment:` block, which is public to anyone who can read the repo): `KOMODO_ADDRESS`, `KOMODO_API_KEY`, `KOMODO_API_SECRET`.
`KOMODO_ADDRESS` takes no trailing slash — the workflow appends `/execute/RunAction` and a double slash 404s.
2. **Create two Komodo Actions**, pasting the file contents from here:
| Action | File | |---|---| | `wisp-release` | `deploy/komodo/wisp-release.ts` | | `wisp-prepare-tag` | `deploy/komodo/wisp-prepare-tag.ts` |
3. **Run `wisp-prepare-tag` once**, by hand, with `{"repo": "<SOURCE_REPO>"}`. It changes nothing without `apply` — it renders the edit beside the real compose files, diffs it, and discards the previews. Read that diff, then re-run with `{"apply": true}`.
It rewrites the `image:` lines that point at our own registry so they read `${WISP_TAG}`, and seeds that variable at each stack's current tag so the first deploy is a no-op.
4. **Check the workflow timeout** on the spindle exceeds a full release. The stock default is 5 minutes, far too short — though the existing flake-check and test workflows already prove it is raised here.
5. **Cut a throwaway tag** and watch it. If nothing fires at all, the spindle may predate the `tag:` trigger field.
## Traps this is shaped around
**Registry tokens here are short lived and expire as 403, not 401.** Dockeronly re-authenticates on a 401, so an expired token is fatal rather thantransparent. Builds run **sequentially** for this reason: parallel buildscontending for one builder stretch each login→push window past the tokenlifetime. A push failure gets one retry against the now-warm cache, whichis the fix — not a config change. Builds configured to push many tags atonce (latest, commit, each semver level) spend longer in that window; fewertags is the cheapest way to shrink the risk.
**A successful deploy call is not evidence of a deploy.** Komodo has beenseen reporting success with only some of a stack's images actually pulled.So the Action trusts none of its own return values and asks the containers:right tag, running, and *recreated within the last 15 minutes*. A containeron the correct tag that was never restarted is precisely the failure mode.
**Image ids and digests are never compared across hosts.** A mixed-archfleet resolves the same multi-arch tag to a different per-arch manifest oneach host, so digest equality would fail on a perfectly healthy fleet.
**Releases never rewrite compose.** Compose files are hand-maintained andcarry per-host detail the release knows nothing about — including whichcredential a given service reads through. The release moves one variable,through the stack environment Komodo already owns. `wisp-prepare-tag` isthe only thing that ever edits compose, it runs once, and it gates on arender diff.
**That render diff never prints values.** A rendered compose file containscredentials. Comparing the render is what catches a service being wired tothe wrong one; printing it is what leaks one. Differences outside `image:`lines are reported by key only, and abort.
**Komodo's execute calls do not wait, and their `success` field lies.** Anexecute returns the Update in its initial state — `success: true,status: InProgress` — before the work has begun. Gating on it makes a wholerelease "succeed" in ten seconds having built and deployed nothing. Insidean Action use the client's own `execute_and_poll`; from CI, capture theupdate id and poll `read/GetUpdate` until `status: Complete`, which is whatthe workflow does.
**Compose is not guaranteed to pull.** Some services carry`pull_policy: never` from before images came from a registry, so`compose up` skips the pull and then dies with a bare "No such image" thatnames no cause. The Action pulls explicitly on each host first, so aregistry or credential problem surfaces as a registry error on the hostthat has it, before anything is torn down.
**Komodo's reported service image is rendered, not observed.** When itrenders without the stack's env file the tag interpolates to nothing andcomes back as `:0`, so a healthy service reads as a failed deploy.Verification asks the container (`InspectStackContainer`) instead.
**Giving up on waiting is not cancelling.** If the poll loop hits itsdeadline the release continues server-side. Treat it as *unknown* and readthe Update before re-running, because a blind retry redeploys.
**Tag pushes do not run `test.yml`** — it triggers on `branch: [main]`, anda tag push has no branch. Spindle has no cross-workflow `needs:`, so a tagis assumed to point at a commit that already went green on main.
## Rollback
Not a new tag. Set `WISP_TAG` back on the affected stacks and redeploy —the Action is idempotent, so re-running a previous version is safe. Everycompose file `wisp-prepare-tag` touches is backed up alongside it as`<name>.bak-wisptag-<UTC stamp>`.
## Type-checking the Actions
`./deploy/typecheck.sh` checks both Actions against Komodo's real clienttypes. An Action is not a module — Komodo pastes its contents inside`async function main()` and supplies the globals — so the script wraps eachfile the same way before checking it, and `komodo-globals.d.ts` declareswhat Komodo injects. Subtract 2 from a reported line to map it back.
The types under `komodo/vendor/` are fetched from a running Komodo andcommitted so the check works offline:
```shfor f in types lib responses terminal; do curl -sL "$KOMODO_ADDRESS/client/$f.d.ts" -o deploy/komodo/vendor/$f.d.tsdone```
Re-fetch them after a Komodo upgrade. This is worth keeping honest: typingthese against the real client is what surfaced `execute_and_poll` (whichreplaced a hand-rolled poller), an Update id read as `.id` when it is`_id.$oid` (so every failure message printed `undefined`), and severaloptional fields being read as if they were guaranteed.
## Open
- Sync the Actions via ResourceSync TOML so changes to these files land as PRs instead of a UI paste.- Verification has no error-log gate. A bad release that starts cleanly and only then floods the logs still passes; a post-deploy scan belongs at the end of `wisp-release`.1234567891011121314{ "compilerOptions": { "target": "ES2022", "lib": ["ES2022", "DOM"], "module": "Preserve", "moduleResolution": "bundler", "allowImportingTsExtensions": true, "strict": true, "noEmit": true, "skipLibCheck": true, "types": [] }, "include": ["komodo/**/*.ts"]}12345678910111213141516171819202122232425262728293031323334353637383940414243444546#!/usr/bin/env bash## Type-check the Komodo Actions.## An Action file is not a module — Komodo pastes its contents inside an# `async function main()` and supplies `ARGS`, `komodo`, `Types`, `YAML` and# `TOML` around it. Checking the files as-is therefore reports noise that is# not real (top-level await "errors", and every Action colliding with every# other at global scope) while hiding what matters.## So each Action is wrapped exactly the way Komodo wraps it and the wrapper# is what gets checked. Line numbers are shifted by the wrapper preamble;# subtract 2 to map a reported line back to the source file.## Types under komodo/vendor are fetched from a running Komodo:# for f in types lib responses terminal; do# curl -sL "https://<komodo>/client/$f.d.ts" -o deploy/komodo/vendor/$f.d.ts# done
set -euo pipefail
here="$(cd "$(dirname "$0")" && pwd)"work="$(mktemp -d)"trap 'rm -rf "$work"' EXIT
mkdir -p "$work/komodo"cp -R "$here/komodo/vendor" "$work/komodo/vendor"cp "$here/komodo/komodo-globals.d.ts" "$work/komodo/"
for action in "$here"/komodo/*.ts; do name="$(basename "$action")" # declaration files describe the wrapper; they must not be wrapped by it case "$name" in *.d.ts) continue ;; esac { echo "export {}" echo "async function main() {" cat "$action" echo "}" echo "void main" } > "$work/komodo/$name"done
cp "$here/tsconfig.json" "$work/tsconfig.json"
bunx tsc -p "$work/tsconfig.json"echo "actions type-check clean"123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107# Release on a v* tag.## This workflow does not build and does not know anything about where the# fleet runs. It validates the tag and hands off to the `wisp-release`# Komodo Action, which discovers the builds and stacks at run time and# reports back. See deploy/README.md.when: - event: ["push"] tag: ["v*"]
engine: microvmimage: nixos
clone: depth: 1
dependencies: - curl - jq
environment: # The mirror the release builds actually clone from. Not a secret — it is # this repo's own public remote — but it is the one thing the release has # to be told, so it lives here rather than inline in two commands. SOURCE_REPO: WaveringAna/wisp.place-monorepo
steps: - name: validate tag command: | export PATH="$HOME/.nix-profile/bin:$PATH" set -euo pipefail
version="${TANGLED_REF_NAME#v}" if ! printf '%s' "$version" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+$'; then echo "refusing to release from tag '$TANGLED_REF_NAME'." >&2 echo "build versions are strictly major.minor.patch — a prerelease" >&2 echo "suffix like -rc1 has nowhere to go." >&2 exit 1 fi
printf '%s' "$version" > /tmp/wisp-version printf '%s' "$TANGLED_SHA" > /tmp/wisp-commit echo "releasing $version from $TANGLED_SHA"
- name: verify commit reached the mirror command: | export PATH="$HOME/.nix-profile/bin:$PATH" set -euo pipefail
# The builds clone $SOURCE_REPO, not the knot. A tag pushed only to # origin would build whatever is on the mirror's default branch # instead — same version number, different tree. sha="$(cat /tmp/wisp-commit)" code="$(curl -s -o /dev/null -w '%{http_code}' \ "https://api.github.com/repos/$SOURCE_REPO/commits/$sha")" if [ "$code" != "200" ]; then echo "commit $sha is not on the mirror (HTTP $code)." >&2 echo "push it there first, then re-run this pipeline." >&2 exit 1 fi echo "commit present on mirror"
- name: run release command: | export PATH="$HOME/.nix-profile/bin:$PATH" set -euo pipefail
payload="$(jq -n \ --arg version "$(cat /tmp/wisp-version)" \ --arg commit "$(cat /tmp/wisp-commit)" \ --arg repo "$SOURCE_REPO" \ '{action: "wisp-release", args: {version: $version, commit: $commit, repo: $repo}}')"
komodo() { curl -fsS -X POST "$KOMODO_ADDRESS/$1" \ -H 'Content-Type: application/json' \ -H "X-Api-Key: $KOMODO_API_KEY" \ -H "X-Api-Secret: $KOMODO_API_SECRET" \ -d "$2" }
# RunAction is ASYNCHRONOUS. It returns in well under a second with # `success: true, status: InProgress` — that field is the initial value # of a record the release has not started writing yet, not a result. # Gating on it would make every tag go green no matter what happened. # The only honest signal is the Update, polled until it completes. started="$(komodo execute/RunAction "$payload")" id="$(echo "$started" | jq -r '.id // ._id."$oid" // empty')" [ -n "$id" ] || { echo "no update id in RunAction response" >&2; exit 1; } echo "release started, update $id"
deadline=$(( $(date +%s) + 5400 )) while :; do update="$(komodo read/GetUpdate "$(jq -n --arg id "$id" '{id: $id}')")" [ "$(echo "$update" | jq -r '.status')" = "Complete" ] && break if [ "$(date +%s)" -ge "$deadline" ]; then echo "still running after 90m — giving up on waiting, NOT on the release." >&2 echo "it continues server-side; read update $id before re-running." >&2 exit 1 fi sleep 15 done
echo "$update" | jq -r '.logs[]? | "── \(.stage)\n\(.stdout // "")\n\(.stderr // "")"'
[ "$(echo "$update" | jq -r '.success')" = "true" ]12345678910111213141516171819202122232425262728293031323334353637383940414243// The globals Komodo injects into an Action script.//// Komodo wraps an Action's file contents in a preamble that imports its// client, declares `ARGS` from the request, and puts the body inside// `async function main()`. None of that is visible in the file itself, so// these declarations stand in for it and let the Actions be type-checked// here rather than only in Komodo's editor.//// Keep in step with Komodo's wrapper (bin/core/src/api/execute/action.rs).// The client types under vendor/ are fetched from a running Komodo:// curl https://<komodo>/client/{types,lib,responses,terminal}.d.ts
import type { KomodoClient } from './vendor/lib.js'import type * as KomodoTypes from './vendor/types.js'
declare global { /** Arguments merged from the Action's defaults and the RunAction request. */ const ARGS: Record<string, unknown>
/** Pre-authenticated client. Actions get one; no keys are configured. */ const komodo: ReturnType<typeof KomodoClient>
// `export import` so `Types` works as both a value and a type // namespace — `Types.Version` as an annotation, `Types.SomeEnum` as a // value, exactly as it behaves inside Komodo. export import Types = KomodoTypes
const YAML: { stringify(value: unknown): string parse(text: string): unknown parseAll(text: string): unknown[] parseDockerCompose(text: string): unknown }
const TOML: { stringify(value: unknown): string parse(text: string): unknown parseResourceToml(text: string): unknown parseCargoToml(text: string): unknown }}
export {}123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224// One-time preparation for tag-driven releases.//// File contents of the Komodo Action `wisp-prepare-tag`. Run it once, by// hand, before the first tagged release; `wisp-release` does the rest.//// ARGS = { repo: "owner/name", apply?: true, seed?: "<tag>" }//// Without `apply` it changes nothing: it renders the edit beside the real// files, diffs it, and throws the previews away. Run it that way first.//// What it does: rewrites the `image:` lines that point at our own registry// so they read a variable instead of a hardcoded tag, and seeds that// variable at each stack's current tag so the first deploy is a no-op.//// Like wisp-release, this hardcodes nothing about the fleet — hosts, compose// paths and env files all come back from Komodo at run time.
const TAG_VAR = 'WISP_TAG'
const repo = String(ARGS.repo ?? '').trim()const apply = ARGS.apply === true || ARGS.apply === 'true'const seedArg = String(ARGS.seed ?? '').trim()
if (!repo) { throw new Error('missing repo arg (the source repo the builds clone)')}
const imageRepo = (ref: string): string => { const slash = ref.lastIndexOf('/') const colon = ref.lastIndexOf(':') return colon > slash ? ref.slice(0, colon) : ref}
// Komodo types these as optional, and they genuinely can be absent — a// resource can exist without a readable config. Failing here names the// resource; letting it through produces an "undefined is not an object"// halfway into a deploy.const required = <T>(value: T | null | undefined, what: string): T => { if (value === null || value === undefined) { throw new Error(`Komodo returned no ${what}`) } return value}
const imageTag = (ref: string): string => { const slash = ref.lastIndexOf('/') const colon = ref.lastIndexOf(':') return colon > slash ? ref.slice(colon + 1) : ''}
const withVar = (env: string, key: string, value: string): string => { const kept = (env ?? '') .split('\n') .filter((line) => !new RegExp(`^\\s*${key}\\s*=`).test(line)) .join('\n') .trimEnd() return `${kept ? `${kept}\n` : ''}${key}=${value}\n`}
// The server terminal is a live bash session, not a script runner. It// appends its own exit-code marker to the LAST LINE of whatever it is sent,// and echoes the shell prompt between lines. A multi-line script therefore// reports only its final line's status, interleaves prompt noise into the// output, and — if any line leaves bash waiting for more input — hangs// forever with no marker and no timeout.//// So everything goes over as exactly one line: the script base64-encoded and// piped into a fresh non-interactive bash. The marker then reflects that// bash's exit status, which is what we actually want to know.const sh = async (server: string, script: string): Promise<string> => { const encoded = btoa(String.fromCharCode(...new TextEncoder().encode(script))) const command = `printf %s '${encoded}' | base64 -d | bash`
const lines: string[] = [] let code: string | undefined await komodo.execute_server_terminal( { server, command, init: { command: 'bash' } }, { onLine: (line) => { lines.push(line) }, onFinish: (c) => { code = c }, }, ) const out = lines.join('\n').trim() // onFinish also fires with "Early exit without code" when the stream // ends without a marker — that is a failure, not a pass. if (code !== '0') { throw new Error(`[${server}] exit ${code ?? '?'}\n${out}`) } return out}
// ------------------------------------------------ discover what we build
const builds = (await komodo.read('ListBuilds', {})).filter((b) => b.info?.repo === repo)if (!builds.length) { throw new Error(`no Komodo build clones '${repo}'`)}
const images: string[] = []for (const build of builds) { const config = required( (await komodo.read('GetBuild', { build: build.name })).config, `config for build ${build.name}`, ) const registry = (config.image_registry ?? [])[0] if (!registry?.domain) continue images.push( [registry.domain, registry.organization || registry.account, config.image_name || build.name] .filter(Boolean) .join('/'), )}
const ours = (image: string) => images.includes(imageRepo(image))
const stacks = (await komodo.read('ListStacks', {})).filter((s) => (s.info?.services ?? []).some((svc) => svc.image && ours(svc.image)),)if (!stacks.length) { throw new Error(`no stack runs any of: ${images.join(', ')}`)}
console.log(`${apply ? 'applying to' : 'dry run over'} ${stacks.length} stacks`)
// -------------------------------------------------------- per stack
for (const stack of stacks) { const server = required(stack.info?.server_name, `server for stack ${stack.name}`) const config = required( (await komodo.read('GetStack', { stack: stack.name })).config, `config for stack ${stack.name}`, ) const dir = config.run_directory const files: string[] = config.file_paths ?? []
console.log(`\n── ${stack.name} (${server})`)
// The seed keeps this stack exactly where it already is. It can only be // inferred when every one of our services here is on the same tag; if // they are drifted, picking one silently rolls the others back. const tags = [ ...new Set((stack.info?.services ?? []).filter((s) => s.image && ours(s.image)).map((s) => imageTag(s.image))), ] const seed = seedArg || (tags.length === 1 ? (tags[0] as string) : '') if (!seed) { throw new Error( `${stack.name} is drifted across tags (${tags.join(', ')}). ` + 'Seeding from one would roll the others back — pass an explicit seed arg.', ) }
// Every env file Komodo passes, in its order, so the render matches what // actually deploys. Komodo's own env file may not exist until the first // deploy writes it, so only pass the ones that are actually there. const envFiles = [config.env_file_path, ...(config.additional_env_files ?? []).map((a) => a.path)] .filter(Boolean) .map((p: string) => (p.startsWith('/') ? p : `${dir}/${p}`))
// A grep-style alternation of our image repos, for sed and for grep. const pattern = images.map((i) => i.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')).join('|')
const script = [ `set -euo pipefail`, `cd '${dir}'`, `envargs=()`, `for f in ${envFiles.map((f) => `'${f}'`).join(' ')}; do [ -f "$f" ] && envargs+=(--env-file "$f"); done`, // a temp env file supplies the variable for the preview render only; // passed last so it wins, and never written into the stack's own files `printf '${TAG_VAR}=%s\\n' '${seed}' > /tmp/wisptag.env`, `before=(); after=(); touched=()`, ...files.map( (f) => `if grep -qE '^[[:space:]]*image:[[:space:]]*(${pattern}):' '${f}'; then sed -E 's#^([[:space:]]*image:[[:space:]]*(${pattern})):[^[:space:]]*#\\1:\\$\\{${TAG_VAR}\\}#' '${f}' > '${f}.wisptag-preview' after+=(-f '${f}.wisptag-preview'); touched+=('${f}')else after+=(-f '${f}')fibefore+=(-f '${f}')`, ), `if [ \${#touched[@]} -eq 0 ]; then echo " nothing to rewrite"; rm -f /tmp/wisptag.env; exit 0; fi`, `echo " rewriting: \${touched[*]}"`, `echo " seed: ${TAG_VAR}=${seed}"`, `docker compose "\${envargs[@]}" "\${before[@]}" config > /tmp/wisptag.before`, `docker compose "\${envargs[@]}" --env-file /tmp/wisptag.env "\${after[@]}" config > /tmp/wisptag.after`, // an empty interpolation still diffs cleanly, so catch it explicitly `if grep -qE "^[[:space:]]*image: '?[^:']*:'?\\$" /tmp/wisptag.after; then`, ` echo " REFUSING: ${TAG_VAR} did not reach the renderer — tags came out empty" >&2; exit 1`, `fi`, // Only image: lines may differ. Everything else is reported by key and // never by value: a rendered compose file contains credentials, and // comparing them is the point while printing them is the hazard. `other="$(diff /tmp/wisptag.before /tmp/wisptag.after | grep -E '^[<>]' | grep -vE '^[<>][[:space:]]*image:' | sed -E 's/^([<>])[[:space:]]*([^:]*):.*/\\1 \\2/' | sort -u || true)"`, `imagediff="$(diff /tmp/wisptag.before /tmp/wisptag.after | grep -E '^[<>][[:space:]]*image:' || true)"`, `[ -n "$imagediff" ] && echo "$imagediff" | sed 's/^/ /' || echo " (identical)"`, `if [ -n "$other" ]; then echo " REFUSING: render differs outside image: lines —" >&2; echo "$other" | sed 's/^/ /' >&2; exit 1; fi`, `echo " gate ok: nothing but image: lines changed"`, `rm -f /tmp/wisptag.env /tmp/wisptag.before /tmp/wisptag.after`, apply ? `stamp=$(date -u +%Y%m%dT%H%M%SZ)for f in "\${touched[@]}"; do cp "$f" "$f.bak-wisptag-$stamp"; mv "$f.wisptag-preview" "$f"; echo " applied $f (backup: $f.bak-wisptag-$stamp)"; done` : `rm -f "\${touched[@]/%/.wisptag-preview}"; echo " dry run — previews discarded"`, ].join('\n')
console.log(await sh(server, script))
if (apply) { await komodo.write('UpdateStack', { id: stack.name, config: { environment: withVar(config.environment ?? '', TAG_VAR, seed) }, }) console.log(` seeded stack environment with ${TAG_VAR}=${seed}`) }}
console.log( apply ? '\nprepared. the next v* tag builds every image at one version and unifies the fleet.' : '\ndry run complete — re-run with apply to commit the change.',)123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333// Fleet release.//// File contents of the Komodo Action `wisp-release`. Kept in git so changes// are reviewable; paste it into the Action after editing here.//// Invoked by .tangled/workflows/release.yml on a v* tag with:// ARGS = { version: "1.2.3", commit: "<40-char sha>", repo: "owner/name" }//// Nothing about the fleet is written down here. Which images exist, which// hosts run them, where their compose lives — all of it is read back from// Komodo at run time, keyed off the one thing this repo legitimately knows// about itself: the repo its builds are built from. Adding or removing a// node or a service needs no change to this file.//// Komodo wraps this in `async function main()` on Deno and provides `ARGS`,// `komodo`, `Types`, `YAML`, `TOML`. Anything thrown fails the Action, which// fails the CI pipeline. Those globals do not exist locally, so `deploy` is// excluded from the root tsconfig and this file is not covered by// `bun check`; Komodo's editor is type-aware and checks it there.
const TAG_VAR = 'WISP_TAG'
// A service already on the right tag but never restarted is the// deploy-reported-success-without-pulling failure. Anything older than this// at verification time means the deploy did not actually happen.const MAX_CONTAINER_AGE_SECONDS = 900
const version = String(ARGS.version ?? '').trim()const commit = String(ARGS.commit ?? '').trim()const repo = String(ARGS.repo ?? '').trim()
const [major, minor, patch] = version.split('.').map(Number)
if (!/^\d+\.\d+\.\d+$/.test(version)) { throw new Error(`bad version arg: '${version}' (want major.minor.patch)`)}if (!/^[0-9a-f]{40}$/.test(commit)) { throw new Error(`bad commit arg: '${commit}' (want a full 40-char sha)`)}if (!repo) { throw new Error('missing repo arg (the source repo the builds clone)')}
// Komodo stores a build version as a struct, not a string.const semver: Types.Version = { major, minor, patch }
// ---------------------------------------------------------------- helpers
// Strip the tag from an image reference, leaving the repository path.// A colon only introduces a tag when it sits in the last path segment.const imageRepo = (ref: string): string => { const slash = ref.lastIndexOf('/') const colon = ref.lastIndexOf(':') return colon > slash ? ref.slice(0, colon) : ref}
// Komodo types these as optional, and they genuinely can be absent — a// resource can exist without a readable config. Failing here names the// resource; letting it through produces an "undefined is not an object"// halfway into a deploy.const required = <T>(value: T | null | undefined, what: string): T => { if (value === null || value === undefined) { throw new Error(`Komodo returned no ${what}`) } return value}
// Replace or append a single KEY=value line, leaving every other line alone.const withVar = (env: string, key: string, value: string): string => { const kept = (env ?? '') .split('\n') .filter((line) => !new RegExp(`^\\s*${key}\\s*=`).test(line)) .join('\n') .trimEnd() return `${kept ? `${kept}\n` : ''}${key}=${value}\n`}
// `komodo.execute(...)` DOES NOT WAIT. It returns the Update record in its// initial state — `success: true, status: "InProgress"` — before the work// has begun, so reading `.success` off it is meaningless: a whole release// will "succeed" in ten seconds having built and deployed nothing.//// `execute_and_poll` is the client's own version that waits for the Update// to reach Complete. Use it for anything whose result matters. It can also// return a batch array, which none of the calls here do — `one()` narrows// that away rather than leaving the union to be indexed blindly.const one = (result: Awaited<ReturnType<typeof komodo.execute_and_poll>>): Types.Update => { if (Array.isArray(result)) { throw new Error('expected a single update, got a batch result') } return result}
// An Update's id is `_id.$oid`, not `.id`. Getting this wrong prints// "undefined" into exactly the error that is supposed to point at the run.const updateRef = (update: Types.Update): string => update._id?.$oid ?? 'unknown'
// The server terminal is a live bash session: it appends its exit-code// marker to the LAST LINE sent, echoes the prompt between lines, and hangs// forever if a line leaves bash awaiting input. So a script goes over as// exactly one line, base64-encoded into a fresh non-interactive bash, and// the marker then reflects that bash's status.const sh = async (server: string, script: string): Promise<string> => { const encoded = btoa(String.fromCharCode(...new TextEncoder().encode(script))) const lines: string[] = [] let code: string | undefined await komodo.execute_server_terminal( { server, command: `printf %s '${encoded}' | base64 -d | bash`, init: { command: 'bash' } }, { onLine: (line) => { lines.push(line) }, onFinish: (c) => { code = c }, }, ) const out = lines.join('\n').trim() if (code !== '0') { throw new Error(`[${server}] exit ${code ?? '?'}\n${out}`) } return out}
// ------------------------------------------- 1. discover what we build
const builds = (await komodo.read('ListBuilds', {})).filter((b) => b.info?.repo === repo)
if (!builds.length) { throw new Error(`no Komodo build clones '${repo}' — nothing to release`)}
// The fully-qualified image each build pushes, so stacks can be matched by// what they actually run rather than by name.const images: string[] = []
for (const build of builds) { const config = required( (await komodo.read('GetBuild', { build: build.name })).config, `config for build ${build.name}`, ) const registry = (config.image_registry ?? [])[0] if (!registry?.domain) { throw new Error(`build ${build.name} has no image registry configured`) } images.push( [registry.domain, registry.organization || registry.account, config.image_name || build.name] .filter(Boolean) .join('/'), )}
console.log(`releasing ${version} from ${commit.slice(0, 7)} — ${builds.length} images`)
// ------------------------------------------ 2. pin version and commit
// Without the commit pin a build clones whatever is on the tracked branch// right now, which is not necessarily what was tagged.for (const build of builds) { await komodo.write('UpdateBuild', { id: build.name, config: { version: semver, auto_increment_version: false, commit }, })}
// ------------------------------------------------------ 3. build images
// Sequential, not parallel, on purpose. Registry tokens here are short lived// and an expired one comes back as 403 rather than 401, so Docker gives up// instead of re-authenticating. Builds contending for one builder stretch// each login->push window past the token lifetime; one at a time keeps each// window short.//// One retry, because the correct response to a push 403 is to try again// against the now-warm cache, not to change any config.for (const build of builds) { let ok = false for (let attempt = 1; attempt <= 2 && !ok; attempt++) { const update = one(await komodo.execute_and_poll('RunBuild', { build: build.name })) ok = update.success if (!ok) { console.log(`build ${build.name} failed (attempt ${attempt}/2), update ${updateRef(update)}`) if (attempt === 2) { throw new Error(`build ${build.name} failed twice — see Komodo update ${updateRef(update)}`) } } } console.log(`built ${build.name}:${version}`)}
// -------------------------------------- 4. find the stacks that run them
const ours = (image: string) => images.includes(imageRepo(image))
const stacks = (await komodo.read('ListStacks', {})).filter((s) => (s.info?.services ?? []).some((svc) => svc.image && ours(svc.image)),)
if (!stacks.length) { throw new Error(`built ${version} but no stack runs any of: ${images.join(', ')}`)}
// ---------------------------------------------------------- 5. deploy
// The tag moves through the stack's own environment, which Komodo owns and// writes out at deploy time. Compose files are hand-maintained and carry// per-host detail this release knows nothing about, so nothing here renders,// templates or rewrites one.// What each service was on before this deploy, so verification can tell// "already correct, nothing to do" apart from "never actually deployed".const before = new Map<string, string>()for (const stack of stacks) { for (const svc of stack.info?.services ?? []) { if (svc.image && ours(svc.image)) before.set(`${stack.name}/${svc.service}`, svc.image) }}
for (const stack of stacks) { const config = required( (await komodo.read('GetStack', { stack: stack.name })).config, `config for stack ${stack.name}`, ) await komodo.write('UpdateStack', { id: stack.name, config: { environment: withVar(config.environment ?? '', TAG_VAR, version) }, })
// Pull explicitly, on the host, before deploying. // // Compose cannot be relied on to fetch these: some services carry // `pull_policy: never` from before images came from a registry, so // `compose up` skips the pull and then dies with a bare // "No such image" that names no cause. Pulling here means a registry or // credential problem surfaces as a registry error, on the host that has // the problem, before anything is torn down. const wanted = [ ...new Set( (stack.info?.services ?? []) .map((s) => s.image) .filter((i: string) => i && ours(i)) .map((i: string) => `${imageRepo(i)}:${version}`), ), ]
const pull = [ 'set -euo pipefail', ...wanted.map((image) => `echo "pulling ${image}"; docker pull -q '${image}' >/dev/null`), ].join('\n')
const server = required(stack.info?.server_name, `server for stack ${stack.name}`) await sh(server, pull) console.log(`pulled ${wanted.length} images on ${server}`)
const update = one(await komodo.execute_and_poll('DeployStack', { stack: stack.name })) if (!update.success) { throw new Error(`DeployStack ${stack.name} failed — see Komodo update ${updateRef(update)}`) } console.log(`deployed ${stack.name}`)}
// ---------------------------------------------------------- 6. verify
// A successful execute is not evidence that anything was pulled, so none of// the above is trusted — the containers are asked directly.//// Deliberately not compared across hosts: image ids and digests. A fleet can// legitimately mix architectures, and the same multi-arch tag then resolves// to a different per-arch manifest on each host.const problems: string[] = []const deployed = await komodo.read('ListStacks', {})
for (const stack of stacks) { const services = deployed.find((s) => s.name === stack.name)?.info?.services ?? []
for (const svc of services.filter((s) => s.image && ours(s.image))) { const want = `${imageRepo(svc.image)}:${version}` const where = `${stack.name}/${svc.service}`
// The image is read off the container, not off the stack listing. // Komodo's listing reports a *rendered* image, and when it renders // without the stack's env file the tag interpolates to nothing and // surfaces as ":0" — a healthy service then looks like a failed // deploy. The container knows what it is actually running. let container: Types.InspectStackContainerResponse try { container = await komodo.read('InspectStackContainer', { stack: stack.name, service: svc.service, }) } catch (error) { problems.push(`${where}: not inspectable — ${error}`) continue }
const state = container.State ?? {} const running = container.Config?.Image
if (running !== want) { problems.push(`${where}: on ${running}, expected ${want}`) continue } if (!state.Running) { problems.push(`${where}: image is correct but the container is not running`) continue } // Only demand a recent restart where the tag actually moved. Re-running // a release that is already live is legitimate and recreates nothing, // so an old uptime there is correct rather than suspicious. const startedAt = state.StartedAt if (!startedAt) { problems.push(`${where}: container reports no start time`) continue } const age = Math.round((Date.now() - Date.parse(startedAt)) / 1000) const was = before.get(where) // A pre-deploy image the listing could not render (":0" or a bare // ":") says nothing about whether this moved, so the age check is // skipped rather than failed on a guess. const knownBefore = was && !/:(0)?$/.test(was) const moved = knownBefore && was !== want if (moved && age > MAX_CONTAINER_AGE_SECONDS) { problems.push(`${where}: tagged ${version} but up for ${age}s — never recreated`) continue } console.log(`ok ${where} ${version} (up ${age}s)`) }}
if (problems.length) { throw new Error(`release ${version} did not land cleanly:\n ${problems.join('\n ')}`)}
console.log(`fleet is on ${version}`)