r45: colab ssh slot + http recovery #

result #

implemented in /Users/dawn/proj/albedo; all edits are uncommitted and unstaged. no nix rebuild, daemon restart, runtime stop/unassign, or remote file writes occurred. nothing here makes the change live in the running daemon. its installed /nix/store/...albedo-server-1.0.0/ code is unchanged. dawn must rebuild/install albedo and start a kernel from that updated bundle when safe. this verification loaded the edited module explicitly into this child kernel, not into the daemon.

what changed #

• priv/python/albedo_plugins/colab.py: after remote.connect() raises RemoteError, a stdlib websocket-upgrade probe checks the ssh endpoint. only HTTP 429 plus already-active-session raises the exported ColabSshSessionError (ColabError, status=429). unrelated refusals or probe transport failures re-raise the original error object unchanged.
• successful connects make no extra round trip. the probe is on the failure path only, with a socket timeout capped at 10 seconds and a 4096-byte body read. a successful probe upgrade is immediately closed, never retained as a tunnel.
• new read-only HTTP helpers refresh proxy credentials independently of ssh: runtime_status(vm), contents(vm, path, content=True), read(vm, path) and pull(vm, remote, local, resume=True). contents(..., content=False) returns metadata without loading file contents. pull returns the local Path and streams in 1 MiB chunks on a worker thread.
• resume validates Content-Range. an ignored Range restarts the local file instead of appending corrupt data. invalid ranges/HTTP errors leave the local file untouched; interrupted downloads retain partial data for retry.
• updated robot-docs/colab.md, the extension's model-facing instructions, and test/harness/colab_plugin_test.py.

live before / after #

runtime: gpu-a100-hm-kkb-euw4a2-268zqw5nqszkj.

ssh's symptom is non-deterministic, not merely unhelpful. these variants occurred against the same occupied slot:

full error captured before editing:

RemoteError: ssh target 'colab-gpu-a100-hm-kkb-euw4a2-268zqw5nqszkj' is unreachable: Connection timed out during banner exchange
Connection to UNKNOWN port 65535 timed out

a direct TCP/TLS stdlib websocket handshake returned:

HTTP 429
{"error":"already-active-session"}

a real connect using the final edited source, normal installed remote.connect and fresh daemon-provided credentials raised:

ColabSshSessionError: Colab allows one ssh session per runtime; a previously held session was not released. This is server-side and does not clear on retry. Use the HTTP file routes to keep working: colab.contents(), colab.read(), colab.pull() (and colab.runtime_status() for liveness). stop()/unassign is the only forceful reset and destroys the VM.

the diagnosis comes from the endpoint, never from pattern-matching ssh stderr. no proxy URL, token, request line, or instance dictionary is interpolated into that message. probe transport errors are discarded in favor of the original RemoteError; new runtime HTTP error messages omit credential-bearing URLs too.

live HTTP verification #

while ssh remained refused, the new helpers successfully read /api/status, listed Jupyter's root, read file metadata, and read the text file NGC-DL-CONTAINER-LICENSE (17,232 decoded characters).

real binary transfer using the final source:

metadata = await candidate.colab.contents(vm, "python-apt.tar.xz", content=False)
await candidate.colab.pull(vm, "python-apt.tar.xz", local, resume=False)

• remote metadata size: 346,012 bytes.
• downloaded local size: 346,012 bytes.
• local artifact: audits/r45_colab_ssh_slot/python-apt.tar.xz.
• sha256: 4ab9c915d1295afbf2ca197993454d3659bb8dee4e9cd39ceba9efe5970873cc.
• truncated only that local artifact to 12,345 bytes, then called the new pull(..., resume=True). observed HTTP 206, with Content-Range: bytes 12345-346011/346012.
• resumed size and sha256 matched the original full download exactly.

all VM requests were read-only; no training files were modified.

regression checks #

• python test/python_test_runner.py test/harness/colab_plugin_test.py: 19 tests passed. local fake HTTP endpoints exercise the actual upgrade request and return the specific 429, an unrelated 429, a 502, and a 101. only the specific 429 is relabelled. tests also cover original-error identity on probe network failure, no probe on successful connect, path escaping, credential refresh, fresh multi-chunk download, 206 resume, ignored Range, invalid ranges, 416/403, and truncated transfer retention.
• uvx ruff@0.16.10 check priv/python/albedo_plugins/colab.py test/harness/colab_plugin_test.py: clean; both Python files were formatted with the same pinned ruff.
• uvx ty@0.0.85 check priv/python/albedo_plugins/colab.py test/harness/colab_plugin_test.py: clean.
• git diff --check: clean.
• logs: tests.log, ruff.log, ty.log beside this report.
• full daemon/e2e suite and Gleam format check were not run. gleam is not on this kernel's PATH; no nix rebuild or daemon restart was attempted.

remaining limits / activation #

this names the refusal and provides a working independent path; it does not release the occupied ssh slot. forceful reset via stop/unassign destroys the VM, so it was not attempted. a later server-side reap may clear it, but retry itself is not a reset.

HTTP paths are relative to Jupyter's root, not the ssh /content working directory. resume assumes unchanged remote content; use resume=False for mutable files. HTTP 416 is conservatively an error, not proof of completeness. text reads/directory listings return their JSON/text in memory; large/binary files belong through pull. HTTP recovery is file/liveness access, not a replacement remote execution transport.

patch: audits/r45_colab_ssh_slot/colab_ssh_slot.patch is the complete git -C /Users/dawn/proj/albedo diff. report and patch were written to .partial files and renamed into place. dawn owns the commit and deployment; the running daemon still does not have this change.

archive note #

The 346,012-byte transfer-test artifact (python-apt.tar.xz, an unrelated Debian package that happened to be in the runtime's Jupyter root) was verified against the recorded sha256 4ab9c915d1295afbf2ca197993454d3659bb8dee4e9cd39ceba9efe5970873cc and then deleted from this archive rather than committed. The digest is the evidence; the bytes are noise in a git tree.