feat: thin-node cluster rearchitecture (headless nodes, keypair enrollment, instant browsing) #6

Merged

james merged 11 commits from feat/thin-node-cluster into main 2026-07-28 13:36:24 +00:00

Conversation 0 Commits 11 Files changed 33 +2265 -606

james commented 2026-07-28 13:26:05 +00:00

Owner

Copy link

Rearchitects the cluster to a thin-node / authoritative-main model
(ADR 0001 — docs/adr/0001-thin-node-cluster-architecture.md; build plan in
docs/plans/thin-node-implementation-plan.md). Nodes become headless,
database-free agents; main owns all state and the UI.

Topology: node → main only over signed HTTPS (main can't dial a node). Ships as
a single unit; validated on a local 2-node harness before merge.

Phases (each independently reviewable)

• A — Headless node: role=node runs no web/UI server; Docker healthcheck
  falls back to process liveness.
• B — Unified adaptive check-in: one keyless POST /api/cluster/checkin
  returns queued commands + a next_poll_after hint (idle 15s / active 1s),
  keep-alive connection, single agent loop. No held connections (scales wide).
• C — Event push for all backups (start/complete/fail) via a
  BackupMonitor reporter hook; bounded EventSpool (JSONL) buffers through a
  main outage and replays on reconnect; idempotent ingest keyed by event_uuid.
• D — Main owns backup manifests (node_files): node reports its full
  manifest on (re)connect + daily, deltas via events; UI browses main's DB
  (instant), ?refresh=1 forces a live re-fetch. Download/restore stay live.
• E — Cryptographic identity + enrollment:
  • Node self-generates an EC (P-256) keypair → identity.json; signs every
    request (X-Bak-*, UTC-bound date, ±300s replay window).
  • First contact from an unknown key → pending + 401; operator approves the
    key on the Nodes page (fingerprint + first-seen IP + time). Optional join
    secret gates the pending list; BT_CLUSTER_AUTO_APPROVE for hands-off.
    "Approve all" (shown only when ≥1 pending); revoke.
  • Node is DB-free (role-aware DI); main gets SQLite WAL; dead bearer /
    long-poll code removed.

Invariant

Events are the fast path; a full manifest reconcile is the authoritative
backstop — so a lossy/bounded spool self-heals and dropped events never diverge.

Tests

Full suite green — 1112 examples, 0 failures. New coverage for signing,
identity, spool, adaptive check-in, idempotent ingest, manifest mirror, DB-free
DI, and the signed enrollment/approval endpoints.

Validation & rollout (post-merge)

• docker-compose.cluster.yml + docs/thin-node-harness-runbook.md — run on an
  isolated Docker context (the node backs up every labelled container on the
  daemon).
• On deploy, citadel re-enrolls (old SQLite identity ignored, keypair
  generated) → appears pending → approve once. Backout = revert the merge;
  citadel's old data is untouched.

🤖 Generated with Claude Code

Rearchitects the cluster to a **thin-node / authoritative-main** model (ADR 0001 — `docs/adr/0001-thin-node-cluster-architecture.md`; build plan in `docs/plans/thin-node-implementation-plan.md`). Nodes become headless, database-free agents; main owns all state and the UI. Topology: node → main only over signed HTTPS (main can't dial a node). Ships as a single unit; validated on a local 2-node harness before merge. ## Phases (each independently reviewable) - **A** — Headless node: `role=node` runs no web/UI server; Docker healthcheck falls back to process liveness. - **B** — Unified adaptive check-in: one keyless `POST /api/cluster/checkin` returns queued commands + a `next_poll_after` hint (idle 15s / active 1s), keep-alive connection, single agent loop. No held connections (scales wide). - **C** — Event push for **all** backups (start/complete/fail) via a `BackupMonitor` reporter hook; bounded `EventSpool` (JSONL) buffers through a main outage and replays on reconnect; idempotent ingest keyed by `event_uuid`. - **D** — Main owns backup manifests (`node_files`): node reports its full manifest on (re)connect + daily, deltas via events; UI browses main's DB (instant), `?refresh=1` forces a live re-fetch. Download/restore stay live. - **E** — Cryptographic identity + enrollment: - Node self-generates an EC (P-256) keypair → `identity.json`; **signs every request** (`X-Bak-*`, UTC-bound date, ±300s replay window). - First contact from an unknown key → `pending` + 401; operator approves the **key** on the Nodes page (fingerprint + first-seen IP + time). Optional join secret gates the pending list; `BT_CLUSTER_AUTO_APPROVE` for hands-off. "Approve all" (shown only when ≥1 pending); revoke. - Node is **DB-free** (role-aware DI); main gets SQLite **WAL**; dead bearer / long-poll code removed. ## Invariant Events are the fast path; a full manifest reconcile is the authoritative backstop — so a lossy/bounded spool self-heals and dropped events never diverge. ## Tests Full suite green — **1112 examples, 0 failures**. New coverage for signing, identity, spool, adaptive check-in, idempotent ingest, manifest mirror, DB-free DI, and the signed enrollment/approval endpoints. ## Validation & rollout (post-merge) - `docker-compose.cluster.yml` + `docs/thin-node-harness-runbook.md` — run on an **isolated** Docker context (the node backs up every labelled container on the daemon). - On deploy, **citadel re-enrolls** (old SQLite identity ignored, keypair generated) → appears **pending** → approve once. Backout = revert the merge; citadel's old data is untouched. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

james added 11 commits 2026-07-28 13:26:05 +00:00

feat(cluster): ADR 0001 + Phase A1 — headless node (no web server) ... 5a8de85e3e

ADR 0001 records the thin-node/authoritative-main rearchitecture: HTTPS
request/response + keep-alive + adaptive interval, keypair identity with
trust-on-first-use approval, the events-are-fast-path / reconcile-is-backstop
invariant, and a phased A–E plan.

Phase A1: a cluster node (role=node) no longer starts the Sinatra web/UI
server even when BT_HEALTH_SERVER_ENABLED is set — it runs as a headless
agent. main/standalone still serve the UI. The Docker healthcheck falls
through to process-liveness for a node so the container stays healthy without
the HTTP endpoint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: full implementation plan for the thin-node rearchitecture ... 88f4116af2

Concrete build plan companion to ADR 0001: per-phase data-model/endpoint/
agent changes (A–E), a local 2-node docker-compose validation harness, and the
single-unit rollout + citadel re-enrollment migration. Ships as a unit off this
branch after harness validation — no incremental prod deploys.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(cluster): Phase B — unified adaptive check-in cycle ... 6f75aa0762

Collapse the node agent's two loops (heartbeat + command long-poll) into a
single node-initiated check-in: the node POSTs its status snapshot and receives
any queued commands plus an adaptive next_poll_after in the same response — no
held connection. The socket is kept alive between check-ins (skips the TCP/TLS
handshake on the steady-state path).

- New endpoint POST /api/cluster/nodes/:id/checkin (main): record status, drain
  pending commands (non-blocking), return commands + next_poll_after.
- Coordinator: commands_for (non-blocking drain) + next_interval (fast cadence
  right after handing out work, else idle).
- Config: BT_CLUSTER_POLL_IDLE (15s) / BT_CLUSTER_POLL_ACTIVE (1s).
- Agent: single checkin_loop with a keep-alive Net::HTTP connection and
  reconnect-on-drop. Legacy heartbeat + command long-poll endpoints kept for
  compat (removed in Phase E).

Per ADR 0001. Full suite green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(cluster): Phase C — event push for all backups + spool + replay ... 15c3754d82

Every backup (scheduled and command-triggered) now reports start/complete/fail
to main out-of-band, so the UI reflects node backups within ~1s instead of
waiting for a heartbeat.

- BackupMonitor gains a reporter hook (single injection point for both backup
  paths); it's set only on a node by the agent. Reporting never breaks a backup.
- EventSpool: bounded, append-only JSONL buffer of pending events. Node keeps
  backing up through a main outage; events replay on reconnect. Lossy by design
  (the Phase D manifest reconcile is the authoritative backstop).
- Agent: report() stamps an event_uuid, spools, and pushes immediately; the
  check-in loop flushes the spool each cycle (ordered, one flush at a time).
  Drops the now-redundant per-run backup event push.
- Main ingest is idempotent: backup_history gains an event_uuid column + UNIQUE
  index (guarded migration; NULLs stay distinct so main's own backups are
  unaffected); INSERT OR IGNORE dedupes replays. New backup_started event is
  transient (no history row). Config: BT_CLUSTER_SPOOL_PATH.

Per ADR 0001. Full suite green (1099 examples).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(cluster): Phase D — main owns backup manifests, instant browsing ... 91a599bc09

Remote backup browsing now reads main's mirror instead of round-tripping to the
node, so it's instant.

- New node_files table + NodeFilesRepository: main's authoritative mirror of each
  node's on-disk backups. replace_for_node reconciles (adds + deletions);
  upsert/delete_file are the event fast path; list_for_node rebuilds the
  list_backups-shaped payload the UI already expects.
- Node agent sends its full manifest on (re)connect and at least daily
  (POST /api/cluster/nodes/:id/manifest); main mirrors it.
- Fast-path deltas: a backup_completed event upserts the new file into the cache;
  a successful remote delete drops it — so the listing stays fresh between
  manifests (reconcile is the backstop).
- GET /api/nodes/:id/files reads the cache for a remote node (self-node still
  reads local disk); ?refresh=1 forces a live re-fetch and refreshes the cache.
  Nodes page gains a "Refresh from node" button.

Per ADR 0001. Full suite green (1108 examples).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(cluster): Phase E1 — EC keypair identity + request signing ... e3cec86cca

Cryptographic identity foundation for the new enrollment model (ADR 0001).

- ClusterSignature: sign/verify over "METHOD\nPATH\nX-Bak-Date\nSHA256(body)"
  with ECDSA P-256 (stdlib OpenSSL, no native gem). Fingerprint = SHA256 of the
  public-key DER (the cluster key id). Timestamps are UTC on both sides — dates
  are parsed to UTC before the ±300s freshness/replay check, so a zone-qualified
  stamp is compared as an absolute instant.
- NodeIdentity: node-side keypair persisted as identity.json (0600, atomic
  write); stable across restarts; stores the numeric node_id main assigns after
  approval. Replaces the SQLite cluster_identity row on a node.

Verified: sign→verify round trip, tamper (body/path) rejection, wrong-key
rejection, stable fingerprint, UTC skew handling.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(cluster): Phase E2 — signature auth, enrollment & approval ... 0da08b94e0

Replace bearer-token enrollment with self-enrolling keypair identity (ADR 0001).

Node (agent):
- Signs every cluster request with its EC key (NodeIdentity); no bearer token.
- Single keyless POST /api/cluster/checkin serves enrollment and steady state;
  on approval the response hands the node its node_id (persisted to identity.json).
- Unknown/pending/revoked key -> 401: the node logs once and keeps its own
  backup schedule, retrying on a jittered interval. Events/manifests wait until
  approved. File upload stays token-authed (signature-exempt; not body-hashed).

Main:
- Cluster before-filter verifies the signature (UTC date freshness ±300s),
  derives identity from the presented key, and enrolls unknown keys as pending.
- Optional join secret (reuses BT_CLUSTER_JOIN_TOKEN, now optional) gates the
  pending list; BT_CLUSTER_AUTO_APPROVE auto-approves valid-secret nodes.
- node_repository: find_by_fingerprint / enroll_pending / approve /
  approve_all_pending / reject / revoke; `status` repurposed to membership
  (pending|approved|revoked) via guarded migration (old 'online' -> 'approved');
  heartbeats no longer touch status. New columns: public_key, first_seen_ip/at,
  approved_at/by.
- Approval UI endpoints: GET /api/nodes/pending, POST .../approve|reject|revoke,
  POST /api/nodes/approve-all (session-authed, write role).

Config: BT_CLUSTER_IDENTITY_PATH, BT_CLUSTER_AUTO_APPROVE; join token optional.
Removed the register/heartbeat/per-id-checkin/commands machine endpoints.

Full suite green (1116 examples).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(cluster): Phase E2c — node approval UI on the Nodes page ... e7984d8132

Surfaces keypair enrollment to the operator:
- "Nodes awaiting approval" section, shown only when >=1 node is pending, with
  each pending node's key fingerprint, first-seen IP, and first-seen time (the
  operator approves a key, not the self-reported name). Per-node Approve/Reject
  plus an "Approve all" button.
- Revoke action on approved non-main node cards.

Wired to POST /api/nodes/:id/approve|reject|revoke and /api/nodes/approve-all.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(cluster): Phase E3 — node is DB-free (role-aware DI) ... 225f94795e

A cluster node now instantiates zero SQLite: the DI returns nil for the three
repositories a node's runtime would otherwise pull — backup_history, alerts, and
rotation_audit. Their consumers already tolerate nil (BackupMonitor falls back to
in-memory tracking, BackupRotation logs audit only when present,
BackupManagementService guards the history repo). A node keeps only identity.json
+ the event spool; history/alerts live on main via events.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

refactor(cluster): Phase E4 — main hardening + drop dead legacy code ... bf2c1d4145

- SQLite WAL mode on main (set at DB init) so UI reads and event/heartbeat
  writes don't contend as node count grows.
- Remove the now-dead bearer/long-poll machinery: NodeRepository#register_or_update,
  ClusterCoordinator#register / #await_commands / #wait_for_commands and the
  per-node condvar + long-poll semaphore (commands are delivered on the node's
  next check-in; only the result path still blocks). HealthCheckServer#bearer_token
  and #valid_join_token?.

Note: event-write batching/coalescing is deferred — not worth the complexity at
current scale; WAL covers the contention concern. Revisit with a Postgres path if
the fleet grows into the hundreds (see ADR 0001).

Full suite green (1112 examples).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs(cluster): 2-node validation harness + runbook ... 662593563b

docker-compose.cluster.yml brings up main + a headless node + two labelled DBs;
the runbook walks enroll->approve->backup->instant-browse->download/restore->
resilience(spool replay)->revoke->DB-free spot check, plus the join-secret/
auto-approve variant and the ship + citadel re-enrollment steps.

Not run against this workstation's Docker daemon (the node would discover live
containers); to be run on an isolated context before shipping. The signing
round-trip itself is already covered by cluster_signature_spec + the signed
request specs (both sides share ClusterSignature).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

james merged commit 1154e3dceb into main 2026-07-28 13:36:24 +00:00

james referenced this pull request from a commit 2026-07-28 13:36:25 +00:00

Merge pull request 'feat: thin-node cluster rearchitecture (headless nodes, keypair enrollment, instant browsing)' (#6) from feat/thin-node-cluster into main

james referenced this pull request 2026-07-28 13:59:05 +00:00

fix: approval-route ordering (missed by #6) + remove-offline-node action #7

james referenced this pull request from a commit 2026-07-28 14:00:44 +00:00

Merge pull request 'fix: approval-route ordering (missed by #6) + remove-offline-node action' (#7) from feat/remove-offline-node into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

wontfix

This won't be fixed

No labels

bug

duplicate

enhancement

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

james/baktainer!6

No description provided.