feat: multi-node clustering (register, heartbeat, remote trigger, cluster dashboard) #4

Merged

james merged 3 commits from feat/clustering-phase1 into main

2026-07-24 17:59:50 +00:00

james commented

2026-07-24 17:58:21 +00:00

Owner

Turns Baktainer from a single instance into a coordinated cluster. A main instance runs the UI and aggregates status across servers; node instances register with it, back up their own Docker host on their own labels/schedule, and report up. Main observes and can trigger a backup on any node — it never reaches into a node.

All communication is node-initiated (register → heartbeat → long-poll for commands → push events), so nodes may sit behind NAT. TLS is expected from a reverse proxy (Traefik); the app authenticates with a shared join token and per-node bearer tokens over it.

Delivered in three self-contained commits:

Phase 1 — foundation (`6d4f155`)

Config: BT_CLUSTER_ROLE (standalone/main/node), BT_CLUSTER_JOIN_TOKEN, BT_MAIN_URL, BT_NODE_NAME, BT_HEARTBEAT_INTERVAL + validation.
nodes + cluster_identity tables; backup_history gains node_id/node_name (guarded PRAGMA/ALTER shim — no migration framework).
Per-node bearer tokens stored only as SHA256, secure_compare, revocation + auto re-register on 401. Authorization: Bearer parsing scoped to /api/cluster/*, above the auth_enabled? short-circuit.
Node agent (register-on-boot with backoff + heartbeat loop, isolated from the local cron); main registers itself as a self-node via in-process heartbeat.
/nodes page + GET /api/nodes.

Phase 2 — command channel + remote trigger (`9824b89`)

node_commands queue; queue_command wakes a per-node condvar so a parked long-poll delivers near-instantly (lost-wakeup-safe). Concurrent polls capped by a semaphore (:busy→204).
Puma pool sized explicitly (BT_HEALTH_MAX_THREADS=16; default was 5) since each long-poll holds a thread. New: BT_COMMAND_POLL_TIMEOUT=25, BT_MAX_LONGPOLL=12.
GET …/commands (long-poll 200/204), POST …/events (records node-tagged backups + command results + alerts).
Node executes run_backup via the existing run_backup_now, pushes backup + result events.
"Run backup now" per node; POST /api/nodes/:id/run-backup (session auth, write role; self-node runs locally).

Phase 3 — cluster dashboard (`f712d1d`)

NodeRepository#cluster_summary, BackupHistoryRepository#recent_for_node; snapshot gains a containers[] list.
GET /api/nodes/overview (aggregate tiles + success rate), GET /api/nodes/:id (parsed snapshot + alerts), GET /api/nodes/:id/backups.
/nodes reworked into overview tiles + clickable grid + per-node drill-down (status/disk/containers/recent-backups/alerts).

Testing

Full suite 1061 examples, 0 failures, 3 pending. Added unit + request specs for repos, coordinator, agent (webmock), and every endpoint; plus three in-process end-to-end smokes (one per phase) exercising the real stack — register → heartbeat → command long-poll → execute → node-tagged history → overview/drill-down. README documents the roles and all new env vars.

Out of scope (future)

Remote file browse/download/restore proxied through main.

🤖 Generated with Claude Code

Turns Baktainer from a single instance into a coordinated cluster. A **main** instance runs the UI and aggregates status across servers; **node** instances register with it, back up their own Docker host on their own labels/schedule, and report up. Main observes and can trigger a backup on any node — it never reaches into a node. All communication is **node-initiated** (register → heartbeat → long-poll for commands → push events), so nodes may sit behind NAT. TLS is expected from a reverse proxy (Traefik); the app authenticates with a shared join token and per-node bearer tokens over it. Delivered in three self-contained commits: ### Phase 1 — foundation (`6d4f155`) - Config: `BT_CLUSTER_ROLE` (`standalone`/`main`/`node`), `BT_CLUSTER_JOIN_TOKEN`, `BT_MAIN_URL`, `BT_NODE_NAME`, `BT_HEARTBEAT_INTERVAL` + validation. - `nodes` + `cluster_identity` tables; `backup_history` gains `node_id`/`node_name` (guarded `PRAGMA`/`ALTER` shim — no migration framework). - Per-node **bearer tokens stored only as SHA256**, `secure_compare`, revocation + auto re-register on 401. `Authorization: Bearer` parsing scoped to `/api/cluster/*`, above the `auth_enabled?` short-circuit. - Node agent (register-on-boot with backoff + heartbeat loop, isolated from the local cron); main registers itself as a self-node via in-process heartbeat. - `/nodes` page + `GET /api/nodes`. ### Phase 2 — command channel + remote trigger (`9824b89`) - `node_commands` queue; `queue_command` wakes a per-node condvar so a parked long-poll delivers near-instantly (lost-wakeup-safe). Concurrent polls capped by a semaphore (`:busy`→204). - Puma pool sized explicitly (`BT_HEALTH_MAX_THREADS=16`; default was 5) since each long-poll holds a thread. New: `BT_COMMAND_POLL_TIMEOUT=25`, `BT_MAX_LONGPOLL=12`. - `GET …/commands` (long-poll 200/204), `POST …/events` (records node-tagged backups + command results + alerts). - Node executes `run_backup` via the existing `run_backup_now`, pushes backup + result events. - "Run backup now" per node; `POST /api/nodes/:id/run-backup` (session auth, write role; self-node runs locally). ### Phase 3 — cluster dashboard (`f712d1d`) - `NodeRepository#cluster_summary`, `BackupHistoryRepository#recent_for_node`; snapshot gains a `containers[]` list. - `GET /api/nodes/overview` (aggregate tiles + success rate), `GET /api/nodes/:id` (parsed snapshot + alerts), `GET /api/nodes/:id/backups`. - `/nodes` reworked into overview tiles + clickable grid + per-node drill-down (status/disk/containers/recent-backups/alerts). ## Testing Full suite **1061 examples, 0 failures, 3 pending**. Added unit + request specs for repos, coordinator, agent (webmock), and every endpoint; plus three in-process end-to-end smokes (one per phase) exercising the real stack — register → heartbeat → command long-poll → execute → node-tagged history → overview/drill-down. README documents the roles and all new env vars. ## Out of scope (future) Remote file browse/download/restore proxied through main. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

james added 3 commits

2026-07-24 17:58:21 +00:00

feat(cluster): Phase 1 — node registration, heartbeat & nodes dashboard 6d4f1556cc

Introduces multi-instance clustering. A 'main' instance runs the UI and
aggregates status; 'node' instances register with main and report up. Each
instance still backs up only its own Docker host on its own schedule/labels —
main observes (Phase 2 will add remote triggers). Communication is
node-initiated (register → heartbeat), so nodes need no inbound reachability;
TLS is expected to be terminated by a reverse proxy in front of main.

- Config: BT_CLUSTER_ROLE (standalone|main|node), BT_CLUSTER_JOIN_TOKEN,
  BT_MAIN_URL, BT_NODE_NAME, BT_HEARTBEAT_INTERVAL + predicates + validation.
- Schema: nodes + cluster_identity tables; backup_history gains node_id/node_name
  (via a guarded PRAGMA/ALTER shim since there's no migration framework).
- NodeRepository (main): register/rotate/authenticate — stores only the SHA256
  of each per-node bearer token, secure_compare on auth, revocation, denormalized
  heartbeat snapshot. NodeIdentityRepository (node/self): persists uuid + token so
  restarts reuse identity.
- NodeStatusSnapshot: shared status builder reused by node agent + main self-node.
- ClusterCoordinator (main): registration, heartbeat ingestion, in-process
  self-node heartbeat. ClusterAgent (node): background thread — register-on-boot
  with backoff, heartbeat loop, 401 → clear token + re-register; never disturbs
  the local backup cron.
- Endpoints: POST /api/cluster/register (join-token auth), POST
  /api/cluster/nodes/:id/heartbeat (bearer). before-filter gains an Authorization
  Bearer branch scoped to /api/cluster/*, above the auth_enabled? short-circuit.
  UI feed GET /api/nodes + /nodes page (session auth, main only), nav link.
- Wiring: DI singletons + Runner start_cluster/stop_cluster lifecycle.
- Tests: repo/agent/coordinator unit specs + cluster request specs (full suite
  1034 examples, 0 failures) and an in-process register→heartbeat→/api/nodes E2E.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(cluster): Phase 2 — command long-poll, remote trigger & event reporting 9824b896f7

Adds the bidirectional control path on top of Phase 1's registration/heartbeat,
keeping everything node-initiated.

- node_commands table + NodeCommandRepository (enqueue/pending/dispatch/result/expire).
- ClusterCoordinator command queue: queue_command wakes a per-node condition
  variable so a parked long-poll delivers near-instantly; await_commands is
  lost-wakeup-safe (DB re-check under the same mutex as the broadcast). Concurrent
  long-polls are capped by a semaphore — over the cap returns :busy → 204 so a
  node simply re-polls, protecting the UI from thread starvation.
- Puma pool sized explicitly (BT_HEALTH_MAX_THREADS=16) in both the embedded and
  standalone runners, since each long-poll holds a thread for its duration
  (default max was 5). New config: BT_COMMAND_POLL_TIMEOUT=25, BT_MAX_LONGPOLL=12.
- Endpoints (bearer): GET /api/cluster/nodes/:id/commands (long-poll, 200/204),
  POST /api/cluster/nodes/:id/events (command_result → mark_result;
  backup_completed/failed → backup_history tagged with node_id/node_name +
  node.last_backup; alert → performance_alerts). record_backup extended with the
  node columns (local backups pass nil = local/main).
- Node agent gains a second thread: long-poll → run_backup via run_backup_now →
  push backup + command_result events; report_status → immediate heartbeat;
  unknown → failed result. Isolated from the local cron with broad rescue.
- UI: POST /api/nodes/:id/run-backup (session auth, write role; main self-node
  runs locally instead of queueing) + "Run backup now" button on each node card.
- Tests: command repo/coordinator/agent unit specs + endpoint request specs
  (full suite 1053 examples, 0 failures) and an in-process E2E covering
  queue → long-poll → execute → events → node-tagged history → 204-at-timeout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(cluster): Phase 3 — cluster overview & per-node drill-down dashboard f712d1d269

Reworks /nodes from a flat card list into a cluster dashboard.

- Backend: NodeRepository#cluster_summary aggregates online/offline, running
  backups and free disk across nodes; BackupHistoryRepository#recent_for_node
  filters by node (nil = local/main via node_id IS NULL); NodeStatusSnapshot now
  includes a lightweight containers[] list (name + engine) for drill-down.
- Endpoints (session-authed, main only): GET /api/nodes/overview (aggregate tiles
  + cluster backup success rate from get_metrics_summary), GET /api/nodes/:id
  (node + parsed snapshot + alerts matched by "[name]" prefix), GET
  /api/nodes/:id/backups. /overview is declared before /:id so it isn't matched
  as an id.
- UI: overview stat tiles + clickable node grid; clicking a card opens a detail
  view (status, disk, backup-container list, recent backups table, alerts) with a
  back button; "Run backup now" on both card and detail.
- Tests: overview/detail/backups request specs + repo specs (full suite 1061
  examples, 0 failures) and an in-process Phase 3 E2E (aggregates, drill-down
  snapshot/containers/backups, NULL-vs-node filtering, route ordering).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

james merged commit 36ab97a38c into main

2026-07-24 17:59:50 +00:00

james referenced this pull request from a commit

2026-07-24 17:59:52 +00:00

Merge pull request 'feat: multi-node clustering (register, heartbeat, remote trigger, cluster dashboard)' (#4) from feat/clustering-phase1 into main