Skip to content

Rotating the Internal CA

Rotating the CA changes the trust anchor for the whole fleet, in both directions (the CA validates the server cert and every client cert). It is therefore a coordinated, fleet-wide ceremony, rare in practice (the CA is 10-year), used either on its natural expiry or as a break-glass response to a confirmed CA-key compromise. It is separate from the routine, no-coordination server-leaf rotation.

It uses a separate set of buttons on the Hub Configuration page (Stage New CA / Activate New CA / Retire Old CA) plus a Re-issue All Client Certs action. The server-leaf buttons are untouched.

Warning

New-site enrollment and bundle generation are blocked while a CA rotation is in progress (ca_rotation_phase ≠ NONE). Complete or finish the rotation before adding clients.

The phases (tracked in hub_listener_config.ca_rotation_phase: NONE → STAGED → ACTIVATED → back to NONE after Retire):

CA rotation trust states by phase: the server leaf flips from the old CA to the new CA at Activate; the listener trusts both CAs for incoming client certs continuously from Stage through Retire; each site's client cert migrates from old to new at Re-issue.

  1. Stage New CA generates a brand-new CA keypair (never reuses the old key) into internal_ca_cert_id_next, restarts the listener so it trusts both CAs for incoming client certs, and broadcasts TRUST_ADDITIONAL_CA_CERT to every connected site. Each engine adds the new CA to its server-trust set and ACKs. The server leaf is unchanged (still old-CA-signed) and no client certs change yet. Per-site progress is tracked on hub_site_status.
  2. Activate New CA is gated. It blocks if any enabled + connected site has not ACKed trust for the new CA, because that site could not validate the new server leaf and would be cut off. It warns and lists enabled + offline stragglers. Promotes the staged CA to active, auto-generates a new server leaf signed by the new CA, swaps it in, and restarts. The Hub keeps trusting the old CA for client certs through this step. Every site is still on an old-CA client cert, so dropping it here would lock the whole fleet out.
  3. Re-issue All Client Certs. Now that the new CA is active, this cycles every enabled site so each client cert is reissued under the new CA (a client-cert cycle). Sites cycle over their still-trusted old-CA tunnels and reconnect on the new cert. Run it after Activate. It is rejected during the staging phase, because a re-issue then would sign certs with the old CA. (Routine rotations may instead let certs migrate lazily at their normal renewal window; break-glass forces it now. Also usable outside a rotation as a general fleet re-issue.)
  4. Retire Old CA is gated, and blocks if any enabled + connected site has not both (a) ACKed trust for the new CA and (b) migrated to a new-CA client cert, and warns and lists any enabled + offline stragglers. When you proceed, the Hub stops trusting the old CA for client certs, broadcasts TRUST_REMOVE_CA_CERT, each engine drops and deletes the old CA and ACKs, and the Hub deletes the old CA. Enrollment unblocks once Retire completes.

Sites offline across the whole rotation must be re-enrolled

A site that stays offline from before Stage until after Retire never migrates its client cert to the new CA, and once the old CA is retired the Hub no longer trusts its old-CA client cert. Such a site cannot reconnect and must be re-enrolled (a fresh bundle). Retire warns and lists offline stragglers before you proceed for exactly this reason. Bring stragglers online to migrate them, or accept that they need re-enrollment.

Throughout a CA rotation, the Sites Dashboard shows a per-site readiness indicator whose meaning is phase-aware. While the new CA is staged (before Activate), green = the site has confirmed trust for the new CA (ready to activate). Once the new CA is activated (before Retire), green tightens to mean the site both trusts the new CA and has migrated its client cert to it, i.e. ready to retire. A site that trusts the new CA but has not yet re-issued its client cert stays red (or grey, if offline) until it migrates. In both phases red = connected but not yet ready (or errored) and grey = offline and not yet ready. This column is shown only during a CA rotation and disappears once the old CA is retired.

The broadcasts at Stage and Retire are re-sent to connected-but- unconfirmed sites by a leader-elected worker, and replayed when a disconnected site reconnects (handleHello), so the ceremony tolerates sites cycling offline mid-rotation.