SSL and PKI¶

The Hub operates its own internal certificate authority, separate from any public CA your organization may use elsewhere. That internal CA signs:

The tunnel listener's server certificate (what remote QIEs see when they connect).
Every enrolled site's client certificate (what the Hub sees during the mTLS handshake).

The internal CA is the root of trust for the entire site-to-Hub relationship. The system trust store is not consulted on the tunnel side. Both ends validate against the internal CA's chain, not a browser-trusted public chain.

This is by design. Browser-trusted certs are intended for human-to-server authentication, while the tunnel is server-to-server with a known fixed peer. Anchoring trust at your own CA eliminates several attack classes that public CA trust would re-introduce: rogue intermediate CAs, mis-issuance by any of the hundreds of CAs in the public store, downstream-trust takeover).

Trust model: certificate authority, not pinned leaf¶

Each remote QIE trusts the internal CA certificate, and validates the Hub's server certificate with a full chain check plus a hostname/SAN match against the configured Hub host. It does not pin the Hub's leaf server certificate byte-for-byte.

The practical consequence drives everything else on this page:

Any server certificate signed by the internal CA validates automatically. A routine server-cert rotation needs zero coordination with the fleet. You generate a new CA-signed leaf, activate it, and sites validate it against the CA they already trust.
A site that is offline during a server-cert rotation simply validates the new leaf the next time it reconnects. There is no "flag day" and no site to abandon.
The one event that does require fleet coordination is rotating the CA itself, because that changes the trust anchor. That is a rare (~10-year) ceremony with its own controls. See Rotating the internal CA.

Two leaf certificates, both signed by the internal CA¶

Cert	Presented by	Validated by
Tunnel server cert	Hub tunnel listener (on `/tunnel` and `/register`)	Each remote QIE, by chain-validating against the internal CA cert it trusts (`hub_registration.pinned_ca_cert_id`) plus a hostname/SAN check
Tunnel client cert	Remote QIE	Hub listener's `HubSiteTrustManager`: chain-validates against the internal CA (this enforces expiry), then matches the cert serial against the live `hub_site` row

The dashboard's TLS cert (the cert the main Jetty presents to operator browsers) is unrelated to either of these. That one is conventional browser-trusted TLS, covered in the Hub Install Guide.

Note

In Hub mode the internal CA and these two leaf certs are the only entries in the global zone's SSL Certs store. There is no separate "engine" PKI to keep straight.

Initialize Internal CA¶

When you first install a Hub there is no internal CA. The button is on the Hub Configuration page, on the Internal CA Cert row.

Clicking Initialize Internal CA:

Generates a fresh self-signed root certificate + private key (RSA-4096, signed SHA256withRSA, lifetime 10 years by default, DEFAULT_CA_VALID_DAYS = 3650).
Stores the cert and key in the global zone's SSL Certs / Keys tables, named Hub Internal CA plus a short unique suffix (e.g. Hub Internal CA 9e86120e).
Sets hub_listener_config.internal_ca_cert_id to the new cert's ID, which the Internal CA Cert field then shows.

You should only need to do this once per Hub deployment. The 10-year lifetime is deliberately long so CA rotation is a rare event. The internal CA's private key is the most security-critical artifact in the Hub: it signs the server cert, every client cert, and the enrollment bundle's JWS signature. Its compromise allows an attacker to forge a client cert for any site identifier. Treat it accordingly; see Security Guidance.

Warning

Initialize Internal CA always creates a new CA and errors if one already exists. There is no "import an existing CA" path: the Internal CA Cert field is read-only, and Save never changes it, so a pre-existing CA (e.g. an enterprise PKI) cannot be selected on the Hub Configuration page.

Rotate Server Cert¶

The tunnel listener needs a server cert signed by the internal CA. Because every site trusts the CA, not the leaf, rotating the listener's server certificate is a local operation. A new leaf signed by the same CA validates against the chain sites already trust, so the rotation pushes no trust frames to the fleet and needs no site coordination.

The Rotate Server Cert button is on the Hub Configuration page, beside the (read-only) Server Cert row. On a new Hub with no server certificate the same button reads Generate Server Cert: it creates and saves the first certificate (steps 1 to 3 below) without a confirmation and without restarting the listener, and Restart Listener then starts the listener with it. Once a certificate exists, Rotate Server Cert does the whole rotation in one step after a confirmation:

Generates a fresh RSA keypair and a CSR with CN = the configured Hostname.
Signs it with the internal CA, stamping a serverAuth extended key usage and a dNSName SAN matching the Hostname. (Both are required: remote QIEs run real chain validation, so a leaf without serverAuth EKU or a matching SAN would be rejected.) Lifetime is hub_listener_config.cert_lifetime_days (default 365).
Stores the cert + key in the global zone's SSL Certs / Keys tables and points hub_listener_config.server_cert_id at the new leaf.
Trial-builds the listener with the new leaf; if the cert or its key fails to load, the rotation is aborted and the current listener keeps running (no outage).
Restarts the tunnel listener (on every HA node) so it presents the new leaf. Existing connections drop and immediately reconnect, validating the new leaf against the internal CA. The Sites Dashboard flashes a brief wave of red and settles back to green. The superseded leaf's cert + key are then deleted.

The Server Cert and Internal CA Cert fields are read-only. The live certificate can only change through Generate Server Cert or Rotate Server Cert (the server leaf), Initialize Internal CA, or the CA-rotation ceremony, so Save can never swap it out from under the fleet.

Warning

Rotation restarts the listener, which momentarily disconnects every site, including anyone actively proxied into a site. This is why it is a deliberate, manual operator action and is never automated. Schedule it for a maintenance window.

Note

Bringing your own externally-issued server certificate, one signed by a different CA (a corporate or public PKI), is a planned enhancement. Because that changes the trust anchor, it uses a staged trust-add before activation (like a CA rotation), not this one-step same-CA rotation.

Server-cert expiry is therefore guarded by an alert rather than automation. See Expiry alerts.

Client certificate lifecycle¶

Client certs are issued during enrollment, when the Hub's /register handler signs the CSR a remote QIE submits. A site's cert carries CN = its site_identifier UUID.

Setting	Default	Configurable in
Lifetime	365 days	Hub Configuration -> Cert Lifetime (days)
Renewal threshold	30 days	Hub Configuration -> Renewal Threshold (days)

A 1-year lifetime ages a stolen cert out within a reasonable window; the 30-day threshold gives the Hub runway to retry the cycle over transient network outages before the cert actually expires.

Hub-initiated cycling (automatic)¶

Unlike earlier builds, the engine no longer schedules its own renewal. There is a single scheduler, on the Hub, so the operator has one place to see and control the fleet's cert state.

The Hub's (HA-leader-elected) scheduler scans hub_site.client_cert_expires_at against the renewal threshold. For each connected, due site it runs a cycle over the existing tunnel:

Hub sends CYCLE_CLIENT_CERT_REQUEST to the site.
The engine generates a fresh keypair + CSR and replies with RENEW_CSR_REQUEST.
The Hub signs it (CN forced to the site_identifier) and stores the new cert as client_cert_pem_next / client_cert_serial_next, so the trust manager now accepts the site's old OR new cert during the overlap. It replies RENEW_CSR_RESPONSE.
The engine installs the new cert + key, swaps its registration to point at it, reconnects presenting the new cert, deletes the old cert + key, and sends a completion ACK asserting new loaded AND old deleted.
Only on that ACK does the Hub promote client_cert_pem_next to current, stop accepting the old serial, and mark the cycle complete.

The cycle is self-healing and lockout-safe:

If the engine installs the new cert but its completion ACK never arrives (for example it crashes right after installing), the next reconnect recovers the cycle. The Hub sees that the presented cert matches the staged serial and promotes on sight, without waiting for the ACK.
The scheduler re-checks on its interval. A cycle that stalls (no ACK and no failure) past a short window is treated as stuck and retried, but only once the site is confirmed back on its current cert. The Hub never re-issues (and thereby stops accepting) a staged cert the engine may already be relying on, so a retry cannot lock a site out.

Accepted limitation

Expiry is enforced at the handshake, and cycling travels over the tunnel. A site that is offline continuously from before its renewal window until past the cert's expiry cannot be renewed in-band and must be re-enrolled. With the default 365-day lifetime and 30-day threshold, that is roughly 11 months of continuous offline before it bites. This is accepted by design. There is deliberately no grace path, because a grace path would let a stolen-but-expired cert regain a foothold.

Rotating one site's cert on demand (manual)¶

You do not have to wait for the renewal window. On the Sites Dashboard, select a site and click View Details, then go to the Certificate tab. The Rotate Certificate button there triggers the exact same cycle described above for that one site, immediately.

The button is enabled only when the site is enabled and connected. the cycle is delivered over the live tunnel, so an offline site cannot take the request. Use it after a suspected key exposure on a single site, or to confirm the cycle path during commissioning.

Revoking a certificate¶

Cert revocation force-closes any open tunnel for the affected site within ~15 seconds and prevents future handshakes with that cert. It is the primary control for responding to a suspected cert compromise.

The hub_revoked_cert table is append-only. Revocations are never modified. Each row records:

Field	Meaning
`cert_serial`	The serial being revoked (UNIQUE)
`revoked_at`	When
`revoked_by_username`	Which Hub user
`reason`	`OPERATOR_ACTION`, `ROTATION`, `SUSPECTED_COMPROMISE`, or `DECOMMISSION`
`details`	Optional free-text reason

The Hub trust manager consults the revocation cache on every handshake and every heartbeat, so a revocation force-closes an already-open tunnel without waiting for the natural handshake retry.

Reason	When to use
`OPERATOR_ACTION`	Routine revocation. The site is being decommissioned or replaced
`ROTATION`	Recorded automatically when a client-cert cycle supersedes the old cert
`SUSPECTED_COMPROMISE`	Use when you have evidence the private key has leaked. Treated as a security incident in audit logs
`DECOMMISSION`	Hospital site is permanently shut down; cert never reconnects

Certificate expiry alerts¶

Because server-cert and CA rotation are deliberate manual actions, the Hub guards their expiry with an admin alert plus an admin-only email, reusing the standard alert framework (it is not a bespoke dialog):

The HA-leader's daily task checks the Hub server leaf and internal CA for upcoming expiry. It raises an AlertSource.HUB_CERT_EXPIRING admin alert plus an email to the administrators configured to receive email alerts, on the standard 90 / 30 / 10 / <5-day cadence (one email per threshold crossing).
Server leaf and CA drive the HUB_CERT_EXPIRING alert above because their rotation is manual. Client certs auto-cycle, so they are not alerted on the same cadence. But a client cert that is within 7 days of expiry and is not making renewal progress (the site is offline, or its cycle keeps failing) raises a separate AlertSource.HUB_CLIENT_CERT_RENEWAL admin alert. It also sends a once-a-day email listing the affected sites, so a stuck auto-renewal cannot fail silently. The Hub's own infra certs are excluded (by identity) from the generic zone cert-expiry alert so they are not double-reported.

The Sites Dashboard and the per-site Certificate tab also surface a days-until-expiry value that turns yellow inside 30 days and red inside 7.

Audit events¶

The cert lifecycle writes searchable audit events (per target_site_id where applicable; see ACL and Audit Log):

CLIENT_CERT_CYCLE_STARTED, CLIENT_CERT_CYCLE_SIGNED, CLIENT_CERT_CYCLE_COMPLETED, CLIENT_CERT_CYCLE_FAILED, CA_ROTATION_STAGED, CA_TRUST_ADDED, CA_ROTATION_ACTIVATED, CA_TRUST_REMOVED, CA_ROTATION_COMPLETED, CA_ROTATION_STEP_FAILED.