High Availability Configuration and Options¶

QIE supports High Availability (HA) to ensure continuous operation by deploying multiple QIE instances (node instances) against a shared database. This configuration minimizes downtime during server updates or failures. Throughout this article a node is one of those running QIE engine instances. The term does not refer to the mapping, condition, or destination nodes that make up a channel.

In an HA setup, if a node becomes unavailable, another node in the cluster automatically takes over processing, maintaining uninterrupted interface operation. Typically, an external load balancer or traffic manager distributes incoming traffic across the QIE cluster.

Note

To enable High Availability (HA), in-memory database tables are required for inter-node communication. For Microsoft SQL Server, ensure the 'Setup In-Memory Tables' step has been completed before proceeding with HA configuration.

Running Multiple Instances Without HA Is Not Supported¶

Running two or more QIE instances against the same database without HA enabled is an unsupported configuration. Because each non-HA instance assumes it is the only one, all of them receive and process the same work, causing duplicate message processing and duplicate API calls that can corrupt data.

Channels are stopped automatically

When a standalone (non-HA) QIE instance detects another instance running against its database, it immediately stops all running channels and enters lock-down: no channel can be started until the conflict is resolved. This prevents duplicate processing. You can still log in to the admin console, where a critical alert explains the situation.

To resolve the lock-down, either enable HA on every instance (see Enabling HA Engine) or run only a single instance against the database. Once no other instance is detected (typically after correcting the configuration and restarting) the lock-down is lifted automatically and the channels that were running are restarted.

Note

This detection applies to standalone (non-HA) instances only and is supported on MySQL/MariaDB and Microsoft SQL Server databases. It detects instances running on different hosts; it does not detect a second instance running on the same host (same IP address).

Enabling HA Engine (Required)¶

To enable HA clustering, configure the following Java option in the "Arguments" field on the "Startup" tab of the QIE Service Manager:

-Dqie.haEngine=EnterpriseHAServiceImpl

Configure HA Engine Priority (Optional)¶

QIE supports a primary/failover configuration within an HA cluster. Assigning a priority to a node determines its role in processing tasks.

Note

This option should NOT be used in a Kubernetes or Docker environment.

Primary Node: Set the qie.instancePriority Java option to 1 on the designated primary node.

-Dqie.instancePriority=1

This directs all QIE-initiated tasks (e.g. file or FTP receivers) to the primary node.

Failover Nodes: Other nodes defer processing to the primary node. If the primary node fails, a failover node assumes processing.

Automatic Failback: When the primary node recovers, it automatically resumes processing.

Load Balancer Control: Note that inbound connections managed by a load balancer (e.g. Socket or HTTP receivers) are not affected by this priority setting.

By default, all nodes have a priority of 99. You can assign different priorities to create a primary/secondary/etc. hierarchy.

Configure Probe Port (Recommended)¶

Network Layout Behind a Load Balancer¶

An external load balancer sitting in front of QIE typically distributes only the traffic destined for interface listener ports: HL7 MLLP sockets, HTTP Listener endpoints, DICOM listeners, and the qie.probePort for health checks. The QIE management console itself (default -Dqie.port=8380) should not be exposed through the public LB. It is the administrative interface and does not need to be internet-reachable to run channels.

Recommended layout:

LB → interface ports. Add each interface listener port to the LB pool: HL7 MLLP, HTTP Listener, DICOM Listener, and any qie.probePort used for LB health checks (see Configuring the Probe Port for Load Balancer Health Checks).
Round-robin as the default. Use round-robin distribution across QIE nodes for non-order-sensitive traffic. For interfaces that require end-to-end ordering (typical HL7 v2 streams), configure sticky routes so all messages from a specific sender stay on one node. See First-In-First-Out (FIFO) Considerations below.
Management console behind a jump box. Do not add the QIE console port (-Dqie.port) to the public LB. Restrict console access to an internal network or reach it through a bastion / jump host that requires its own authentication. Administrators sign in to the jump host first, then to the QIE console. The console itself is never directly reachable from the public network.
Probe port isolation. The qie.probePort is intentionally unauthenticated so the LB can check it without credentials. Confine it to the LB's health-check IP range at the network layer; do not expose it to the public network.

Configuring the Probe Port for Load Balancer Health Checks¶

For HA clusters using a load balancer, configure the qie.probePort Java option to enable health checks. This allows the load balancer to determine the health of each QIE node.

-Dqie.probePort=8080

Basic Health Check: A simple HTTP GET request to the configured probe port returns an HTTP status code 200 if the node is ready to receive traffic.

Detailed Statistics (Optional): Appending query parameters provide detailed node statistics in JSON format:

individualNode: Returns statistics for the specific node.
allNodes: Returns aggregated statistics for the entire cluster.

The following JSON payload is returned:

{
   "instanceId": "...",
   "processorThreadCount": ...,
   "processorQueueSize": ...,
   "processorThroughput": ...,
   "completionThreadCount": ...,
   "completionQueueSize": ...,
   "completionThroughput": ...,
   "loggerThreadCount": ...,
   "loggerQueueSize": ...,
   "loggerThroughput": ...
}

Description of JSON fields:

instanceId: The instance ID of the node.
processorThreadCount: Number of processor threads.
processorQueueSize: Number of messages in the processor queue.
processorThroughput: Messages processed per second (10-second average).
completionThreadCount: Number of completion threads.
completionQueueSize: Number of messages in the completion queue.
completionThroughput: Messages completed per second (10-second average).
loggerThreadCount: Number of logger threads.
loggerQueueSize: Number of log messages in the logger queue.
loggerThroughput: Log messages written per second (10-second average).

First-In-First-Out (FIFO) Considerations¶

QIE ensures FIFO message processing within a single node. However, in a multi-node HA cluster, FIFO is guaranteed only at the node level.

Load Balancer Routing: To maintain end-to-end FIFO for message streams like HL7v2, route all messages from a specific source to a single QIE node.

Sticky Routes/Persistent Routes: Use sticky routes or persistent routes in the load balancer to direct traffic from the same source IP to the same node.
Sticky Route Reset: Bear in mind that sticky routes are reset when the cluster's health changes (e.g. nodes added or removed).

Workaround for Non-Sticky Load Balancers (Two-Node Configuration):¶

For load balancers without sticky route capabilities, a two-node (primary/backup) configuration can be used.

Primary Node: Enable the probe port on the primary node.
Backup Node Configuration: Set the 'qie.primaryInstanceIp' Java option to the primary node's IP address.
Configure the probe interval and missed probes using 'qie.primaryInstanceProbeInterval' (default: 5 seconds) and 'qie.primaryInstanceMissedProbes' (default: 3).
Backup Node Behavior: When the primary node is healthy, the backup node reports itself as unhealthy to the load balancer.
- If the primary node fails, the backup node becomes healthy and receives traffic.
- When the primary node recovers, the backup node returns to an unhealthy state.
Limitations: This workaround is only applicable to VM installations and is not compatible with Docker or Kubernetes deployments.

-Dqie.primaryInstanceIp=192.168.1.100
-Dqie.primaryInstanceProbeInterval=5
-Dqie.primaryInstanceMissedProbes=3

Upgrading High Availability Cluster¶

When performing an upgrade of a QIE High Availability (HA) cluster, it is crucial to understand that the process requires system downtime. All nodes in the cluster are upgraded simultaneously, and there is no rolling upgrade capability.

The downtime occurs because the QIE engine requires a database schema update with the new version. The sequence of events is as follows:

First Node Startup and Schema Update: When the first upgraded QIE instance is started, it checks and updates the shared database schema.
Engine Blockade: During this essential database schema update, all other QIE instances are prevented from starting. They wait for the first node to complete the update process.

Note

When a node is blocked from starting up, the logs show a hibernate error: Duplicate entry 'mem_misc_job_mutex' for key 'mem_misc_job_mutex' or something similar while it is being blocked. Once the node working on the schema update completes, these exceptions stop and the node continues its startup process as normal.

Sequential Startup: Once the first node has successfully updated the database schema, the remaining nodes in the cluster complete their start up process one at a time.

Because the entire cluster is unavailable until the first node completes the database schema migration and starts up completely, Qvera strongly recommends that all updates to the HA cluster are performed only during scheduled system maintenance times. This ensures that the necessary system downtime is managed and communicated appropriately.