Skip to content

Automating Error Queue Remediation with Scheduled Scripts

QIE's Scheduled Scripts can work the channel's error queue automatically, deciding per message whether to retry it with the original content, retry it with a fix applied, or discard it. Errors usually arrive in batches with a single cause: a downstream system that was offline, a credential that has since been refreshed, or a message format the channel handles after a code change. Working through those one row at a time is wasteful.

The errorManager binding that drives this is only available inside Scheduled Scripts. It is not exposed in mapping, condition, or destination scripts.

The errorManager binding

Method Purpose
searchErrors(query) Returns a list of erred-message IDs matching query (blank for "all").
searchErrors(query, maxResults) Same, capped at maxResults rows (0 = unlimited).
getErrorDetail(id) The error message / stack trace for an erred message.
getMessage(id) The processed message at the point it errored (mutable. Edit and pass to resolveError).
getSource(id) The original unmodified source message (mutable. Edit and pass to resolveError).
hasParent(id) true if the erred message is itself the result of an earlier resolve or resubmit.
resolveError(id) Replay the erred message starting from the original source.
resolveError(id, newContent) Replay with new content (a MessageModel or byte[]).
discardError(id) Permanently remove the erred message.

Example: retry messages whose error matches a known fix

Schedule this script to run every hour on a channel that previously errored against an unreachable downstream system. Once the downstream is restored, the next run drains the backlog.

var errorIds = errorManager.searchErrors("Connection refused");
qie.info("Retrying " + errorIds.size() + " messages that errored with 'Connection refused'.");
for (var i = 0; i < errorIds.size(); i++) {
    errorManager.resolveError(errorIds.get(i));
}

Example: discard old errors that match a known dead-letter pattern

Run nightly to keep the error queue from growing unbounded with unrecoverable cases.

var deadLetterIds = errorManager.searchErrors("Unparseable HL7", 500);
for (var i = 0; i < deadLetterIds.size(); i++) {
    errorManager.discardError(deadLetterIds.get(i));
}
qie.info("Discarded " + deadLetterIds.size() + " unparseable messages.");

Example: fix and re-process

Some errors only recover after editing the message. Pull the processed message, mutate it, and replay with the corrected content.

var ids = errorManager.searchErrors("MSH-3 missing");
for (var i = 0; i < ids.size(); i++) {
    var msg = errorManager.getMessage(ids.get(i));
    msg.setNode("MSH-3", "HOSP_A");                   // inject the missing sender
    errorManager.resolveError(ids.get(i), msg);
}

Example: correct the source and replay from the top

getMessage and getSource return different things, and both are mutable. getMessage returns the message as it stood at the node that errored, so a fix lands after whatever mapping already ran. getSource returns the original inbound message, so the fix applies before any node runs and the whole channel replays against the corrected content.

Correct a value in the source message and let the channel reprocess it from the beginning:

var searchText = "PID-33 invalid";

if (qie.getErrorCount() > 0) {
    var errorIds = errorManager.searchErrors(searchText);
    for (var i = 0; i < errorIds.size(); i++) {
        var messageQueueId = errorIds.get(i);
        if (!errorManager.hasParent(messageQueueId)) {
            var errorSource = errorManager.getSource(messageQueueId);
            if (errorSource !== null) {
                errorSource.setNode("PID-33.1", "Good Value");
                errorManager.resolveError(messageQueueId, errorSource);
            }
        }
    }
}

getSource returns null when the message is no longer in the error queue, so test for it before calling setNode. The hasParent guard skips messages that already failed a previous retry.

Caveats

  • Logs everything. Both resolveError and discardError are visible in the channel log and the message's revision history; an audit trail of automated remediation is preserved.
  • searchErrors matches on error text. It does not parse HL7 fields or evaluate templates. Keep the query specific enough that you do not sweep up unrelated errors.
  • Be careful with resolveError without a query. Resolving every error blindly re-queues messages that errored for permanent reasons (bad schema, missing required field) and re-error them on the next pass, growing log noise.
  • hasParent identifies a message that has already been retried. A resolve or a resubmit clones the original source, and the replayed message keeps a reference to the source it came from. When that replay errors again, hasParent returns true. Test it before resolving so a permanently broken message is not retried on every scheduled run.