Dealing with new line or carriage returns in address field (or any field) in the inbound message

Question

the error is this:

Msg 109088303 [path=1] - exception (prc: 0, actPrc: -1, err: 1):

Unknown segment: 'Apt'

The new line or CR is in the address field (PID-11) where they put the apt number in.  Here is how it came in:

|100 Main St

Apt 3^New York^NY^11110|

The hex dump shows this:

7c313030204d61696e205374200d0a41707420...

The highlighted part is where I show the carriage return.  The site sending the message swears it only has a 20 20 and not a 20 0d 0a.

How do I remove a random carraige return or new line in the inbound message automatically so it does not error out every time?

 

Also, I had to copy the message and paste it into a ascii to hex converter to see the hex.  How can I do this in Qvera?  If I cannot, I'd like to request a new feature :)

Answer 1

First, It would be more reliable if the source system cleaned up the embedded new lines.  Here is one aproach that could be modified as needed.

In the above screen shot we have changed the source node (Node 1) to use Text.  In the first mapping node (Node 2) we clean up the message and convert it to an HL7 message.  By the time we get to node 3 we are dealing with a valid HL7 message.

NOTE: This approach may not be fool proof.  For example it might fail if the partial lines start with three uppercase character or three uppercase characters and pipe character (BAD or BAD|).  Test.  Test.  Test  

//split the message up into an array.  Note: newlines are removed

var sourceArray = source.getNode('/').split("\\r?\\n|\\r");

var testMSHSegment = 'MSH|^~\\&|TestToValidateLine';

var newLine = "\r\n";

var newMessage = '';

for (var i=0 ; i < sourceArray.length ; i++) {

   qie.debug("Line was: " + sourceArray[i]);

   try {

      //Try to validate the line

      qie.parseHL7String(testMSHSegment + newLine + sourceArray[i]);

      //If valid add it to the new message as a new line

      newMessage += newLine + sourceArray[i];

   } catch (err) {

      //If it threw an exception parsing then add it to the new message as a partial line

      newMessage += sourceArray[i];

   }

}

//Now that the message has been fixed, convert

//it from text to an HL7 message.

message = qie.createHL7Message();

message.setNode("/", newMessage);

All the nodes after mapping node 2 that used souce.getNode(...) would need to use message.getNode(...) and this would need to be thought out in case the message is transformed along the way such that what was originally pulled from source wasn't modified along the way.

 

If the source node returned an acknowledgement, change it to return it in a mapping:

 

Then insert another mapping after fixing the carriage returns to return the response:

 

Here is the code to return the response and shown in the above image:

var response = qie.parseHL7String(message.toString());

response.setNode('MSH-3', message.getNode('MSH-5'));

response.setNode('MSH-4', message.getNode('MSH-6'));

response.setNode('MSH-5', message.getNode('MSH-3'));

response.setNode('MSH-6', message.getNode('MSH-4'));

response.setNode('MSH-7', qie.formatDate('yyyyMMddhhmmss'));

response.setNode('MSH-8', message.getNode('MSH-8'));

response.setNode('MSH-9', 'ACK');

response.setNode('MSH-10', qie.formatDate('yyyyMMddhhmmssSSS'));

response.setNode('MSH-11', message.getNode('MSH-11'));

response.setNode('MSH-12', message.getNode('MSH-12'));

response.setNode('MSA-1', 'AA');

response.setNode('MSA-2', message.getNode('MSH-10'));

 

qie.postMessageResponse(response);

 

To address the feature related questions:

The current version of QIE has a Show Whitespaces option under the View menu that display a character for new lines (not in HEX).

 

 

The next version of QIE will have a new message type called Binary.  This will allow a sample message to be created and viewed as HEX by switching the sample message from Hl7 to Binary.  

I think what you really want is a direct way to view Text/HL7 as HEX.  We will log that as an enhancement but until then the Binary message model can be leveraged withing QIE to view the new line charactes as HEX

 

See also:

 

Can I intercept or preprocess messages? How to handle systems that send too much or invalid data?

Comments

Ok this did not work as the messages causes an error when creating the ack from script.  There may be another way around that though.

and unfortunately the source of the message refuses to acknowledge they are adding a new line.

I'll see if the binary works better than copying and pasting the string to a hex converter.

Thanks.

---

Another way of doing this would be to create a new channel that fixes the embedded carriage returns then forward that to the existing channel.  Using a two channel approach will leave the original channel pretty much intact.  ie. no need to switch from using source to message in the rest of the mapping nodes etc.

---

changing from source to message was no problem at all.  The problem was the ack.  When trying to build an ack from script it couldn't find the source nodes the script needed.  I may try to rewrite the script to change the message to HL7 then store the value to a var then create a new HL7 message and fill in the ack values from the var.  With the couple that come in per week, I can wait and will update this once tested.

---

I am curious what your ack script looks like.  Also, is this an in house channel or a purchased package?  I don't know if this is helpful but if the ack script uses node tags ie {PID-3}.  {PID-3} and {s:PID-3} both refer to the source.  {m:PID-3} refers to the message.  See the section in the reference manual under "Node Tags" for more options.  The reference manual can be accessed from the help menu.  Again a two channel approach would avoid having to make that switch.  I should have thought to mention that side effect in the original answer to your question.

---

This is what our ack script always looks like:

message = qie.createHL7Message();
message.setNode('MSH-3', source.getNode('MSH-5'));
message.setNode('MSH-4', source.getNode('MSH-6'));
message.setNode('MSH-5', source.getNode('MSH-3'));
message.setNode('MSH-6', source.getNode('MSH-4'));
message.setNode('MSH-7', qie.formatDate('yyyyMMddhhmmss'));
message.setNode('MSH-8', source.getNode('MSH-8'));
message.setNode('MSH-9', 'ACK');
message.setNode('MSH-10', qie.formatDate('yyyyMMddhhmmssSSS'));
message.setNode('MSH-11', source.getNode('MSH-11'));
message.setNode('MSH-12', source.getNode('MSH-12'));
message.setNode('MSA-1', 'AA');
message.setNode('MSA-2', source.getNode('MSH-10'));

---

Okay, yeah I was thinking about the ack on a destination and not the ack on the source.  That would be a problem either way.  What you can do is change the source node.  In the "Response:" drop down select "From Mapping or Destination node".  Then in the mapping insert another row with this modified code to post the response:

var response = qie.parseHL7String(message.toString());
response.setNode('MSH-3', message.getNode('MSH-5'));
response.setNode('MSH-4', message.getNode('MSH-6'));
response.setNode('MSH-5', message.getNode('MSH-3'));
response.setNode('MSH-6', message.getNode('MSH-4'));
response.setNode('MSH-7', qie.formatDate('yyyyMMddhhmmss'));
response.setNode('MSH-8', message.getNode('MSH-8'));
response.setNode('MSH-9', 'ACK');
response.setNode('MSH-10', qie.formatDate('yyyyMMddhhmmssSSS'));
response.setNode('MSH-11', message.getNode('MSH-11'));
response.setNode('MSH-12', message.getNode('MSH-12'));
response.setNode('MSA-1', 'AA');
response.setNode('MSA-2', message.getNode('MSH-10'));

qie.postMessageResponse(response);

I will fix the answer to account for the ack.

---

I added a couple screen shots to the answer to address the ack issue on the source node.

Answer 2

Using the new feature available in version 44 discused in this question:

Can I intercept or preprocess messages? How to handle systems that send too much or invalid data?

We can now use this preprocess script to fix the message and send a response when the message is discarded:

Again as text:

//Convert bytesIn to a string

var messageToValidate = new java.lang.String(bytesIn, "UTF-8");

//Split the string up into a string array for each line.  Note: newline characters are removed from the array

var sourceArray = messageToValidate.split('\\r?\\n|\\r');

// Create a valid HL7 message header used below to validate each line in the array

var testMSHSegment = 'MSH|^~\\&|ValidationHeader';

//Loop over the array and validate each line

var newLine = '\r\n';

var fixedMessageAsText = '';

for (var i=0 ; i < sourceArray.length ; i++) {

   try {

      //Try to validate each line by parsing it with a valid MSH segment

      qie.parseHL7String(testMSHSegment + newLine + sourceArray[i]);

      //If valid, add it to the new message as a new line or complete line

      fixedMessageAsText += newLine + sourceArray[i];

   } catch (err) {

      //If it throws an exception parsing then add it to the new message as a partial line

      fixedMessageAsText += sourceArray[i];

   }

}

 

//Convert fixedMessageAsText to a message model

var fixedMessage = qie.parseHL7String(fixedMessageAsText);

//Filter out or discard any message types that are NOT a VXU

var newBytes = fixedMessage.getBytes();

if (!StringUtils.equalsIgnoreCase(fixedMessage.getNode('MSH-9.1'), 'VXU')) {

   newBytes = null;

}

//If message was discarded send a response from this script.  If the newBytes are not null then the reponse will be handled by the channel.

if (newBytes === null) {

   var ackResponse = qie.parseHL7String('MSH|^~\\&|LinkLogic-Test|MHS|');

   

   ackResponse.setNode('MSH-3', fixedMessage.getNode('MSH-5'));

   ackResponse.setNode('MSH-4', fixedMessage.getNode('MSH-6'));

   ackResponse.setNode('MSH-5', fixedMessage.getNode('MSH-3'));

   ackResponse.setNode('MSH-6', fixedMessage.getNode('MSH-4'));

   ackResponse.setNode('MSH-7', '20160912034734513');

   ackResponse.setNode('MSH-8', fixedMessage.getNode('MSH-8'));

   ackResponse.setNode('MSH-9', 'ACK');

   ackResponse.setNode('MSH-10', '20160912034734513');

   ackResponse.setNode('MSH-11', fixedMessage.getNode('MSH-11'));

   ackResponse.setNode('MSH-12', fixedMessage.getNode('MSH-12'));

   ackResponse.setNode('MSA-1', 'AA');

   ackResponse.setNode('MSA-2', fixedMessage.getNode('MSH-10'));

   

   responseBytes = ackResponse.getBytes();

}

 

//Set bytesOut to the modified bytes

bytesOut = newBytes;