Sidebar

How to be notified and prevent automatic error resolution from getting out of hand?

0 votes
391 views
asked Jun 10, 2016 by rich-c-2789 (16,240 points)
I have a scheduled script that automatically resolves errors that are due to the destination system going down temporarily.  How ever over the weekend the destination system was down all weekend.  When I returned the error had a lot of errors due to the script resolving the messages over and over again.  Is there a way to tell more errors are happing than what I expect to be normal so that the script will not keep processing errors that will likly fail again.

1 Answer

0 votes

Use "hasParent(messageQueueId)" to prevent resolving an error that has already been resolved and errored again.  This scheduled script also leverages the fact that searchErrors returns the more recent errors first.  By adding a limit to the results we can establish an expected error threshold and then send an email to notify when this threshold has been exceeded.  

//See if there are any errors using the existing code wizard function before we do anything.
//This is a quick test that avoids the searchErrors call if it is not needed.
if (qie.getErrorCount() > 0) {
   //DateUtils will be used for date manipulation
   var DateUtils = org.apache.commons.lang.time.DateUtils;
   
   //Create the to date by subtracting 1 day from the current date.
   //We add the 'to=' predicate set to a date 1 day ago so that we don't
   //get stuck processing the same message repeatedly if it errors again.
   var toDate = DateUtils.addHours(qie.deduceDate(qie.getSystemDate()), -24);
   var toDateText = "to=" + qie.formatDate('yyyyMMddHHmmssSSS', toDate) + ", ";
   qie.debug('toDateText: ' + toDateText);
   
   var searchText = toDateText + 'error=disgracefulWebServiceFailure';
   qie.debug('searchText: ' + searchText);
   
   //NEW: Search for errors that match the searchText and limit the results
   var maxResults = 20;
   var results = errorManager.searchErrors(searchText, maxResults);
   qie.debug('searchErrors returned ' + results.size() + ' messageQueueIds.');
   qie.debug('The messageQueueIds returned: ' + results);
   
   var errorsResolved = 0;
   var errorsPreviouslyResolved = 0;
   for (var i=0 ; i < results.size() ; i++) {
      var messageQueueId = results.get(i);
 
      //if there is no parent then this message has not been previously resubmitted or resolved
      if (!errorManager.hasParent(messageQueueId)) {
         errorManager.resolveError(messageQueueId);
         errorsResolved++;
      } else {
         qie.debug('This message queue was previously resolved.  Skipping');
         errorsPreviouslyResolved++;
      }
   }
   qie.debug('Finished resolving errors');
   //Send update of warning.  Delete the else block if you do not want a summary email.
   if (errorsPreviouslyResolved === maxResults || (errorsPreviouslyResolved > 0 && errorsResolved === 0)) {
      qie.queueEmail('rabbit@acme.ltunes.com',
         'WARNING: too many previous errors expected',
         'Please review the error queue to ensure nothing is wrong.  It appears that the messages continue to error after being resolved.' +
         'Found ' + results.size() +' errors.\nResolved ' +
            errorsResolved + ' errors.\n' +
            errorsPreviouslyResolved + ' were previously resolved.\n\n');
   } else {
      qie.queueEmail('rabbit@acme.ltunes.com',
         'Auto resolution summary',
         'Found ' + results.size() +' errors.  Resolved ' +
            errorsResolved + ' errors.  ' +
            errorsPreviouslyResolved + ' were previously resolved.');
   }
} else {
   qie.debug('No errors found.  Nothing to resolve.');
}
 
Also see these questions:
 
 
 
 
 
answered Jun 10, 2016 by rich-c-2789 (16,240 points)
...