External Connections like Web Services fail after some time (hours or days) but work again after a restarting QIE.

Question

I have some web service calls configured in my channel.  After restarting QIE they all work correctly.  After some time.  Sometimes hours, days, or weeks they stop working.  After restarting QIE they work again.  How can I figure out what is going on?  

My logs contain this exception:

java.net.ConnectException: Connection refused

Answer 1

The following link provides some ideas about debugging ConnectExceptions but that fact that QIE works again after a restart may be related to DNS caching.  This may occur if the web services that you are trying to connect to are hosted by Amazon.

http://javarevisited.blogspot.com/2013/02/java-net-ConnectException-Connection-refused.html

This article describes the behavior seen in QIE with a solution to use the networkaddress.cache.ttl java.security setting. 

http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-jvm-ttl.html

Note this is a JRE or system wide java.security setting found in the file %JRE%\lib\security\java.security and can not be set using a -D.  However, the sun.net.inetaddr.ttl is a -D java option that can be used instead.  This allows for setting the DNS caching per JVM instance.  Note this is a depricated java option and will be removed in a future version of java. 

To debug this issue you might try using the DNSCache.jar located on Qveras download directory (private access).

This would tell you the IP addresses that a domain is resolved to within a JVM.  Which can be compared to a new jvm instance to see if the domain now resolves to a new IP.

 

To run it in QIE:

• Add the DNSCache.jar to QIE
• Setup a channel to call it (This could be a run once or test channel and the code could be placed in the source node or a mapping). Here is the code:

importPackage(Packages.com.qvera);
importPackage(Packages.com.qvera.diagnotics.network);

//qie.info(com.qvera.Try.hello());  
qie.info(com.qvera.diagnotics.network.DNSCache.getDNSCacheInfo(["www.msn.com"]));

To run it from the command line:

• cd to path where where jar is located.
• Run this command: java -cp DNSCache.jar com.qvera.diagnotics.network.DNSCache www.msn.com

Replace "www.msn.com" with the domain that is failing.

Additional resources:

http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html#nct

http://stackoverflow.com/questions/1835421/java-dns-cache-viewer

Other Question that may be useful in diagonsis connection issues:

https://www.qvera.com/kb/index.php/59/configure-produce-additional-logging-for-web-service-calls?show=60#a60

https://www.qvera.com/kb/index.php/20/configure-qie-produce-additional-logging-for-connections?show=20#q20

https://www.qvera.com/kb/index.php/19/error-building-failed-unable-certification-requested-target?show=19#q19