How can I gather extended information on a failed sftp file read and prevent that failure?

Question

I'm executing the following code, but each time i call readSFTPFile my returned object comes back as null.  I'm sure that my connection info is correct (it is used to list all the files successfully).  I can use the same account info to manually pull down files and remove them from the sftp site, so I'm fairly certain it isn't an issue with my permissions.  

I'm trying to ultimately find out why the file object is coming back as null, but i'd also like to know if there is a certain way i can gather extended information about why the sftp call failed.

var fileNames = qie.listSFTPEndpoint(sftpHost, sftpPort, '', true, true, null, userName, password);
      
      // loop through each file and pull it down from the ftp server
      for (var currentFile = 0; currentFile < fileNames.length; currentFile++) 
      {
         var filePath = fileNames[currentFile];
         qie.info('Retrieving File: ' + filePath);
         
         // this line will retrieve the file from the ftp server and then DELETE it
         var file = qie.readSFTPFile(sftpHost, sftpPort, filePath, true, null, true, false, null, userName, password, 60000);
         if(file !== null)
         {
            qie.info('File downloaded: ' + filePath);
         }
         else
         {
            qie.info('File NOT downloaded: ' + filePath);
         }
      }

Comments

What are the file names that are returned from the listSFTPEndpoint call?  Do they have file extensions?

Answer 1

QIE uses apache camel to perform all of the FTP/SFTP operations.  To adjust the logging level for all of the QIE libraries including the apache camel libraries, you will need to update the system configuration 'Log Level' option to 'Info'.  This option can be found in the 'Other System Settings' area of the 'Advanced' system configuration.

 

Once this has been updated, you will be able to see log entries in your QIE installation logs directory (in windows this defaults to 'C:\ProgramData\QIE\logs'.  The file will be 'qie.log' and will contain some lines that look like:

2017-07-06 09:25:56,406 [Test Mapping Function Runner] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> Local version string: SSH-2.0-JSCH-0.1.54 
2017-07-06 09:25:56,406 [Test Mapping Function Runner] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256 
2017-07-06 09:25:56,407 [Test Mapping Function Runner] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> aes256-ctr is not available. 
2017-07-06 09:25:56,407 [Test Mapping Function Runner] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> aes192-ctr is not available. 
2017-07-06 09:25:56,407 [Test Mapping Function Runner] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> aes256-cbc is not available. 
2017-07-06 09:25:56,407 [Test Mapping Function Runner] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> aes192-cbc is not available. 
2017-07-06 09:25:56,407 [Test Mapping Function Runner] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> CheckKexes: diffie-hellman-group14-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521 
2017-07-06 09:25:56,430 [Test Mapping Function Runner] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> diffie-hellman-group14-sha1 is not available. 
2017-07-06 09:25:56,430 [Test Mapping Function Runner] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> CheckSignatures: ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 
2017-07-06 09:25:56,430 [Test Mapping Function Runner] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> SSH_MSG_KEXINIT sent 
2017-07-06 09:25:56,430 [Test Mapping Function Runner] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> SSH_MSG_KEXINIT received 
2017-07-06 09:25:56,430 [Test Mapping Function Runner] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,

Specifically you will look for the lines that contain 'org.apache.camel' for the log entries that are ouput from the SFTP operation.

Comments

It looks like the issue is that the remote files have commas in the filenames.  Is there a way I can tell camel to escape those or anything like that?

---

Do you have the actual Camel FTP error? There is a preMove parameter (http://camel.apache.org/file2.html) that might be applicable, but I'm not sure if it is a fit for what you are needing.

---

I do not have the actual camel ftp error, as I couldn't find an actual error in the stack trace.  However, if i rename the file on the source server to not have the comma it gets picked up successfully.

---

If the source system can configure the file name pattern, I would recommend fixing it there.  If not, we will have to find what error Camel is producing (or what URL that Camel ends up building to execute the FTP query.)  If you contact support, and we can do a screen share to help diagnose the issue.