Sidebar

Is there a way to download a file via sftp without removing it from the sftp server?

0 votes
1.6K views
asked Aug 14, 2020 by scott-b-8720 (560 points)

I'm trying to download a file from an sftp site without deleting the file.  The file I'm trying to download will always have today's date in the filename.

I couldn't see a way to do this using the standard sftp node, so I'm using the following custom script.  My expectation is that when I execute the qie.readSFTPFile() function i'll be able to read the file without deleting it, but i'm getting an exception that says this:

       Cannot rename file from: path/file-2020-08-14.csv to: path/file-2020-08-14.csv.camelExclusiveReadLock

This seems to indicate that the sftp call is attempting to move the file (though the source location is the same as the dest location).  Is there a way to work around this without directly calling the camel sftp libraries myself?

var sftpHost = 'host.com';
var sftpPort = 22;
var userName = 'username';
var today = qie.formatDate('yyyy-MM-dd');
var currentFile = 'file-' + today + '.csv';
var fullPath = 'path/' + currentFile;

// note that i'm using camel params since my sftp uses a private key with a passphrase
var params = 'privateKeyFile=c:\\Keys\\myPrivateKey.ppk\nprivateKeyPassphrase=privateKeyPassphrase';

// get count of files in  directory
qie.debug('Checking ' + fullPath);
var fileCount = 0;
try
{
   // note that i'm using camel params since my sftp uses a private key with a passphrase
   fileCount = qie.countSFTPEndpoint(sftpHost, sftpPort, fullPath, true, true, params, userName, null);
   qie.debug('Count in ' + fullPath + ': ' + fileCount);
}
catch(err)
{
   qie.error("Unable to enumerate files in " + fullPath + ": " + err);
}

if (fileCount === 1) {
   try
   {
      var file = qie.readSFTPFile(sftpHost, sftpPort, fullPath, false, null, true, true, params, userName, null, 60000);
      if(file !== null)
      {
         var encodedBytes = file.getNode('/ftpFile/encodedBytes');
         var fileBytes = qie.base64DecodeToBytes(encodedBytes);

         // add file to inbound queue
         qie.addInboundMessage(fileBytes, currentFile);
      }
      else
      {
         qie.debug('Unable to process File: ' + fullPath);
      }
   }
   catch(err)
   {
      qie.error("Unable to download recon files from source: " + err);
   }
}
else
{
   qie.error('There were ' + fileCount + ' files at source for the path [' + fullPath + '] instead of just 1 file.');
}
commented Aug 15, 2020 by scott-b-8720 (560 points)
Thanks, Brandon, but that's what i'm already doing.  When my call indicates that I do not want to delete the file but also do not want to move it to an archive location, i get an error that appears like the call i'm making is trying to move the file or do something to it.
commented Aug 17, 2020 by scott-b-8720 (560 points)
...in other words, I need to read a file from a directory where I only have read permissions, so I cannot create any "done" files or have camel rename the file in order to get an exclusive lock or anything like that.
commented Aug 17, 2020 by scott-b-8720 (560 points)
Thanks, Brandon.  The camel parameter setting I was looking for is the one you mentioned to me:

readLock=Changed

1 Answer

+1 vote
 
Best answer

QIE has added ftp functions that can be used in a source script or mapping node.

Please see KB: https://www.qvera.com/kb/index.php/468/is-there-a-way-to-sftp-a-file-in-a-mapping-function?show=468#q468

The qie.readSFTPFile will return a xml with the file base64 encoded:

<ftpFile>
   <fileName>tempcps.txt</fileName>
   <fileLastModified>1513926000000</fileLastModified>
   <filePath>/tempcps.txt</filePath>
   <fileLength>711</fileLength>
   <encodedBytes>........</encodedBytes>
</ftpFile>

Here is a sample script you can use on the source node:

//List Files (if Needed)
var fileList = qie.listSFTPEndpoint('host', port, 'path', binary, passive, 'ftpParameters', 'userName', 'password');

//Get File and Metadata
var ftpResponse = qie.readSFTPFile('host', port, 'path', deleteFile, 'archivePath', binary, passive, 'ftpParameters', 'userName', 'password', timeout);

//Get CSV from response
var csv = qie.base64Decode(ftpResponse.getNode('/ftpFile/encodedBytes'));

//Add the inbound message
qie.addInboundMessage(csv, fileName);

answered Aug 14, 2020 by brandon-w-8204 (33,170 points)
selected Aug 17, 2020 by scott-b-8720
...