Sidebar

Alternate to using "Large File Handling Options" for FTP?

0 votes
418 views
asked Jul 14, 2022 by rich-c-2789 (16,180 points)

Is there a alternative way of downloading large files from an FTP server other than using the "Large File Handling Options"?  I use the "Custom Apache Camel parameters" which are not compatible with this new feature.

See How to create a pass through channel to handle moving large files?

1 Answer

0 votes

Yes. As of version .51 the FTP source can be configured to use the following "Custom Apache Camel parameters" to stream large files to a directory without loading the files into QIEs memory. 

streamDownload=true
stepwise=false
localWorkDirectory=/kb/examples/in/localFolder
binary=true
resumeDownload=true

"NOTE: When "localWorkDirectory" option is used, the message is not loaded into memory, but passed to the channel as a filename which points to the file on disk. The channel will need to dispose of the file once it is finished using it.  QIE will create a subfolder if the "localWorkDirectory" called "download" where you can find the downloaded files.  The "localWorkDirectory" is used by camel as a temporary location to stream the file.  When complete, QIE moves it to the "download" folder"  

"NOTE: The "resumeDownload" must be supported by the FTP server.  Most FTP servers support this.  If it fails, remove it and try again.  See resumeDownload in the camel documentation.  SFTP does not support resumeDownload.

 

To configure an example, to move large files from an FTP server to local folder, follow these steps:

On source node:

  • Step 1: Configure FTP Settings to connect to the FTP server and path containing the large files to be moved.
  • Step 2: Click "Custom Apache Camel paramters"
  • Step 3: Enter the above mentioned parameters and set the path to the desired folder for the "localWorkDirectory" parameter into the field below the check box. This is a local folder, not an FTP path. Make sure this folder has plenty of disk space to handle the volume for the large files to be processed. And permission to allow QIE to create a "download" subfolder.

On destination node:

  • Step 4: Change the "Destination" to "Discard".

NOTE: It is not recommended to use anything other than a "Discard" destination node since the message content is never loaded into memory. Only the filename is set and available in mapping and destination nodes.

 

In Mapping node (adjust these steps based on your needs):

  • Step 5: Insert new mapping function and set "Function" to "Custom Script".
  • Step 6: Define a file handle variable that points to the file in the "Stream Path" using these "Code Wizard" functions: qie.newFile(), qie.getStreamPath() and qie.getFileName();
    • Example: var streamedFile = qie.newFile('/kb/examples/in/localFolder/download/' + qie.getFileName());
  • Step 7: Define a file handle variable for the destination folder.
    • Example: var destinationFile = qie.newFile('/destinationFolder/' + qie.getFileName());
  • Step 8: Move the file using the two handles and qie.moveFile() Code Wizard function.
    • Example: qie.moveFile(streamedFile, destinationFile);

The final code needed in the mapping function looks like this:

//Define a file handle to the source file stored in the download folder
var streamedFile = qie.newFile('/kb/examples/in/localFolder/download/' + qie.getFileName());

//Define a file handle for the destination file location
var destinationFile = qie.newFile('/destinationFolder/' + qie.getFileName());

//Move the file from the Stream Path to the destination using the two file handles
qie.moveFile(streamedFile, destinationFile);

 

NOTE: For file and netFile avoid using qie.readFile(), qie.writeFile() in mapping nodes. These load the file into memory and may cause the memory issues this question is trying to avoid.

 

To move a file on an FTP server to another FTP server, change the code to do something like this:

//Define a file handle to the source file stored in the Stream Path
var streamedFile = qie.newFile('/kb/examples/in/localFolder/download/' + qie.getFileName());

//Use a String instead of a file handle for the destination FTP folder and file name
var destinationPath = '/destinationFTPFolder/' + qie.getFileName();

//Use one of the qie.writeFTPFile...() methods or qie.writeSFTPFile() method and specify the host, port, etc. as needed.
//This example only changed the "path" and "data" arguments to use the streamedFile
//and destinationPath variables defined above.
qie.writeFTPFileNoEncryption('host', port, destinationPath, binary, passive, 'ftpParameters', 'userName', 'password', streamedFile);

 

NOTE: For FTP avoid using qie.readFTPFileNoEncryption(), qie.readFTPFileExplicitTLS(), qie.readFTPFileImplicitTLS(), and qie.readSFTPFile(). These load the file into memory and may cause the memory issues this question is trying to avoid.

 

See also:

FTP Example using "Large File Handling options": How to create a pass through channel to handle moving large files?

Other examples: How to use Large File Handling Options?

answered Jul 14, 2022 by rich-c-2789 (16,180 points)
edited Apr 14, 2023 by rich-c-2789
...