How to create a pass through channel to handle moving large files?

Question

I have some large files that I need to move to another location. The files are created daily by another system and saved on an FTP server. I need to download/move them to another location. I thought I could use QIE so everything is automated within QIE. However, after trying to create a channel that would download them from the FTP and move them to the new destination, I get a memory error. Note: I do not need to do any transformations or make any changes of any kind to these files. I just need to move them. Is there a way to create a pass through channel to move large files without loading the files into memory?

Answer 1

Yes. As of version .51 the FTP, File, and Network Share sources can be configured using the "Large File Handling Options". These new configuration options can be found on the 2nd tab of the source node Labeled "FTP Configuration", "Directory Configuration", or "Network Share Configuration" respectfully. As shown here:

As noted in the above screenshot:

"NOTE: When this option is checked, the message is not stored in memory, but passed to the channel as a filename which points to the file on disk. The channel will need to dispose of the file once it is finished using it."

 

To configure an example, to move large files from an FTP server to local folder, follow these steps:

On source node:

• Step 1: Configure FTP Settings to connect to the FTP server and path containing the large files to be moved.
• Step 2: Click "Stream messages to disk"
• Step 3: Enter a folder in "Stream Path". This is a local folder, not an FTP path. Make sure this folder has plenty of disk space to handle the volume for the large files to be processed. And permission to allow qie to create a tmp_lwd subfolder.
• Step 4: Accept the default for "Enable resume download option" for now, but you may need to come back and uncheck this if the FTP server does not support it. (Note: this option only applies to FTP)

On destination node:

• Step 5: Change the "Destination" to "Discard".

NOTE: It is not recommended to use anything other than a "Discard" destination node since the message content is never loaded into memory. Only the filename is set and available in mapping and destination nodes.

 

In Mapping node (adjust these steps based on your needs):

• Step 5: Insert new mapping function and set "Function" to "Custom Script".
• Step 6: Define a file handle variable that points to the file in the "Stream Path" using these "Code Wizard" functions: qie.newFile(), qie.getStreamPath() and qie.getFileName();
  • Example: var streamedFile = qie.newFile(qie.getStreamPath() + '/' + qie.getFileName());
• Step 7: Define a file handle variable for the destination folder.
  • Example: var destinationFile = qie.newFile('/destinationFolder/' + qie.getFileName());
• Step 8: Move the file using the two handles and qie.moveFile() Code Wizard function.
  • Example: qie.moveFile(streamedFile, destinationFile);

The final code needed in the mapping function looks like this:

//Define a file handle to the source file stored in the Stream Path
var streamedFile = qie.newFile(qie.getStreamPath() + '/' + qie.getFileName());

//Define a file handle for the destination file location
var destinationFile = qie.newFile('/destinationFolder/' + qie.getFileName());

//Move the file from the Stream Path to the destination using the two file handles
qie.moveFile(streamedFile, destinationFile);

 

NOTE: For file and netFile avoid using qie.readFile(), qie.writeFile() in mapping nodes. These load the file into memory and may cause the memory issues this question is trying to avoid.

 

To move a file on an FTP server to another FTP server, change the code to do something like this:

//Define a file handle to the source file stored in the Stream Path
var streamedFile = qie.newFile(qie.getStreamPath() + '/' + qie.getFileName());

//Use a String instead of a file handle for the destination FTP folder and file name
var destinationPath = '/destinationFTPFolder/' + qie.getFileName();

//Use one of the qie.writeFTPFile...() methods or qie.writeSFTPFile() method and specify the host, port, etc. as needed.
//This example only changed the "path" and "data" arguments to use the streamedFile
//and destinationPath variables defined above.
qie.writeFTPFileNoEncryption('host', port, destinationPath, binary, passive, 'ftpParameters', 'userName', 'password', streamedFile);

 

NOTE: For FTP avoid using qie.readFTPFileNoEncryption(), qie.readFTPFileExplicitTLS(), qie.readFTPFileImplicitTLS(), and qie.readSFTPFile(). These load the file into memory and may cause the memory issues this question is trying to avoid.

NOTE: For FTP the Stream Path needs permision to create a tmp_lwd folder under it.  This folder will automatically be configured for camels Local Working Directory when using the Stream Path.  

 

See also:

FTP Example: Alternate to using "Large File Handling Options" for FTP?

Other examples: How to use Large File Handling Options?