FTP large files from a folder

0 votes
asked May 3, 2021 by ben-s-7515 (12,320 points)
I have a folder that gets large files (100-900MB) and I need to SFTP the files to an FTP server, but I am having trouble with running out of memory.  I don't need to transform the message, just need to move it.

Is there a more efficient way to FTP a large file without using so much memory?

2 Answers

0 votes
Best answer
As of the 5.0.51 release in (Aug, 2022) there is an option to stream the file to disk by using the "Large File Handling Options".   This new option makes it easier to handle large files in the FTP receiver.


answered Aug 24 by ben-s-7515 (12,320 points)
0 votes

If you don't need to transform the message, but only need to move the files from one folder to another or an SFTP endpoint, the best way to do it is using Streams.  This process would be as follows:

1) Create a custom receiver, this reciever reads the list of filenames and filters out any you don't want.  Instead of reading the whole file, you add a message that is the full file path only.  You will only send in a single file at a time.  The script should skip processing if there is a message inbound, processing, or outbound to prevent duplicate processing of files.
2) In the mapping node, you create a new'/')) object that will be passed into the send SFTP command.  By passing in the File object, the FTP command will stream the FTP instead of pushing the bytes directly meaning the whole file is never loaded into memory at any time.
3) Create a second mapping function that will stream the file bytes to a archive location if desired.
4) After moving the file where you want it to go, you will then delete the source file from another mapping function.  This will prevent it from being processed a second time.
5) Since the file move happened in a mapping node, you will send the message to a discard destination node.

This exmple channel shows you how to accomplish this process.  NOTE: By using a WS Connection, we are able to store the username and password to the SFTP endpoint without exposing it in plain text.

answered May 3, 2021 by ben-s-7515 (12,320 points)