Skip to content

Passing Through Large Files Without Loading Them Into Memory

Large files (anything from multi-GB clinical attachments to nightly dumps) exhaust the JVM heap if they are loaded as the working message in the usual way. The FTP, File, and Network Share source nodes have a Large File Handling mode that streams the file straight to a local working directory and passes the channel only the file name, not the contents. From there a mapping script uses qie.moveFile (for local destinations) or one of the qie.writeFTPFile* / qie.writeSFTPFile functions (for FTP/SFTP destinations) to relocate the file without ever opening it in memory.

This recipe covers the source-side configuration, the destination choice, and the mapping script that does the move. The same pattern works on all three streaming-capable source types.

Source-side configuration

On the source node, expand Large File Handling Options and check Stream messages to disk. Two sub-fields appear:

  • Stream Path: a local folder QIE can write to. QIE creates a tmp_lwd subfolder inside it (the Camel "local working directory") and writes each incoming file there before handing the channel its name. The path must be on a volume with enough room for the largest expected file and must be writable by the QIE service account.
  • Enable resume download (FTP source only). When checked, QIE asks the FTP server to resume a partial transfer instead of restarting from scratch. Leave it off if the server does not support REST / APPE. SFTP never supports resume; the option is ignored for SFTP connections.

Once streaming is enabled, the message that flows through the channel is just the file name on disk. Reading or templating the message body in a mapping node, using message.getNode('/'), qie.readFile(...) or qie.readFTPFile*(...), pulls the whole file back into memory and defeat the purpose. Stick to script functions that operate on file handles.

Destination node

Set the destination to Discard. The destination has no content to send (only the file name is on the message) so any other sender (File, FTP, MLLP, Web Service, etc.) would either fail or quietly send the file name as the payload. The mapping node is what actually moves the bytes; the destination just marks the message complete.

Mapping script

Get a handle to the streamed file with qie.newFile(qie.getStreamPath() + '/' + source.getFileName()), then either move it to another local path or upload it to a remote server.

Local-to-local

var streamedFile = qie.newFile(qie.getStreamPath() + '/' + source.getFileName());
var destinationFile = qie.newFile('/destinationFolder/' + source.getFileName());
qie.moveFile(streamedFile, destinationFile);

qie.moveFile is a rename on the same volume and a streamed copy + delete across volumes. Either way the data never touches the JVM heap.

To an FTP / SFTP server

Use the matching qie.writeFTPFile* or qie.writeSFTPFile function. The file handle goes in as the last argument:

var streamedFile = qie.newFile(qie.getStreamPath() + '/' + source.getFileName());
var destinationPath = '/destinationFTPFolder/' + source.getFileName();
qie.writeFTPFileNoEncryption('host', port, destinationPath, binary, passive,
                             'ftpParameters', 'userName', 'password',
                             streamedFile);

The available variants (qie.writeFTPFileNoEncryption, qie.writeFTPFileExplicitTLS, qie.writeFTPFileImplicitTLS, and qie.writeSFTPFile) all accept a file handle in their final argument and stream the file to the remote server without loading it.

Alternative: raw Camel parameters

If the source needs Custom Apache Camel Parameters that conflict with the Large File Handling Options UI section, you can drive the same behavior by hand. On the FTP source's Custom Apache Camel Parameters field, set:

streamDownload=true
stepwise=false
localWorkDirectory=/path/to/stream/dir
binary=true
resumeDownload=true

When localWorkDirectory is set, QIE creates a download subfolder there (rather than tmp_lwd) and uses the same file-name-only message contract. The mapping script is identical.

Caveats

  • The Stream Path / localWorkDirectory folder must exist and be writable by the QIE service account. QIE creates the tmp_lwd (or download) subfolder automatically but cannot create the parent.
  • Never call qie.readFile, qie.writeFile, or any qie.readFTPFile* / qie.readSFTPFile from a mapping node when streaming is on. They all load the whole file into memory.
  • resumeDownload only works against FTP servers that advertise the capability. SFTP servers do not support resume regardless of the setting.
  • The channel is responsible for removing the streamed file after it has been used. qie.moveFile removes the source as part of the move; if the script copies instead, add an explicit delete or QIE leaves files in the Stream Path indefinitely.