Handling Large DICOM Images¶

A single DICOM instance can be enormous, multi-frame CT and tomosynthesis series, whole-slide imaging, and cine runs routinely exceed a gigabyte, and instances above 2 GB are possible. Two limits make these messages fail when they are handled naively:

The Java single-array limit. No Java array can hold more than 2,147,483,639 bytes (about 2 GiB). Any operation that materializes a whole DICOM instance into one byte array or string fails for instances at or above that size. No matter how much heap the server has.
Heap pressure. Even below the hard limit, holding whole images in memory multiplies per-message memory usage and can exhaust the heap under load.

When a script or sender hits the array limit, the message errors with a failure like this, and adding memory to the server does not fix it:

Required array length 2147483639 + 1334 is too large

Handled correctly, neither limit applies: QIE can receive, route, and deliver instances of any size without ever holding the pixel data in memory. That takes the source-node option below, plus (only when a script must produce the complete file itself) the streaming script pattern.

Source node: store pixel data in a separate folder¶

On a DICOM Listener, File, or Custom Script source node whose message format is DICOM, the DICOM Format section has a checkbox:

Enable storing DICOM Pixel Data in a separate folder

When checked, QIE peels the pixel data off the incoming instance as it arrives, streams it (encrypted) to a file under the configured DICOM Storage path, and passes only the non-pixel attributes through the channel. Mapping and condition scripts work on a small message; when the instance is sent onward, QIE streams the pixel data back in from disk and the receiving system gets a complete DICOM file.

Warning

Without this option, the entire instance (pixel data included) is held in memory while the message is processed. Channels that can receive large images should always enable it. The DICOM Storage path must be on a volume with enough space for the pixel data of all in-flight messages.

With pixel data stored in a separate folder, the built-in File and DICOM (DIMSE) destinations stream automatically and need nothing further. The rest of this recipe is only needed when a script has to produce or transmit the complete file itself, for example a REST upload to a DICOMweb endpoint.

Script rules: never materialize a large instance¶

These operations rebuild the complete instance (pixel data included) into a single in-memory value, so they fail at the array limit and should be avoided for large images:

message.getDICOMBytesForFile() and message.getBytesForFileSystem(), one byte array of the whole file
Wrapping those bytes in a java.lang.String, or base64-encoding them, same limit, with the memory roughly doubled
message.changeTransferSyntax(...). Transcodes through full in-memory copies, and decompressing a compressed image can inflate it several-fold

Streaming the file from a script¶

message.writeDICOMToFile(filePath) streams the complete DICOM file (preamble, File Meta Information, dataset, and the separated pixel data) to disk without loading it into memory. Combined with the File content form of qie.callRESTWebService, which streams the request body straight from disk, a script can upload an instance of any size:

var tempPath = java.lang.System.getProperty("java.io.tmpdir") +
  java.io.File.separator + "dicom-" + java.util.UUID.randomUUID() + ".dcm";
try {
  // Stream the complete DICOM file to disk (constant memory, any size)
  message.writeDICOMToFile(tempPath);

  // POST it; a File as content streams from disk to the socket
  var response = qie.callRESTWebService(
    "myConnection",
    qie.getWsEndpointUrl("myConnection") + "studies",
    "POST",
    qie.newFile(tempPath),
    "application/dicom",
    qie.newParameterMap(),
    600000, // timeout (ms) - size to the worst-case upload time
    true);  // fullResponse
  // handle the response ...
} finally {
  java.nio.file.Files.deleteIfExists(java.nio.file.Paths.get(tempPath));
}

Note

The temp file is as large as the instance itself. Point it at a volume with room to spare (the DICOM Storage volume is a natural choice) and always delete it in a finally block so failed uploads do not accumulate files.

Checking the size before choosing a path¶

When the pixel data is stored in a separate folder, QIE records its size on the message itself, so a script can branch on size without touching the file:

var pixelBytes = message.getSeparatePixelDataLength();

An in-memory approach is fine below a comfortable threshold (say 900 MB) and avoids the temp file; above it, use the streaming pattern. The recorded size is the pixel data length in the original transfer syntax (the same bytes that are reattached on output) so it predicts the written file's size to within the (small) size of the non-pixel attributes.

getSeparatePixelDataLength() returns 0 when the pixel data is not stored separately (use message.isPixelDataStoredSeparately() to distinguish that case), and 4294967295 when the instance arrived with encapsulated (compressed) pixel data of undefined length, the true size is unknown, so a pixelBytes > threshold check naturally routes it to the streaming path.

Caveats¶

QIE tracks the separately stored pixel data in private tags (group 0511, or the next available odd group when the source system already uses 0511). These tags are QIE-internal. Do not modify or remove them, or the pixel data cannot be reattached on output. Read the storage state through message.isPixelDataStoredSeparately() and message.getSeparatePixelDataLength() rather than the raw tags.
The File destination's streaming path requires the channel persistence level to be 1 or higher; at level 0 it rebuilds the file in memory.
The FTP destination and the Web Service destination's DICOMweb STOW-RS operation currently assemble the outbound file in memory, so both share the array limit for very large instances. Use the script pattern (or a File destination) for those.
A REST upload of a multi-GB file runs for minutes. Size the callRESTWebService timeout to the worst-case transfer time, never the 60-second reflex.
Sending to a DICOMweb endpoint that accepts only multipart/related requires wrapping the file in a multipart body; see Posting multipart/form-data from a Script for the multipart mechanics.