How to stream large files or binary files to disk with qie.callRESTWebService() in mapping function?

Question

I need to call a REST web service in a mapping node to download large files or binary files like images etc. I am worried that the size of the files may be larger than the amount of memory allocated to the Java JVM for QIE. I just need to download the files to a directory so there is no need to open them in QIE.  Can you help me?

Answer 1

As of 24.2.1 a new optional parameter has been added to qie.callRESTWebService() called "streamToFile". This option is helpful when you need to save binary files or large files that do not need to be loaded into memory for processing. It can be used to prevent the QIE JVM from running out of memory.

The "streamToFile" option can be a String to a path and filename, a File(), or a NetFile() object that identifies where the
file should be saved.  I will do a couple examples using the httpbin REST web service to download a jpeg image.

Example using streamToFile with fullResponse=true:

//Uncomment to use a string
var path = '/var/folders/96/z8jsdjcn7fgddrr86thj7v300000gn/T/qie-stream/dog.jpeg';

//Uncomment to use a File
//var file = qie.newFile('/var/folders/96/z8jsdjcn7fgddrr86thj7v300000gn/T/qie-stream/dog.jpeg');

//Uncomment to use a NetFile
//var netFile = qie.newNetFile('uncPath', 'domainUsername', 'password');

var response = qie.callRESTWebService(
   'httpbin',
   "https://httpbin.org/image/jpeg",
   "GET",
   null,
   null,
   null,
   15000,
   true,  //fullResponse
   5000,
   false, //returnBytes
   null,
   path   //streamToFile
   // file
   // netfile
);

qie.info('response: ' + response);

After running the "qie.callRESTWebService()" function, if we go to the path specififed using the "path" variable we find the dog.jpeg was saved:

This response is also returned and assigned to the "response" variable:

<httpResponse>
<responseStatus>
<statusCode>200</statusCode>
<reasonPhrase>OK</reasonPhrase>
<protocolVersion>HTTP/1.1</protocolVersion>
</responseStatus>
<headers>
<header name="Date"><![CDATA[Fri, 12 Jan 2024 18:56:00 GMT]]></header>
<header name="Content-Type"><![CDATA[image/jpeg]]></header>
<header name="Content-Length"><![CDATA[35588]]></header>
<header name="Connection"><![CDATA[keep-alive]]></header>
<header name="Server"><![CDATA[gunicorn/19.9.0]]></header>
<header name="Access-Control-Allow-Origin"><![CDATA[*]]></header>
<header name="Access-Control-Allow-Credentials"><![CDATA[true]]></header>
</headers>
<content>/var/folders/96/z8jsdjcn7fgddrr86thj7v300000gn/T/qie-stream/dog.jpeg</content>
</httpResponse>

Note: The content element in the response XML is set to the path and filename used to save the image.

 

Example using streamToFile with fullResponse=false:

//Uncomment to use a string
var path = '/var/folders/96/z8jsdjcn7fgddrr86thj7v300000gn/T/qie-stream/dog2.jpeg';

//Uncomment to use a File
//var file = qie.newFile('/var/folders/96/z8jsdjcn7fgddrr86thj7v300000gn/T/qie-stream/dog2.jpeg');

//Uncomment to use a NetFile
//var netFile = qie.newNetFile('uncPath', 'domainUsername', 'password');

var response = qie.callRESTWebService(
   'httpbin',
   "https://httpbin.org/image/jpeg",
   "GET",
   null,
   null,
   null,
   15000,
   false,  //fullResponse
   5000,
   false, //returnBytes
   null,
   path   //streamToFile
   // file
   // netfile
);

qie.info('response: ' + response);

When fullResponse is false the image is still saved, this time we called it dog2.jpeg.  But, the response is: null

Note: When using the "streamToFile" option, the "returnBytes" option must be false.  Otherwise, you will see this error:

[line: 11] WrappedException: Wrapped com.qvera.qie.exception.NoResultsException: Cannot use streamToFile when returnBytes is true. 

Stack Trace:
  at script [line: 11]

 

See also:

Handling HTTP requests and responses

Making Requests via qie.callRESTWebService()