Sidebar

How can I read or write data to an AWS S3 Bucket?

+1 vote
849 views
asked Jul 26, 2022 by ben-s-7515 (12,320 points)
I need to send and receive data from an AWS S3 bucket.  I can't find any QIE standard receiver or sender to accomplish this.  How do I read or write data to an AWS S3 bucket?

1 Answer

+1 vote

When you can't find a standard receiver or sender for connecting to a remote system, but that remote system has a Java library to help with the connection, you can use a "Custom Script" receiver and sender to accomplish the task.  The "Custom Script" gives you the ability to write your own receiver or sender.

To connect to an S3 bucket, we will use the 'aws-java-sdk-s3' libraries with any dependencies.  You will need to follow these instructions to load the libraries into the QIE system.  Once you have them loaded, you can create a new channel that will read from the S3 bucket using the following custom script:

/*
   NOTE: This example requires the use of the aws-java-sdk-s3 libraries and their dependencies to function correctly
*/
// This is the bucket that we will read from
var bucketName = bucketName = "test-bucket-12345678899-123141231";
// This example uses the basic aws credentials to establish the connection to the bucket
var credentials = new com.amazonaws.auth.BasicAWSCredentials(qie.getVarUsername('awsCredentials'), qie.getVarPassword('awsCredentials'));
// The s3Client is used to actually read from the bucket
var s3Client = com.amazonaws.services.s3.AmazonS3ClientBuilder.standard().withCredentials(new com.amazonaws.auth.AWSStaticCredentialsProvider(credentials)).withRegion(com.amazonaws.regions.Regions.US_EAST_1).build();

// We list out all of the files found in the bucket and then get the summaries of them
var objects = s3Client.listObjects(bucketName);
var objectSummaries = objects.getObjectSummaries();
// For each object, we will read the bytes, place the file into the inbound queue, then delete the object from the bucket
for (var i = 0; i < objectSummaries.size(); i++) {
   var os = objectSummaries.get(i);
   var object = s3Client.getObject(bucketName, os.getKey());
   var inputStream = object.getObjectContent();
   qie.warn('Reading ' + os.getKey() + ' from S3 bucket');
   var buffer = java.lang.reflect.Array.newInstance(java.lang.Byte.TYPE, 1048576);
   var count = inputStream.read(buffer);
   var messageOutputStream = new java.io.ByteArrayOutputStream();
   while (count != -1) {
      qie.debug('Read Count: ' + count);
      messageOutputStream.write(buffer, 0, count);
      count = inputStream.read(buffer);
   }
   // add to inbound
   qie.addInboundMessage(messageOutputStream.toByteArray(), os.getKey());
   // delete the source file
   s3Client.deleteObject(bucketName, os.getKey());
}

 

Next, we will use a "Custom Script" sender to write our output to a bucket.  This script will be much simpler because we just write the file out.  We are not getting a list of files to process.

/*
NOTE: This example requires the use of the aws-java-sdk-s3 libraries and their dependencies to function correctly
*/
var bucketName = "test-bucket-12345678899-123141231";
var credentials = new com.amazonaws.auth.BasicAWSCredentials(qie.getVarUsername('awsCredentials'), qie.getVarPassword('awsCredentials'));
var s3Client = com.amazonaws.services.s3.AmazonS3ClientBuilder.standard().withCredentials(new com.amazonaws.auth.AWSStaticCredentialsProvider(credentials)).withRegion(com.amazonaws.regions.Regions.US_EAST_1).build();

s3Client.putObject(bucketName, qie.getFileName(), message.getNode('/'));
 

NOTE: The username and password for the basic authentication is stored in a system variable of type credentials that is named 'awsCredentials'.

 

answered Jul 26, 2022 by ben-s-7515 (12,320 points)
...