Aniket
Feb 25, 2023
  969
(0 votes)

Sending Files to Amazon S3 storage

I am sure you have been asked by clients to create a scheduled job that generates a report and sends it to third party system for further processing.

One of our client asked us to generate a daily report of all the completed orders for further processing in Snowflake. We looked at multiple options, but one option that stood out was creating a scheduled job and storing these report/CSV files in Amazon's S3 storage (considering it was heavily used by the client). 

The integration to Amazon's S3 was relatively simpler than I thought.

Step 1: Keys/Credentials for S3 storage

Once the client to generate the keys and the bucket and provide us with the following:

  • User Name
  • Access Key ID
  • Secret Access Key

Step 2: Install S3 Browser

This tool allows you to browse all the buckets with the client's S3 storage. You need to use the above credentials to browse the reports generated and published to this bucket.

URL: https://s3browser.com/download.aspx

Step 2: Install the AWS S3 SDK

Step 3: Code

Add the required code to transfer CSV files to AWS S3 storage

namespace ClientName.Orders.DailyOrders.Services
{
    using System;
    using Amazon;
    using Amazon.S3;
    using Amazon.S3.Transfer;

    public static class AmazonUploaderService
    {
        public static bool SendMyFileToS3(System.IO.Stream fileStream, string fileNameInS3)
        {
            try
            {
                var client = new AmazonS3Client(Amazon.RegionEndpoint.USEast1);

                // create a TransferUtility instance passing it the IAmazonS3 created in the first step
                TransferUtility utility = new TransferUtility(client);

                // making a TransferUtilityUploadRequest instance
                TransferUtilityUploadRequest request = new TransferUtilityUploadRequest();

                request.BucketName = System.Configuration.ConfigurationManager.AppSettings["AWSBucketName"];
                request.Key = fileNameInS3;
                request.InputStream = fileStream;
                utility.Upload(request);
            }
            catch (Exception ex)
            {
                return false;
            }

            return true; // indicate that the file was sent
        }
    }
}

Call the above function using the following code.

_s3FileName = $"{DateTime.UtcNow.ToString("yyyy-MM-dd-hh-mm-ss")}-Snowflake-W2-orders.csv";
StringBuilder _report = "Col1, Col2, Col3, Col4..."; // List of all orders using string builder
if (!string.IsNullOrWhiteSpace(_report?.ToString()))
{
  byte[] byteArray = Encoding.ASCII.GetBytes(_report.ToString());
  using (MemoryStream memoryStream = new MemoryStream(byteArray))
  {
    uploaded = AmazonUploaderService.SendMyFileToS3(memoryStream, _s3FileName);
  }
}

Step 4: Test using S3 Browser

Once the code is run successfully you should be able to view a list of all the reports generated using S3 browser you installed in step 2.

That's it! Happy coding :)

Feb 25, 2023

Comments

Please login to comment.
Latest blogs
The missing globe can finally be installed as a nuget package!

Do you feel like you're dying a little bit every time you need to click "Options" and then "View on Website"? Do you also miss the old "Globe" in...

Tomas Hensrud Gulla | Feb 14, 2025 | Syndicated blog

Cloudflare Edge Logs

Optimizely is introducing the ability to access Cloudflare's edge logs, which gives access to some information previously unavailable except throug...

Bob Davidson | Feb 14, 2025 | Syndicated blog

Comerce Connect calatog caching settings

A critical aspect of Commerce Connect is the caching mechanism for the product catalog, which enhances performance by reducing database load and...

K Khan | Feb 14, 2025

CMP DAM asset sync to Optimizely Graph self service

The CMP DAM integration in CMS introduced support for querying Optimizly Graph (EPiServer.Cms.WelcomeIntegration.Graph 2.0.0) for metadata such as...

Robert Svallin | Feb 13, 2025