Uploading file to S3 in NodeJs using multer

There are several ways to upload files where usually when a file is uploaded to the server, it is saved in the server and then the server reads the file and sends it to S3. Using multer we can avoid the middle step of saving the file in the server and directly upload it to S3. This is more efficient and reduces time when uploading a file. Now let's start by looking into S3.

Amazon Simple Storage Service (S3) is an object storage service that allows you to store and retrieve data in the cloud. S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. A S3 bucket is a container for storing files in the S3 service. Each object is stored in a bucket and is identified by a unique key. Buckets are used to organize and manage data in the S3 service, and can be thought of as similar to a directory or folder on a file system.

Before creating a S3 bucket we need a set of credentials to access the bucket programmatically, for that first you need to create an account in AWS, once the account is set up we need to create a IAM user. For that click your name on the top right and then “My Security Credentials”.

On the left navigation menu click “Users” and then add “Add user”.

Enter the desired username and click next.

In “Set permissions” page choose “Attach policies directly” and search for “S3” then choose “AmazonS3FullAccess”

Finally in “Review and create” page review the user details and click on “Create user”.

Now our user is created, now we need to generate access keys for that user to access our bucket programmatically. Go to the IAM page and click users in the left navigation menu and then click on the user who you created. On the right pane, click on “Security credentials” and scroll to bottom. Under “Access keys” click on “Create access key”

On the "Access key best practices & alternatives" page, choose “Application running outside AWS” and then click no “Next”.

In the “Set description tag” page, type a meaningful description which will help you identify the key.

That's it, the keys are created, in the next page, copy the “Access key”, “Secret access key” and save them, we'll be using these to access the bucket.

To create a S3 bucket go to S3 homepage.

Click on the Create bucket button.

S3 bucket names need to be unique, and they can't contain spaces or uppercase letters. In Region, choose the AWS Region where you want the bucket to reside.

Remove the checkbox next to Block all public access if your S3 bucket has public access. S3 bucket can also be private, if the bucket is private we cannot view the file uploaded directly by accessing the link. We can only download the file. In this tutorial we have kept the bucket public

Finally click Create bucket to create the bucket.

Even now our bucket will not be fully public, to make it go public we need to modify the bucket permissions.Edit the permission under policy and paste the below values in it

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AddPublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::{{bucket-name}}/*"
        }
    ]
}

Now all tutorials will read the local file in the system and upload it to S3, but we will use multer to read the file from form-data and upload it to s3 without saving the file locally.

To make this work we need the following npm modules to be installed
 1. express
 2. body-parser
 3. express-xml-bodyparser
 4. multer
 5. aws-sdk

After installing the above packages using npm command, create a file named app.js and paste the below code in it.

const express = require("express");
  const app = express();
  const BodyParser = require("body-parser");
  const xmlparser = require('express-xml-bodyparser');
  const multer = require('multer');
  const AWS = require('aws-sdk');
  const path = require('path');
  
  app.use(express.json())
  app.use(BodyParser.json());
  app.use(BodyParser.text());
  app.use(BodyParser.urlencoded({ extended: true }));
  app.use(express.urlencoded({ extended: true }));
  app.use(xmlparser());
  const storage = multer.memoryStorage({
      destination: function (req, file, cb) {
          cb(null);
      },
      filename: function (req, file, cb) {
          cb(null)
      }
  });
  const upload = multer({ storage: storage });
  
  app.post("/uploadimage-to-s3", upload.single('imagefile'), async (req, res) => {
      try {
          let incomingFile = req.file;
          let extensionUsingPath = path.extname(incomingFile.originalname);
          let fileName = Date.now() + '_' + 'your_file_name' + extensionUsingPath;
          const s3 = new AWS.S3({
              accessKeyId: YOUR_AWS_S3_ACCESS_KEY_ID,
              secretAccessKey: YOUR_AWS_S3_SECRET_ACCESS_KEY,
              region: YOUR_AWS_S3_REGION
          });
          const params = {
              Bucket: YOUR_AWS_S3_BUCKET,
              Key: fileName,
              ACL: 'public-read',
              ContentType: incomingFile.mimetype,
              Body: incomingFile.buffer
          };
          let uploadedLocation = await s3.upload(params).promise();
          res.json({ fileURL: uploadedLocation.Location });
      }
      catch(err) {
          console.log(err);
          res.json({ error: err });
      }
  });

Run the application using the command

node app.js

We'll be using postman to upload the file, so let hit the below url with the file being sent as form-data

Therefor we have successfully uploaded a file to S3 in node using multer without saving the file locally in the system/server.

VS Online Services : Custom Software Development

VS Online Services has been providing custom software development for clients across the globe for many years - especially custom ERP, custom CRM, Innovative Real Estate Solution, Trading Solution, Integration Projects, Business Analytics and our own hyperlocal e-commerce platform vBuy.in and vsEcom.

We have worked with multiple customers to offer customized solutions for both technical and no technical companies. We work closely with the stake holders and provide the best possible result with 100% successful completion To learn more about VS Online Services Custom Software Development Solutions or our product vsEcom please visit our SaaS page, Web App Page, Mobile App Page to know about the solution provided. Have an idea or requirement for digital transformation? please write to us at siva@vsonlineservices.com