Uploading file to S3 in NodeJs using multer

There are several ways to upload files where usually when a file is uploaded to the server, it is saved in the server and then the server reads the file and sends it to S3. Using multer we can avoid the middle step of saving the file in the server and directly upload it to S3. This speeds efficient and reduces time when uploading a file. Now let's start by looking into S3.

Amazon Simple Storage Service (S3) is an object storage service that allows you to store and retrieve data in the cloud.S3 provides a simple web services affiliate that can be used to store and recoup any quantum of data, at any time, from anywhere on the web. A S3 bucket is a container for storing files in the S3 service. Every item is identifiable by a special key and kept in its own bucket. Buckets are used to organize and manage data in the S3 service, and can be thought of as similar to a directory or folder on a file system.

Before creating a S3 bucket we need a set of credentials to access the bucket programmatically, for that first you need to create an account in AWS, once After the account is set up, an IAM user must be created. . For that click your name on the top right and then “My Security Credentials”.

On the left navigation menu click “Users” and then add “Add user”.

Enter the desired username and click next.

In “Set permissions” page choose “Attach policies directly” and search for “S3” then choose “AmazonS3FullAccess”

Finally in “Review and create” page review the user details and click on “Create user”.

Now our user is created, now we need to generate access keys for that user to access our bucket programmatically. Go to the IAM page and click users in the left navigation menu and then click on the user who you created. On the right pane, click on “ Security credentials ” and scroll to bottom. Under “ Access keys ” click on “ produce access key ”

On the" Access crucial stylish practices & druthers " runner, choose “ operation running outside AWS ” and also click no “ Next ”.

In the “ Set description label ” runner, type a meaningful description which will help you identify the key.

That's it, the keys are created, in the next page, copy the “Access key”, “Secret access key” and save them, we'll be using these to access the bucket.

To create a S3 bucket go to S3 homepage.

Click on the Create bucket button.

S3 bucket names need to be unique, and they can't contain spaces or uppercase letters. Select the AWS Region that you want the bucket to live in under Region.

Remove the checkbox next to Block all public access if your S3 bucket has public access. S3 bucket can also be private, if the bucket is private we cannot view the file uploaded directly by accessing the link. We can only download the file. In this tutorial we have kept the bucket public

Finally click Create bucket to create the bucket.

Indeed now our pail won't be completely public, to make it go public we need to modify the pailpermissions.Edit the authorization under policy and bury the below values in it

                                
                                    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AddPublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::{{bucket-name}}/*"
        }
      ]
    }
                                
                            

Now all tutorials will read the local file in the system and upload it to S3, but we will use multer to read the file from form-data and upload it to s3 without saving the file locally.

The installation of the following npm modules is required for this to function.
 1. express
 2. body-parser
 3. express-xml-bodyparser
 4. multer
 5. aws-sdk

After installing the above packages using npm command, create a file named app.js and paste the below code in it.

                                
                                    
  const express = require("express");
  const app = express();
  const BodyParser = require("body-parser");
  const xmlparser = require('express-xml-bodyparser');
  const multer = require('multer');
  const AWS = require('aws-sdk');
  const path = require('path');
  
  app.use(express.json())
  app.use(BodyParser.json());
  app.use(BodyParser.text());
  app.use(BodyParser.urlencoded({ extended: true }));
  app.use(express.urlencoded({ extended: true }));
  app.use(xmlparser());
  const storage = multer.memoryStorage({
      destination: function (req, file, cb) {
          cb(null);
      },
      filename: function (req, file, cb) {
          cb(null)
      }
  });
  const upload = multer({ storage: storage });
  
  app.post("/uploadimage-to-s3", upload.single('imagefile'), async (req, res) => {
      try {
          let incomingFile = req.file;
          let extensionUsingPath = path.extname(incomingFile.originalname);
          let fileName = Date.now() + '_' + 'your_file_name' + extensionUsingPath;
          const s3 = new AWS.S3({
              accessKeyId: YOUR_AWS_S3_ACCESS_KEY_ID,
              secretAccessKey: YOUR_AWS_S3_SECRET_ACCESS_KEY,
              region: YOUR_AWS_S3_REGION
          });
          const params = {
              Bucket: YOUR_AWS_S3_BUCKET,
              Key: fileName,
              ACL: 'public-read',
              ContentType: incomingFile.mimetype,
              Body: incomingFile.buffer
          };
          let uploadedLocation = await s3.upload(params).promise();
          res.json({ fileURL: uploadedLocation.Location });
      }
      catch(err) {
          console.log(err);
          res.json({ error: err });
      }
  });
                                
                            

Run the application using the command

                                
                                    
  node app.js
  
                                
                            

We'll be using postman to upload the file, so let hit the below url with the file being sent as form-data

Therefor we have successfully uploaded a file to S3 in node using multer without saving the file locally in the system/server.

• VS Online Services : Custom Software Development

VS Online Services has been providing custom software development for clients across the globe for many years - especially custom ERP, custom CRM, Innovative Real Estate Solution, Trading Solution, Integration Projects, Business Analytics and our own hyperlocal e-commerce platform vBuy.in and vsEcom.

We have worked with multiple customers to offer customized solutions for both technical and no technical companies. We work closely with the stake holders and provide the best possible result with 100% successful completion To learn more about VS Online Services Custom Software Development Solutions or our product vsEcom please visit our SaaS page, Web App Page, Mobile App Page to know about the solution provided. Have an idea or requirement for digital transformation? please write to us at siva@vsonlineservices.com