Share on facebook
Share on twitter
Share on linkedin

Learning AWS Lambda Can Help Automate Your Build Pipeline

A Cloud Guru News
A Cloud Guru News

How to create an AWS Lambda serverless function that automatically invokes Webpack and bundles JavaScript

AWS Lambda, and the serverless ecosystem in general, are changing the way we create and think about software in the cloud.

With serverless, we don’t have to fret over our infrastructure — we can just focus on coding our service. And while Serverless isn’t perfect for all occasions, there is a diverse set of circumstances in which its use becomes both quite handy and very powerful.

Consider the advantages of applying serverless to automate a build pipeline. What if you could use serverless to eliminate reliance on tools like Jenkins or Bamboo? Instead of relying on an automation server, what if you could leverage the power of cloud services to use Jenkins as a launcher that invokes multiple AWS Lambda functions, in parallel or in sequence?

Last night, I explored the possibilities while working on a spike to automate the generation of a Javascript bundle using Webpack. Using AWS Lambda, I invested some time to create a serverless function that invokes Webpack and bundles JavaScript.

I started with a simple example that contains lodash and some ES6 code using this example provided in Webpack’s docs:

import _ from ‘lodash’;
function component () {
     var element = document.createElement(‘div’);
     element.innerHTML = _.join([‘Hello’,’webpack’], ‘ ‘);
     return element;
}
document.body.appendChild(component());

The goal was to bundle this ES6 code and its library into a unique Javascript file that can be used inside an application or website — mimicking a build pipeline. In my example, I use Webpack since that’s what is used at work — but you can choose any executable to invoke from you AWS Lambda function.

AWS Lambda to the Rescue

If you use a cloud-based automation pipeline and don’t have DevOps resources on your team or at your company, you should spend some time learning AWS Lambda — it could help out with these kind of activities.

AWS Lambda is a compute service that lets you run code without provisioning or managing servers. AWS Lambda executes your code only when needed and scales automatically, from a few requests per day to thousands per second.

Too good to be true? No. But AWS Lambda does have some limitations:

Limitations as of March 2017.— current service limitations are available in the Lambda documentation.

Building the AWS Lambda Function

So how can we use AWS Lambda serverless functions to bundle our Javascript project with Webpack?

First things first: I created a git repository where you can find a Javascript project to use inside an AWS Lambda function. This way, you won’t need to create a custom project, and you can focus more on the AWS side.

Here are few tips to save you some time:

  • Lambda functions can save temporary files inside the /tmp/ folder — bear in mind that you are running your code inside a Docker container!. If you try to save somewhere else, you will receive an error.
  • With AWS Lambda, you can run executables or node CLI tools like NPM or Webpack by uploading them into your Lambda environment and referring to them with relative path.
  • AWS Lambda will not run for more than 300 seconds. Therefore, if you have a complex operation, you should split it into different chained Lambda functions that are triggered in sequence. — which should help you in debugging your operations, too.

In the project I set up the webpack config file in this way:

var path = require('path');
module.exports = {
   entry: './app/index.js',
   output: {
      filename: 'bundle.js',
      path: path.resolve('/tmp/')
   }
};

As you can see, I’m saving my bundle in the tmp folder because it’s the only one with write permissions. Remember the capacity limit of 512MB for that folder.

Then, I created an index.js file where I describe my Lambda function:

var spawn = require('child_process').spawn;
var fs = require('fs');
exports.handler = (event, context, callback) => {
   var wp = spawn('./node_modules/.bin/webpack', ['--config', 'webpack.config.js']);
   wp.stdout.on('data', function(data){
     console.log('stdout: ' + data);
   });
   wp.stderr.on('data', function(err){
     context.fail("writeFile failed: " + err);
   });
   wp.on('close', (code) => {
     fs.readFile('/tmp/bundle.js', 'utf8', function (err, data) {
         if (err) context.fail("read file failed: " + err);
         context.succeed(data);
     });
   });
};

I’m using Node for this very simple code. You could also use Python or Java which are also supported by AWS Lambda at the moment — pick your favorite.

I import the spawn method in order to run Webpack. Once it has finished, I read the content inside the Javascript bundle file created by Webpack in the tmp folder, and I return it via the context.succeed method.

Context is an object, always available inside a Lambda function, that will allow you to interact with Lambda to retrieve some useful information or, in this case, to define when the function succeed or failed.

We are now ready to upload the application into the Lambda function.
In order to do that, you will need to zip your project files — just the files and not the parent folder — and upload then to AWS.

If you didn’t install the project dependencies after cloning the repository, you should do it before uploading it to AWS.

Select and compress only the files inside your project not the parent folder

Inside your AWS console, after selecting Lambda service, you should be able to create a new function — just please pick an AWS Region where Lambda is available!

Choose your favorite language,in my case Node 4.3, and define the basic information like “function name”, “description”, and so on. Then, instead of writing the Lambda function code inside the editor provided by AWS, open the dropdown and select Upload a ZIP file.

Select upload a ZIP file

Next, set up the handler, role, and advanced settings as follows:

it’s important set at least 30 seconds as Timeout period

The important part here is setting up the Docker container where the Lambda will run with enough memory size and with a decent timeout. Because we are running an executable, we don’t want to block the execution due to a Lambda timeout.

If, for any reason, you need to increase the 300 seconds soft limit set by default, you will need to contact Amazon and ask to increase it.

Another important thing to remember is when the Lambda is not active for a certain amount of time (it’s estimated to 5 mins), the container used by your code will be re-used for other Lambda functions.

If your function times-out, it will be recreated when you trigger your function again. But if the Lambda function runs several times in few mins (warm Lambda), you will have better performance because the container will already be available and ready to execute a new activity again.

If you want to live test the function I created, you can trigger it from this link. To test your own Lambda function, click the “test” button and you should see output similar to this:

Output produced by Lambda function that is the content inside the Javascript bundle created by Webpack

Where To Go From Here

The example described is very basic, but it’s useful to consider how these ideas could be expanded:

  1. AWS Lambda functions accept arguments passed when we trigger them, therefore you could potentially upload your project files in S3 and trigger the Lambda function directly after the upload. In fact, Lambda can be triggered by several AWS services, like S3, DynamoDB, SNS, and so on.
  2. In order to expose the Lambda externally, you will need to connect it via API Gateway, another tool provided by AWS. In the example I shared above, I configured the API Gateway to trigger my Lambda function when someone is calling a specific URL.
  3. The fastest way, and my personal recommendation, to work with AWS Lambda is via a CLI tool like Serverless Framework — you won’t need to manually configure the API Gateway and your Lambda environment because Serverless Framework will provide boilerplate to work with. On top of that, it will allow you to test your Lambda functions locally — without uploading them every time on AWS. There are many other CLI tools available, but at the moment Serverless Framework is the most complete and documented, backed by a large user community.
  4. Recently, Amazon added the possibility to set environment variables for Lambda functions. Therefore, if your project requires them, you will be able to easily configure them via the AWS console or inside Serverless framework configuration file.
  5. If you don’t want to upload a CLI tool with the node_modules folder, you can create an executable with all static dependencies and upload that file inside the ZIP file. For Node, I found a tool that works pretty well with Webpack and NPM: EncloseJS.
  6. Remember to not abuse the power of the serverless ecosystem. Rather, try to understand the pros and the cons before starting to use it. Remember, in some cases it’s definitely not the right choice.
  7. An important consideration re: the combination API Gateway + Lambda is that it could work with HTTP2 protocol out of the box, and you can use Cloudfront to cache your responses (also if they are dynamic) with just few configurations to set in the API Gateway console.
  8. With Lambda, you only pay for what you use. Therefore, if you use it for cron services, automation pipelines, triggering database backup operations or similar, you could end up savings quite a few dollars compared to an EC2 instance.

Recommended

Get more insights, news, and assorted awesomeness around all things cloud learning.

Get Started
Who’s going to be learning?
Sign In
Welcome Back!
Thanks for reaching out!

You’ll hear from us shortly. In the meantime, why not check out what our customers have to say about ACG?

How many seats do you need?

  • $499 USD per seat per year
  • Billed Annually
  • Renews in 12 months

Ready to accelerate learning?

For over 25 licenses, a member of our sales team will walk you through a custom tailored solution for your business.


$2,495.00

Checkout