Automate Rollback of Serverless app with AWS CodeDeploy and Serverless Framework.

Introduction

In this article, we’re going to understand how we can leverage AWS CodeDeploy and AWS Lambda versions to deploy our functions in a blue-green deployment manner to reduce the occurrence of errors of our application in a production system.

Let’s have a look at a typical scenario. You have deployed a couple of lambda functions in production, and there’s a steady traffic flow. There’s a new business requirement that requires a logic update in the lambda function. When deploying the updated lambda, it may work as expected or fail because of invocation errors, code errors, higher execution times, timeouts, etc. Remember that, unless you specify explicitly, the lambda will be invoked on its $LATEST version. The problem with this approach is that the lambda failure results in application flow failures associated with the lambda.

Is there a better way to handle this scenario? Yes! Introducing AWS CodeDeploy, a fully managed deployment service. Using AWS CodeDeploy and Amazon CloudWatch, an application and infrastructure monitoring tool, you can automate the blue-green deployment of lambdas. Based on the criteria set on CloudWatch, CodeDeploy will incrementally shift the traffic from the older version of lambda to the newer version. If the newly lambda version fails during this traffic shift, a CloudWatch alarm will be triggered, forcing CodeDeploy to roll back the traffic to the older stable version of the lambda function.

Architecture overview

Implementation

Pre-requisites

AWS Account
Familiarity with AWS Lambda and Serverless Framework

Steps

Install serverless framework packages.
Define and configure resources in serverless.yaml file
Deploying the resources in AWS
Update Lambda function and watch the rollback in action.

Step 1:

Here, we’re using Serverless Framework for deployment, for that we need to install two additional dependencies for the framework to work.

serverless-plugin-canary-deployments: This package helps to split the traffic between different lambda versions.
serverless-plugin-aws-alerts: This package is related to CloudWatch alarms. It helps us in setting conditional alarms for other resources.

npm install serverless-plugin-aws-alerts serverless-plugin-canary-deployments

Step 2:

After installing the packages, time to configure the setup in the serverless.yml file. Use the sample code below to replace the content in the template file,

# Phase-1:
service: sls-canary-sls
provider:
  name: aws
  runtime: nodejs12.x
  region: ap-south-1
package:
  exclude:
    - 'node_modules/**'
    - 'yarn.lock'
    - 'package.json'
plugins:
  - serverless-plugin-aws-alerts
  - serverless-plugin-canary-deployments


#Phase-2:
custom:
  alerts:
    dashboards: true
    definitions: 
      FunctionErrors:
        namespace: 'AWS/Lambda'
        metric: Errors
        threshold: 1
        statistic: Minimum
        period: 60
        evaluationPeriods: 1
        comparisonOperator: GreaterThanOrEqualToThreshold


#Phase-3:
functions:
  Order:
    handler: orders.getOrders
    events:
      - http:
          method: GET /order
        alarms:
          - FunctionErrors
    deploymentSettings:
      type: Linear10PercentEvery1Minute
      alias: Live
      alarms: 
        - OrderFunctionErrorsAlarm
  Product:
    handler: products.getProducts
    events:
      - http: get /product
    alarms:
      - FunctionErrors
    deploymentSettings:
      type: Linear10PercentEvery1Minute
      alias: Live
      alarms: 
        - ProductFunctionErrorsAlarm

To understand what’s happening, we’ve broken down the into three phases.

Phase1 - describes the general configuration, i.e., cloud provider name, deployment region, runtime for the code, excluding files/folders that are not required, and importing the packages.
Phase2 - we’re defining the alarm based on the metric type Errors followed by evaluation period 1, duration 60s with a minimum threshold of one. What does it mean? Within the deployment process, if at least one error occurred in-between every 60second, the alarm will be triggered, which will force CodeDeploy to roll back the lambda to the previous version.
Phase3 - we’re configuring our lambda functions. In our case, we’re using two Lambda functions, namely Order and Product. As you would have noticed, we’re setting up two events for each function, i.e., an API Gateway and the CloudWatch Alarm. The configuration under the deployment settings is related to CodeDeploy. In this configuration, CodeDeploy will shift traffic to the new lambda version at the rate of 10% in every 1min of interval.

Step 3:

Now, it’s time to deploy our configuration to AWS. To make it happen, I would assume you have done all the setup from your end. If not, you may check out the sample here or run the following command,

sls deploy

If you want to see the modeling of the resources you defined in the .yml template, you could view the deployed resources in the AWS CloudFormation console. Note that the CloudFormation will take some time to spin up the resources. Every time you make some changes and deploy, traffic will move to the newer version at a specific rate of the interval of time instead of shifting the entire traffic towards the newer version.

Step 4:

We have reached the last step of this article. Here we’ll make some changes to the code and deploy it to visually see how the traffic is shifting and navigate to the AWS CodeDeploy console. The image below shows the process.

And there we have it! Worth noting that you could eventually make some errors in the code, deploy it and spot the changes in the CodeDeploy console.

Conclusion

We’ve seen how we can leverage AWS CodeDeploy and Amazon CloudWatch alarms to deploy our Lambda functions in a blue/green deployment manner and gracefully roll back to the previous version in case of an error.