Deploy serverless microservices on AWS Lambda: using Terraform

Author: Zen and the Art of Computer Programming

1. Introduction

1.1 Purpose of the article

The main purpose of the article is to help readers better understand how to deploy applications in a serverless microservice architecture on the AWS platform, especially the use of AWS Lambda, a new service type, and the use of a new tool – Terraform. This article will explain the process of deploying serverless microservices from the following aspects:

1. A brief introduction to AWS Lambda;

2. Introduction to Terraform;

3. Use Terrafrom to deploy AWS Lambda functions;

4. Create, test, and publish AWS Lambda functions;

5. Use Amazon API Gateway and Amazon DynamoDB on AWS Lambda for HTTP calls and data storage;

6. Use AWS Step Functions on AWS Lambda to implement state machines;

7. Use Amazon CloudWatch Logs on AWS Lambda to track logs.

   1.2 The content structure of the article

   This article is divided into seven parts:

8. Background introduction

9. Explanation of basic concepts and terms

10. Explanation of core algorithm principles, specific operating steps, and mathematical formulas

11. Specific code examples and explanations

12. Future development trends and challenges

13. Appendix: Frequently Asked Questions and Answers

The author of this article himself has some experience in cloud computing, microservices, Terraform and other fields, so the article will not belittle these technologies or concepts. Therefore, anyone interested in exploring these technologies and gaining a deeper understanding of them is welcome to read this article.

2. Explanation of basic concepts and terms

2.1 What is AWS Lambda?

AWS Lambda is a service that allows you to run small snippets of code or functions that only trigger execution in response to events. Once created, a Lambda function automatically scales on demand without the need to manage servers or reserve capacity. It supports multiple programming languages, including Node.js, Python, Java, C#, Go, and PowerShell, providing high availability and scalability. You can use the AWS Management Console or the command line interface (CLI) to create, debug, and manage Lambda functions.

2.2 What is Terraform?

Terraform is an open source tool from Hashicorp for creating and managing infrastructure as code. Use Terraform to automatically create, update, and delete resources such as virtual machines, networks, and databases in the cloud without writing code directly. You can specify the required resource configuration through declarative configuration files and let Terraform manage changes to the underlying infrastructure.

2.3 What is microservice architecture?

Microservices architecture is an application architecture pattern composed of multiple independent, cooperating services. Each service is responsible for a specific function or business area, and a lightweight communication mechanism is used to communicate between them. This architectural model can solve some complex application scenarios, such as elastic scaling, reliability, maintainability, scalability, etc.

2.4 Terraform template used in this article

The Terraform template used in this article comes from the official documentation: https://github.com/terraform-aws-modules/terraform-aws-lambda. This template implements the creation of a simple Lambda function in AWS and sets the default VPC, IAM, Lambda service role permissions policy.

2.5 Sample project used in this article

The example project used in this article is a simple and extremely boring Python script that takes two parameters (x and y) and returns the result of x + y.

2.6 Basic Usage of Terraform

To install Terraform, please refer to the official website https://www.terraform.io/downloads.html. Download the installation package corresponding to the system version and follow the prompts to install it. Terraform configuration files generally end with a tf file. It is assumed here that the reader is already familiar with the basic syntax and usage of Terraform. If you are not familiar with it, it is recommended to read the tutorial on the Terraform official website first. All Terraform commands in this article are completed in the Linux environment.

3. Explanation of core algorithm principles, specific operating steps and mathematical formulas

3.1 Create AWS Lambda function

The process for creating an AWS Lambda function using Terraform is as follows:

1. Install Terraform;
2. Set AWS Access Key;
3. Write Terraform configuration file and define Lambda function related information;
4. Execute Terraform init to create the remote host;
5. Execute Terraform apply to create the Lambda function.

Here are the detailed steps:

3.1.1 Install Terraform

This article uses Terraform v0.12.9. Download the installation package and unzip it to /usr/local/bin.

3.1.2 Setting AWS Access Key

After installing Terraform, you need to set up an AWS Access Key. Visit https://console.aws.amazon.com/iam/home#/security_credentials and find Create New Access Key under the Access keys (Access key ID and Secret access key) tab. Click Download.csv file to record the credentials and save the csv file.

Open a terminal and execute the following command to log in to AWS:

$ export AWS_ACCESS_KEY_ID=$(head -n1 ~/.aws/credentials | cut -d'=' -f2)
$ export AWS_SECRET_ACCESS_KEY=$(tail -n1 ~/.aws/credentials | cut -d'=' -f2)

3.1.3 Write Terraform configuration file

Write a Terraform configuration file named lambda.tf with the following content:

provider "aws" {
  region = "${var.region}"
}

variable "region" {
  default = "us-east-1"
}

resource "aws_lambda_function" "example" {
  function_name = "my-lambda-function"
  filename = "./main.zip"
  source_code_hash = filebase64sha256("./main.zip")

  role = aws_iam_role.iam_for_lambda.arn
  handler = "handler.main"
  runtime = "python3.7"
  vpc_config {
    subnet_ids = [aws_subnet.default.id]
    security_group_ids = [aws_security_group.allow_all.id]
  }

  depends_on = [aws_iam_policy_attachment.s3_access]
}

data "archive_file" "main" {
  output_path = "main.zip"
  type = "zip"

  source_dir = "."
}

resource "aws_iam_role" "iam_for_lambda" {
  name = "iam_for_lambda"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_policy_document" "s3_read_only_for_lambda" {
  statement {
    actions = ["s3:*"]

    resources = [
      "arn:${data.aws_partition.current.partition}:s3:::*",
      "arn:${data.aws_partition.current.partition}:s3:::*/*",
    ]

    principals {
      type = "*"
      identifiers = ["arn:aws:iam::${var.account_id}:root"]
    }

  }
}

resource "aws_iam_policy" "s3_read_only_for_lambda" {
  name = "s3_read_only_for_lambda"
  path = "/"
  policy = data.aws_iam_policy_document.s3_read_only_for_lambda.json
}

resource "aws_iam_policy_attachment" "s3_access" {
  role = aws_iam_role.iam_for_lambda.name
  policies = [aws_iam_policy.s3_read_only_for_lambda.arn]
}

data "aws_partition" "current" {}

resource "aws_security_group" "allow_all" {
  ingress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = { Name = "allow_all"}
}

resource "aws_subnet" "default" {
  availability_zone = "us-east-1a"
  vpc_id = aws_vpc.default.id

  cidr_block = "10.0.1.0/24"

  map_public_ip_on_launch = true
}

resource "aws_vpc" "default" {
  cidr_block = "10.0.0.0/16"
  enable_dns_hostnames = true
  tags = { Name = "default" }
}

Among them, except for a few variable values that need to be modified according to the actual situation, other parts do not need to be adjusted. Among them, aws_lambda_function defines information related to the Lambda function, filename specifies the file name of the local compressed package, source_code_hash uses sha256 to generate the hash value of the compressed package, role specifies the execution role of the Lambda function, handler specifies the entry point of the function, and runtime The runtime environment is specified, vpc_config specifies the VPC configuration information, and depends_on specifies the dependency relationship. data defines the data sources required by the Lambda function, such as compressed packages, S3 buckets, etc. aws_iam_role defines the IAM execution role, and assume_role_policy specifies the Lambda service role permission policy. aws_iam_policy_document defines the S3 read-only permissions policy document. aws_iam_policy uploads the S3 read-only permission policy to the IAM service, and aws_iam_policy_attachment binds the IAM execution role and the S3 read-only permission policy. aws_partition gets the current AWS partition. aws_security_group creates a default security group that allows access from all IP addresses, aws_subnet creates a default subnet, and aws_vpc creates a default VPC.

Note: Because Terraform requires permissions to create various resources, please ensure that the current user’s AccessKey and SecretKey have sufficient permissions for the corresponding resources before running.

3.1.4 Execute Terraform initialization

Execute the following command to initialize the Terraform environment:

$ cd terraform & amp; & amp; terraform init
Initializing modules...
- module.lambda_function

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws" (terraform-providers/aws) 2.44.0...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 2.44"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

3.1.5 Execute Terraform plan

Execute the following command to check whether the Lambda function configuration is correct:

“`bash $ cd terraform & & terraform plan Refreshing Terraform state in-memory prior to plan… The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage.

An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols:

• create

Terraform will perform the following actions:

aws_iam_policy.s3_read_only_for_lambda will be created

• resource “aws_iam_policy” “s3_read_only_for_lambda” {

  • arn = (known after apply)
  • description = “Managed by Terraform”
  • id = (known after apply)
  • name = “s3_read_only_for_lambda”
  • path = “/”
  • policy = jsonencode(

    {
       + Statement = [
           + {
               + Action = [
                   + "s3:GetObject",
                ]
               + Effect = "Allow"
               + Principal = {
                   + Type = "*"
                }
               + Resource = [
                   + "*/*",
                   + "arn:aws:s3:::*/*",
                   + "arn:aws:s3:::*",
                ]
            },
        ]
       + Version = "2012-10-17"
    }

    )

  • unique_id = (known after apply) }

  aws_iam_policy_attachment.s3_access will be created

• resource “aws_iam_policy_attachment” “s3_access” {

  • id = (known after apply)
  • policy_arn = “arn:aws:iam::123456789012:policy/s3_read_only_for_lambda”
  • roles = [
    • “iam_for_lambda”, ] }

  aws_lambda_function.example will be created

• resource “aws_lambda_function” “example” {

  • arn = (known after apply)

  • dead_letter_config = (known after apply)

  • environment = {}

  • ephemeral_storage = (known after apply)

  • file_system_configs = []

  • function_name = “my-lambda-function”

  • handler = “handler.main”

  • id = (known after apply)

  • image_uri = (known after apply)

  • invoke_arn = (known after apply)

  • kms_key_arn = (known after apply)

  • last_modified = (known after apply)

  • layers = []

  • memory_size = 128

  • package_type = “Zip”

  • publish = false

  • qualified_arn = (known after apply)

  • reserved_concurrent_executions = -1

  • role = “arn:aws:iam::123456789012:role/iam_for_lambda”

  • runtime = “python3.7”

  • s3_bucket = (known after apply)

  • s3_key = (known after apply)

  • signing_job_arn = (known after apply)

  • signing_profile_version_arn = (known after apply)

  • source_code_hash = “hkvNAjK0TIpHttAjX7NtwzllhsCXreQrFVRRO/ZCLcU=”

  • source_code_size = (known after apply)

  • timeout=3

  • tracing_config = (known after apply)

  • version = “$LATEST”

  • vpc_config {

    • security_group_ids = [
      • “sg-0abcc85b4ff2ae345”, ]
    • subnet_ids = [
      • “subnet-0bafefa05cd1828ad”, ] } }

  data.archive_file.main will be read during apply

  (config refers to values not yet known)

• data “archive_file” “main” {

  • output_path = “main.zip”
  • source_dir = “.”
  • type = “zip” }

  data.aws_partition.current will be read during apply

  (config refers to values not yet known)

• data “aws_partition” “current” {

  • dns_suffix = “amazonaws.com”
  • partition = “aws”
  • partition_dns_suffix = “amazonaws.com”
  • regions = tolist([
    • “af-south-1”,
    • “ap-east-1”,
    • “ap-northeast-1”,
    • “ap-northeast-2”,
    • “ap-northeast-3”,
    • “ap-south-1”,
    • “ap-southeast-1”,
    • “ap-southeast-2”,
    • “ca-central-1”,
    • “cn-northwest-1”,
    • “eu-central-1”,
    • “eu-north-1”,
    • “eu-south-1”,
    • “eu-west-1”,
    • “eu-west-2”,
    • “eu-west-3”,
    • “me-south-1”,
    • “sa-east-1”,
    • “us-east-1”,
    • “us-east-2”,
    • “us-gov-east-1”,
    • “us-gov-west-1”,
    • “us-west-1”,
    • “us-west-2”, ]) }

  aws_iam_role.iam_for_lambda will be created

• resource “aws_iam_role” “iam_for_lambda” {

  • arn = (known after apply)
  • assume_role_policy = jsonencode(

    {
       + Statement = [
           + {
               + Action = "sts:AssumeRole"
               + Effect = "Allow"
               + Principal = {
                   + Service = "lambda.amazonaws.com"
                }
               + Sid = ""
            },
        ]
       + Version = "2012-10-17"
    }

    )

  • create_date = (known after apply)
  • force_detach_policies = false
  • id = (known after apply)
  • max_session_duration = 3600
  • name = “iam_for_lambda”
  • path = “/”
  • unique_id = (known after apply) }

  aws_security_group.allow_all will be created

• resource “aws_security_group” “allow_all” {

  • arn = (known after apply)
  • description = “Managed by Terraform”
  • egress = [
    • {
      • cidr_blocks = [
        • “0.0.0.0/0”, ]
      • description = “”
      • from_port = 0
      • ipv6_cidr_blocks = []
      • prefix_list_ids = []
      • protocol = “-1”
      • security_groups = []
      • self = false
      • to_port = 0 }, ]
  • ingress = [
    • {
      • cidr_blocks = [
        • “0.0.0.0/0”, ]
      • description = “”
      • from_port = 0
      • ipv6_cidr_blocks = []
      • prefix_list_ids = []
      • protocol = “-1”
      • security_groups = []
      • self = false
      • to_port = 0 }, ]
  • name = “allow_all”
  • owner_id = (known after apply)
  • revoke_rules = false
  • tags = {
    • “Name” = “allow_all” }
  • vpc_id = “vpc-0c0e779c23c4c7d4c” }

  aws_subnet.default will be created

• resource “aws_subnet” “default” {

  • arn = (known after apply)
  • assign_ipv6_address_on_creation = false
  • availability_zone = “us-east-1a”
  • availability_zone_id = (known after apply)
  • cidr_block = “10.0.1.0/24”
  • enable_dns_hostnames = true
  • filter = [
    • {
      • name = “availability-zone”
      • values = [
        • “us-east-1a”, ] }, ]
  • id = (known after apply)
  • ipv6_cidr_block = (known after apply)
  • ipv6_native = false
  • map_public_ip_on_launch = true
  • owner_id = (known after apply)
  • private_dns_name_options = {
    • hostname_type = “ip-name” }
  • public_dns_name = (known after apply)
  • public_ip = (known after apply)
  • tags = {
    • “Name” = “default” }
  • vpc_id = “vpc-0c0e779c23c4c7d4c” }

  aws_vpc.default will be created

• resource “aws_vpc” “default” {

  • arn = (known after apply)
  • assign_generated_ipv6_cidr_block = false
  • cidr_block = “10.0.0.0/16”
  • default_network_acl_id = (known after apply)
  • default_route_table_id = (known after apply)
  • default_security_group_id = (known after apply)
  • dhcp_options_id = (known after apply)
  • enable_classiclink = (known after apply)
  • enable_classiclink_dns_support = (known after apply)
  • enable_dns_hostnames = true
  • id = (known after apply)
  • instance_tenancy = “default”
  • ipv6_association_id = (known after apply)
  • ipv6_cidr_block = (known after apply)
  • main_route_table_id = (known after apply)
  • owner_id = (known after apply)
  • tags = {
    • “Name” = “default” } }

Plan: 13 to add, 0 to change, 0 to destroy.

Note: You didn’t specify an “-out” parameter to save this plan, so Terraform can’t guarantee that exactly these actions will be performed if “terraform apply” is subsequently run.