AWS Velocity Series: EC2 based app CI/CD pipeline

In the previous article, you learned how to use CloudFormation to describe a production-ready infrastructure for an EC2 based app. In this article you will learn to:

Automate the creation of an AMI that contains the app with Packer
Deploy a CloudFormation stack based infrastructure/ec2.yml with AWS CodePipeline
Run the Acceptance tests on AWS CodeBuild against the infrastructure created in the previous step
Deploy another CloudFormation stack for the production environment

You can follow step by step or get the full source code here: https://github.com/widdix/aws-velocity

The pipeline is based on the CI/CD Pipeline as Code part of this series.

AWS Velocity Series

Most of our clients use AWS to reduce time-to-market following an agile approach. But AWS is only one part of the solution. In this article series, I show you how we help our clients to improve velocity: the time from idea to production. Discover all posts!

Additionally, the EC2 based app pipeline contains:

add a BuildAMI action to the Build stage.
add an Acceptance stage
add a Production stage

Copy the deploy/pipeline.yml file to deploy/pipeline_ec2.yml to get the starting point right. If you don’t have the deploy/pipeline.yml file you can download it from https://github.com/widdix/aws-velocity.

BuildAMI Action

EC2 instances start from an image (AMI) that contains the operating system including all the files that are needed. You can create your own AMI as well. Usually, you take one of the available AMIs like the Amazon Linux, make your modifications, and then create a new image from that. This whole procedure can also be automated with a tool called Packer. You will now see how you can run Packer in CodeBuild to create a new AMI that contains the app.

Packer itself needs configuration files. Create a file infrastructure/packer.json with the following content that build a new AMI based on ami-c51e3eb6 (Amazon Linux) and a Bash script that you will create later:

{
 "variables": {
 "ami_name": "{{env `CODEBUILD_BUILD_ID`}}"
 },
 "builders": [{
 "type": "amazon-ebs",
 "region": "eu-west-1",
 "source_ami": "ami-c51e3eb6",
 "instance_type": "t2.micro",
 "ssh_username": "ec2-user",
 "ami_name": "{{user `ami_name` | clean_ami_name}}",
 "ami_regions": ["REGION"]
 }],
 "provisioners": [{
 "type": "file",
 "source": "app",
 "destination": "/tmp"
 }, {
 "type": "shell",
 "script": "infrastructure/packer.sh"
 }]
}

Packer is configured to run a Bash script to provision the AMI and to upload the app folder. Create a file infrastructure/packer.sh with the following content to:

install the latest patches
install Node.js 6.x
install the CloudWatch Logs agent
install forever, a tool to run Node.js script in the background

infrastructure/packer.shGitHub

#!/bin/bash -ex
# this script runs as ec2-user

sudo yum -y update
curl --silent --location https://rpm.nodesource.com/setup_6.x | sudo bash -
sudo yum -y install nodejs awslogs
sudo npm install -g forever@0.15.3
sudo mv /tmp/app /opt

The script also moves the application files to the right place.

To integrate Packer into the pipeline, add the following resources to the Resources section of deploy/pipeline_ec2.yml to create a CodeBuild project to run Packer with the above configuration. Packer also needs a bunch of IAM permissions which are also added.

deploy/pipeline_ec2.ymlGitHub

# Packer needs a set of access rights as defined in https://www.packer.io/docs/builders/amazon.html
AMICodeBuildRole:
 DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
 Type: 'AWS::IAM::Role'
 Properties:
 AssumeRolePolicyDocument:
 Version: '2012-10-17'
 Statement:
 - Effect: Allow
 Principal:
 Service:
 - 'codebuild.amazonaws.com'
 Action:
 - 'sts:AssumeRole'
 Policies:
 - PolicyName: ServiceRole
 PolicyDocument:
 Version: '2012-10-17'
 Statement:
 - Sid: CloudWatchLogsPolicy
 Effect: Allow
 Action: 
 - 'logs:CreateLogGroup'
 - 'logs:CreateLogStream'
 - 'logs:PutLogEvents'
 Resource: '*'
 - Sid: CodeCommitPolicy
 Effect: Allow
 Action: 'codecommit:GitPull'
 Resource: '*'
 - Sid: S3GetObjectPolicy
 Effect: Allow
 Action: 
 - 's3:GetObject'
 - 's3:GetObjectVersion'
 Resource: '*'
 - Sid: S3PutObjectPolicy
 Effect: 'Allow'
 Action: 's3:PutObject'
 Resource: '*'
 - PolicyName: EC2
 PolicyDocument:
 Version: '2012-10-17'
 Statement:
 - Sid: CloudFormation
 Effect: Allow
 Action: 
 - 'ec2:AttachVolume'
 - 'ec2:AuthorizeSecurityGroupIngress'
 - 'ec2:CopyImage'
 - 'ec2:CreateImage'
 - 'ec2:CreateKeypair'
 - 'ec2:CreateSecurityGroup'
 - 'ec2:CreateSnapshot'
 - 'ec2:CreateTags'
 - 'ec2:CreateVolume'
 - 'ec2:DeleteKeypair'
 - 'ec2:DeleteSecurityGroup'
 - 'ec2:DeleteSnapshot'
 - 'ec2:DeleteVolume'
 - 'ec2:DeregisterImage'
 - 'ec2:DescribeImageAttribute'
 - 'ec2:DescribeImages'
 - 'ec2:DescribeInstances'
 - 'ec2:DescribeRegions'
 - 'ec2:DescribeSecurityGroups'
 - 'ec2:DescribeSnapshots'
 - 'ec2:DescribeSubnets'
 - 'ec2:DescribeTags'
 - 'ec2:DescribeVolumes'
 - 'ec2:DetachVolume'
 - 'ec2:GetPasswordData'
 - 'ec2:ModifyImageAttribute'
 - 'ec2:ModifyInstanceAttribute'
 - 'ec2:ModifySnapshotAttribute'
 - 'ec2:RegisterImage'
 - 'ec2:RunInstances'
 - 'ec2:StopInstances'
 - 'ec2:TerminateInstances'
 Resource: '*'
# This IAM User is only temporarily necessary until https://github.com/mitchellh/packer/pull/4613 is fixed!
AMICodeBuildUser:
 DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
 Type: 'AWS::IAM::User'
 Properties:
 Policies:
 - PolicyName: Packer
 PolicyDocument:
 Version: '2012-10-17'
 Statement:
 - Sid: EC2
 Effect: Allow
 Action:
 - 'ec2:AttachVolume'
 - 'ec2:AuthorizeSecurityGroupIngress'
 - 'ec2:CopyImage'
 - 'ec2:CreateImage'
 - 'ec2:CreateKeypair'
 - 'ec2:CreateSecurityGroup'
 - 'ec2:CreateSnapshot'
 - 'ec2:CreateTags'
 - 'ec2:CreateVolume'
 - 'ec2:DeleteKeypair'
 - 'ec2:DeleteSecurityGroup'
 - 'ec2:DeleteSnapshot'
 - 'ec2:DeleteVolume'
 - 'ec2:DeregisterImage'
 - 'ec2:DescribeImageAttribute'
 - 'ec2:DescribeImages'
 - 'ec2:DescribeInstances'
 - 'ec2:DescribeRegions'
 - 'ec2:DescribeSecurityGroups'
 - 'ec2:DescribeSnapshots'
 - 'ec2:DescribeSubnets'
 - 'ec2:DescribeTags'
 - 'ec2:DescribeVolumes'
 - 'ec2:DetachVolume'
 - 'ec2:GetPasswordData'
 - 'ec2:ModifyImageAttribute'
 - 'ec2:ModifyInstanceAttribute'
 - 'ec2:ModifySnapshotAttribute'
 - 'ec2:RegisterImage'
 - 'ec2:RunInstances'
 - 'ec2:StopInstances'
 - 'ec2:TerminateInstances'
 Resource: '*'
# This IAM Access Key is only temporarily necessary until https://github.com/mitchellh/packer/pull/4613 is fixed!
AMICodeBuildUserAccessKey:
 DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
 Type: 'AWS::IAM::AccessKey'
 Properties:
 UserName: !Ref AMICodeBuildUser
AMIProject:
 DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
 Type: 'AWS::CodeBuild::Project'
 Properties:
 Artifacts:
 Type: CODEPIPELINE
 Environment:
 ComputeType: 'BUILD_GENERAL1_SMALL'
 Image: 'amazonlinux:2016.09'
 Type: 'LINUX_CONTAINER'
 EnvironmentVariables: # pass in the AWS credentials as environment variables is only temporarily necessary until https://github.com/mitchellh/packer/pull/4613 is fixed!
 - Name: 'AWS_ACCESS_KEY_ID'
 Value: !Ref AMICodeBuildUserAccessKey
 - Name: 'AWS_SECRET_ACCESS_KEY'
 Value: !GetAtt 'AMICodeBuildUserAccessKey.SecretAccessKey'
 Name: !Sub '${AWS::StackName}-ami'
 ServiceRole: !GetAtt 'AMICodeBuildRole.Arn'
 Source:
 Type: CODEPIPELINE
 BuildSpec: !Sub |
 version: 0.1
 phases:
 install: # install Packer
 commands:
 - 'yum -y install unzip'
 - 'curl -s -m 60 -o /opt/packer.zip https://releases.hashicorp.com/packer/0.12.3/packer_0.12.3_linux_amd64.zip'
 - 'unzip /opt/packer.zip -d /opt'
 pre_build: # replace the REGION placeholder with the stack's region
 commands:
 - 'sed -i "s:REGION:${AWS::Region}:g" infrastructure/packer.json'
 build:
 commands:
 # run packer
 - '/opt/packer -machine-readable build infrastructure/packer.json | tee infrastructure/packer.txt'
 # extract the AMI id into a JSON file
 - 'echo "{" > infrastructure/ami.json'
 - 'cat infrastructure/packer.txt | grep '',amazon-ebs,artifact,0,id,'' | awk -F'','' ''{print $6}'' | sed ''s/%!(PACKER_COMMA)/\''$''\n/g'' | awk -F'':'' ''{print "\"image\": \""$2"\","}'' | sed ''$ s/.$//'' >> infrastructure/ami.json'
 - 'echo "}" >> infrastructure/ami.json'
 artifacts:
 files:
 - 'infrastructure/packer.txt'
 - 'infrastructure/ami.json'
 TimeoutInMinutes: 10

You also need to make a small modification to the exiting BuildSpec in the AppProject resource to include the packer files into the App artifact. This is a hack because CodeBuild does not support multiple input artifacts at the moment. Change the artifacts section to:

deploy/pipeline_ec2.ymlGitHub

artifacts:
 files:
 - 'app/**/*'
 - 'infrastructure/packer.json' # this is a hack because we can not pass multiple Arifacts as an input to CodeBuild at the moment
 - 'infrastructure/packer.sh' # this is a hack because we can not pass multiple Arifacts as an input to CodeBuild at the moment

Now the CodeBuild project needs to be called in the pipeline, therefore change the Pipeline resource in the file deploy/pipeline_ec2.yml and add a new build action:

deploy/pipeline_ec2.ymlGitHub

Pipeline:
 Type: 'AWS::CodePipeline::Pipeline'
 Properties:
 # [...]
 - Name: BuildAndTestAcceptance
 ActionTypeId:
 Category: Build
 Owner: AWS
 Provider: CodeBuild
 Version: 1
 Configuration:
 ProjectName: !Ref AcceptanceProject
 InputArtifacts:
 - Name: Source
 OutputArtifacts:
 - Name: Acceptance
 RunOrder: 1
 # NEW STUFF!
 - Name: BuildAMI
 ActionTypeId:
 Category: Build
 Owner: AWS
 Provider: CodeBuild
 Version: 1
 Configuration:
 ProjectName: !Ref AMIProject
 InputArtifacts:
 - Name: App
 OutputArtifacts:
 - Name: AMI
 RunOrder: 2

Now, a new AMI is automatically created whenever the pipeline runs. The AMI will include the app and the latest patches.

It’s time to deploy the app to the acceptance stage and too see if the app works.

Acceptance stage

The acceptance stage consists of a CloudFormation stack based on infrastructure/ec2.yml and the execution of the acceptance tests. To create the CloudFormation stack, you first have to provide a few parameters. Create a file infrastructure/ec2.json with the following content:

infrastructure/ec2.jsonGitHub

{
 "Parameters": {
 "ImageId": {"Fn::GetParam": ["AMI", "infrastructure/ami.json", "image"]},
 "ParentVPCStack": "vpc-2azs",
 "ParentSSHBastionStack": "",
 "KeyName": "",
 "AdminEmail": "your@email.com"
 }
}

If you don’t have a VPC stack based of our Free Templates for AWS CloudFormation (https://github.com/widdix/aws-cf-templates/tree/master/vpc) create a VPC stack first. Make sure to change the ParentVPCStack parameter in the infrastructure/ec2.yml accordingly. Also change the value of the AdminEmail parameter. The other values can be stay as they are. Look at the ImageId parameter value. This is the way of getting a value out of a JSON artifact file in CoePipeline.

To run the acceptance tests, you also need another CodeBuild project, add the following resources to the Resources section of deploy/pipeline_ec2.yml:

deploy/pipeline_ec2.ymlGitHub

ExtendedCodeBuildRole:
 DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
 Type: 'AWS::IAM::Role'
 Properties:
 AssumeRolePolicyDocument:
 Version: '2012-10-17'
 Statement:
 - Effect: Allow
 Principal:
 Service:
 - 'codebuild.amazonaws.com'
 Action:
 - 'sts:AssumeRole'
 Policies:
 - PolicyName: ServiceRole
 PolicyDocument:
 Version: '2012-10-17'
 Statement:
 - Sid: CloudWatchLogsPolicy
 Effect: Allow
 Action: 
 - 'logs:CreateLogGroup'
 - 'logs:CreateLogStream'
 - 'logs:PutLogEvents'
 Resource: '*'
 - Sid: CodeCommitPolicy
 Effect: Allow
 Action: 'codecommit:GitPull'
 Resource: '*'
 - Sid: S3GetObjectPolicy
 Effect: Allow
 Action: 
 - 's3:GetObject'
 - 's3:GetObjectVersion'
 Resource: '*'
 - Sid: S3PutObjectPolicy
 Effect: 'Allow'
 Action: 's3:PutObject'
 Resource: '*'
 - PolicyName: CloudFormation
 PolicyDocument:
 Version: '2012-10-17'
 Statement:
 - Sid: CloudFormation
 Effect: Allow
 Action: 
 - 'cloudformation:DescribeStacks'
 Resource: !Sub 'arn:aws:cloudformation:${AWS::Region}:${AWS::AccountId}:stack/${AWS::StackName}-acceptance/*'
RunAcceptanceProject:
 DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
 Type: 'AWS::CodeBuild::Project'
 Properties:
 Artifacts:
 Type: CODEPIPELINE
 Environment:
 ComputeType: 'BUILD_GENERAL1_SMALL'
 Image: 'aws/codebuild/nodejs:6.3.1'
 Type: 'LINUX_CONTAINER'
 Name: !Sub '${AWS::StackName}-run-acceptance'
 ServiceRole: !GetAtt 'ExtendedCodeBuildRole.Arn'
 Source:
 Type: CODEPIPELINE
 BuildSpec: !Sub |
 version: 0.1
 phases:
 build: # execute acceptance tests against the acceptance stack
 commands:
 - 'cd acceptance/ && ENDPOINT=`aws cloudformation describe-stacks --stack-name ${AWS::StackName}-acceptance --query "Stacks[0].Outputs[?OutputKey==''URL''].OutputValue" --output text` ./node_modules/jasmine-node/bin/jasmine-node .'
 TimeoutInMinutes: 10

Now the CodeBuild project needs to be called in the pipeline, therefore change Pipeline resource in the file deploy/pipeline_ec2.yml to:

deploy the CloudFormation stack suffixed with -acceptance
run the acceptance tests

deploy/pipeline_ec2.ymlGitHub

Pipeline:
 Type: 'AWS::CodePipeline::Pipeline'
 Properties:
 # [...]
 - Name: BuildAMI
 ActionTypeId:
 Category: Build
 Owner: AWS
 Provider: CodeBuild
 Version: 1
 Configuration:
 ProjectName: !Ref AMIProject
 InputArtifacts:
 - Name: App
 OutputArtifacts:
 - Name: AMI
 RunOrder: 2
 # NEW STUFF!
 - Name: Acceptance
 Actions:
 - Name: Deploy
 ActionTypeId:
 Category: Deploy
 Owner: AWS
 Provider: CloudFormation
 Version: 1
 Configuration:
 ActionMode: CREATE_UPDATE
 Capabilities: CAPABILITY_IAM
 RoleArn: !GetAtt 'CloudFormationRole.Arn'
 StackName: !Sub '${AWS::StackName}-acceptance'
 TemplatePath: 'Source::infrastructure/ec2.yml'
 TemplateConfiguration: 'Source::infrastructure/ec2.json'
 InputArtifacts:
 - Name: Source
 - Name: AMI
 RunOrder: 1
 - Name: Test
 ActionTypeId:
 Category: Build
 Owner: AWS
 Provider: CodeBuild
 Version: 1
 Configuration:
 ProjectName: !Ref RunAcceptanceProject
 InputArtifacts:
 - Name: Acceptance
 RunOrder: 2

The acceptance stage is now ready.

Production stage

The production stage is pretty simple, just one CloudFormation stack. Change the Pipeline resource in the file deploy/pipeline_ec2.yml to add a new stage that looks familiar to the acceptance stage:

deploy/pipeline_ec2.ymlGitHub

Pipeline:
 Type: 'AWS::CodePipeline::Pipeline'
 Properties:
 # [...]
 - Name: Test
 ActionTypeId:
 Category: Build
 Owner: AWS
 Provider: CodeBuild
 Version: 1
 Configuration:
 ProjectName: !Ref RunAcceptanceProject
 InputArtifacts:
 - Name: Acceptance
 RunOrder: 2
 # NEW STUFF! 
 - Name: Production
 Actions:
 - Name: Deploy
 ActionTypeId:
 Category: Deploy
 Owner: AWS
 Provider: CloudFormation
 Version: 1
 Configuration:
 ActionMode: CREATE_UPDATE
 Capabilities: CAPABILITY_IAM
 RoleArn: !GetAtt 'CloudFormationRole.Arn'
 StackName: !Sub '${AWS::StackName}-production'
 TemplatePath: 'Source::infrastructure/ec2.yml'
 TemplateConfiguration: 'Source::infrastructure/ec2.json'
 InputArtifacts:
 - Name: Source
 - Name: AMI
 RunOrder: 1

Now the AMI containing the application is deployed to production with confidence and without disturbing the users. Try it and run the pipeline!

Summary

Let’s use my production-ready definition to summarize how each point is implemented:

Highly available: The load balancer (which is HA) sits in front of a fleet of EC2 instances managed by the Auto Scaling Group for maximum availability. In case of an unhealthy instance, the Auto Scaling Group will replace that instance.
Scalable: If the CPU utilization gets over 70%, a Cloud Watch Alarm triggers a Scaling Policy to add new instances automatically.
Frictionless deployment: To deploy a new version of the app, a new AMI is created. This AMI is then rolled out to the acceptance environment by updating the CloudFormation stack with the new ImageId parameter. CloudFormation and the Auto Scaling Group perform a rolling update to avoid the application being down during deployment. If the application can not be started the Rolling Update fails and CloudFormation rolls back.
Secure: During AMI creation, the latest patches are applied. You must ensure that the pipeline runs often enough to keep up with new patches. Besides that, Security Groups control network traffic to the EC2 instances. When following the bastion host approach, you get maximum security. The EC2 instance is only allowed to send logs to CloudWatch Logs by following the least privileges approach.
Operations: All logs are stored in CloudWatch Logs, important metrics are monitored and alarms are defined.

If you now have the impression that running an app on EC2 is a lot of work you are right. In the next two articles, you will learn about other options with fewer responsibilities.

Series

AWS Velocity Cover

Set the assembly line up
Local development environment
CI/CD Pipeline as Code
Running your application
EC2 based app
a. Infrastructure
b. CI/CD Pipeline (you are here)
Containerized ECS based app
a. Infrastructure
b. CI/CD Pipeline
Serverless app
Summary

You can find the source code on GitHub.

AWS Velocity Series: EC2 based app CI/CD pipeline

AWS Velocity Series

BuildAMI Action

Acceptance stage

Production stage

Summary

Series

Leave a Reply Cancel reply

Categories

Subscribe Our Newsletter