Cloudibn News

Be updated with Technology

  • BIG Data & Analytics
  • CLOUD
  • Data Center
  • IOT
  • Machine Learning & AI
  • SECURITY
  • Blockchain
  • Virtualization
You are here: Home / CLOUD / AWS Velocity Series: Containerized ECS based app infrastructure

AWS Velocity Series: Containerized ECS based app infrastructure

March 27, 2017 by cbn Leave a Comment

EC2 Container Service (ECS) is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon EC2 instances. To run an application on ECS, you need the following components:

  • Docker image
  • ECS cluster: EC2 instances running Docker and the ECS agent
  • ECS service: Managed Docker containers on the ECS cluster

ECS provides the ability to run a production-ready application on EC2 with reduced responsibilities and increased deployment speed compared to the EC2 based approach. By production-ready, I mean:

  • Highly available: no single point of failure
  • Scalable: increase or decrease the number of instances/containers based on load
  • Frictionless deployment: deliver new versions of your application automatically without downtime
  • Secure: patching operating systems and libraries frequently, follow the least privilege principle in all areas
  • Operations: provide tools like logging, monitoring and alerting to recognize and debug problems

The overall architecture will consist of a load balancer, forwarding requests to containers running on multiple EC2 instances, distributed among different availability zones (data centers).

ECS based app architecture

The diagram was created with Cloudcraft – Visualize your cloud architecture like a pro.

AWS Velocity Cover

AWS Velocity Series

Most of our clients use AWS to reduce time-to-market following an agile approach. But AWS is only one part of the solution. In this article series, I show you how we help our clients to improve velocity: the time from idea to production. Discover all posts!

Let’s start with the needed infrastructure. The ECS cluster and the service.

ECS cluster

The ECS cluster is a fleet of EC2 instances with the ECS agent and Docker installed. The ECS cluster is responsible for scheduling the work (containers) to the EC2 instances.

ECS cluster architecture

The diagram was created with Cloudcraft – Visualize your cloud architecture like a pro.

Your Docker containers will run on those EC2 instances. You don’t need to care about the ECS cluster too much. We provide you a free and production-ready CloudFormation template. Please setup the ECS cluster now if you want to setup the scenario. AWS charges will likely occur! The ECS cluster need to run in a VPC, so if you don’t have a VPC stack based of our Free Templates for AWS CloudFormation (https://github.com/widdix/aws-cf-templates/tree/master/vpc) create a VPC stack first.

This first step was easy. Now you learn to use the ECS cluster with ECS services.

ECS service

The ECS service is responsible for launching Docker containers in the cluster. The service also makes sure that failed containers are replaced, and it also takes care about performing rolling updates for you. You also need a load balancer to route traffic to the containers. The following ECS service template is based on free and production-ready CloudFormation template.

Load balancer

You can follow step by step or get the full source code here: https://github.com/widdix/aws-velocity

Create a file infrastructure/ecs.yml. The first part of the file contains the load balancer. To fully describe an Application Load Balancer, you need:

  • A Security Group that allows traffic on port 80
  • The lApplication Load Balancer itself
  • A Target Group, which is a group of EC2 instances. Each of them runs containers that can receive traffic from the load balancer
  • A Listener, which wires the load balancer together with the target group and defines the listening port

Watch out for comments with more detailed information in the code.

infrastructure/ecs.ymlGitHub
---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'ECS: service that runs on an ECS cluster based on ecs/cluster.yaml and uses a dedicated ALB, a cloudonaut.io template'
Parameters:
# You can reuse a VPC for multiple applications. In this case, we use one of our Free Templates for AWS CloudFormation (https://github.com/widdix/aws-cf-templates/tree/master/vpc).
ParentVPCStack:
Description: 'Stack name of parent VPC stack based on vpc/vpc-*azs.yaml template.'
Type: String
ParentClusterStack:
Description: 'Stack name of parent Cluster stack based on ecs/cluster.yaml template.'
Type: String
Resources:
# Allow traffic from the load balancer to the EC2 instances in the cluster. This is only necessary because we reuse the cluster template in the way it is!
SecurityGroupInALB:
Type: 'AWS::EC2::SecurityGroupIngress'
Properties:
GroupId:
'Fn::ImportValue': !Sub '${ParentClusterStack}-SecurityGroup'
IpProtocol: tcp
FromPort: 0
ToPort: 65535
SourceSecurityGroupId: !Ref ALBSecurityGroup
# The load balancer accepts HTTP traffic. Therefore the firewall must allow incoming traffic on port 80.
ALBSecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupDescription: 'ecs-cluster-alb'
VpcId:
'Fn::ImportValue': !Sub '${ParentVPCStack}-VPC'
SecurityGroupIngress:
- CidrIp: '0.0.0.0/0'
FromPort: 80
ToPort: 80
IpProtocol: tcp
# The load balancer needs to run in public subnets because our users should be able to access the app from the Internet.
LoadBalancer:
Type: 'AWS::ElasticLoadBalancingV2::LoadBalancer'
Properties:
Scheme: 'internet-facing'
SecurityGroups:
- !Ref ALBSecurityGroup
Subnets:
- 'Fn::ImportValue': !Sub '${ParentVPCStack}-SubnetAPublic'
- 'Fn::ImportValue': !Sub '${ParentVPCStack}-SubnetBPublic'
# A target group groups a bunch of backend instances that receive traffic from the load balancer. the health check ensures that only working backends are used.
DefaultTargetGroup:
Type: 'AWS::ElasticLoadBalancingV2::TargetGroup'
Properties:
HealthCheckIntervalSeconds: 15
HealthCheckPath: '/1'
HealthCheckProtocol: HTTP
HealthCheckTimeoutSeconds: 10
HealthyThresholdCount: 2
Matcher:
HttpCode: '200'
Port: 80
Protocol: HTTP
UnhealthyThresholdCount: 4
VpcId:
'Fn::ImportValue': !Sub '${ParentVPCStack}-VPC'
# The load balancer should listen on port 80 for HTTP traffic
HttpListener:
Type: 'AWS::ElasticLoadBalancingV2::Listener'
Properties:
DefaultActions:
- TargetGroupArn: !Ref DefaultTargetGroup
Type: forward
LoadBalancerArn: !Ref LoadBalancer
Port: 80
Protocol: HTTP
# A CloudFormation stack can return information that is needed by other stacks or scripts.
Outputs:
DNSName:
Description: 'The DNS name for the ECS cluster/service load balancer.'
Value: !GetAtt 'LoadBalancer.DNSName'
Export:
Name: !Sub '${AWS::StackName}-DNSName'
URL:
Description: 'URL to the ECS service.'
Value: !Sub 'http://${LoadBalancer.DNSName}'
Export:
Name: !Sub '${AWS::StackName}-URL'

But how do you get notified if something goes wrong? Let’s add a parameter to the Parameters section to make the receiver configurable:

infrastructure/ecs.ymlGitHub
AdminEmail:
Description: 'The email address of the admin who receives alerts.'
Type: String

Alerts are triggered by a CloudWatch Alarm which can send an alert to an SNS topic. You can subscribe to this topic via an email address to receive the alerts. Let’s create an SNS topic and two alarms in the Resources section:

infrastructure/ecs.ymlGitHub
# A SNS topic is used to send alerts via Email to the value of the AdminEmail parameter 
Alerts:
Type: 'AWS::SNS::Topic'
Properties:
Subscription:
- Endpoint: !Ref AdminEmail
Protocol: email
# This alarm is triggered, if the load balancer responds with 5XX status codes
LoadBalancer5XXAlarm:
Type: 'AWS::CloudWatch::Alarm'
Properties:
EvaluationPeriods: 1
Statistic: Sum
Threshold: 0
AlarmDescription: 'Load balancer responds with 5XX status codes.'
Period: 60
AlarmActions:
- !Ref Alerts
Namespace: 'AWS/ApplicationELB'
Dimensions:
- Name: LoadBalancer
Value: !GetAtt 'LoadBalancer.LoadBalancerFullName'
ComparisonOperator: GreaterThanThreshold
MetricName: HTTPCode_ELB_5XX_Count
# This alarm is triggered, if the backend responds with 5XX status codes
LoadBalancerTargetGroup5XXAlarm:
Type: 'AWS::CloudWatch::Alarm'
Properties:
EvaluationPeriods: 1
Statistic: Sum
Threshold: 0
AlarmDescription: 'Load balancer target responds with 5XX status codes.'
Period: 60
AlarmActions:
- !Ref Alerts
Namespace: 'AWS/ApplicationELB'
Dimensions:
- Name: LoadBalancer
Value: !GetAtt 'LoadBalancer.LoadBalancerFullName'
ComparisonOperator: GreaterThanThreshold
MetricName: HTTPCode_Target_5XX_Count

Let’s recap what you implemented: A load balancer with a firewall rule that allows traffic on port 80. In the case of 5XX status codes, you will receive an Email. But the load balancer alone is not enough. Now it’s time to add the ECS service.

ECS service

I already talked about the ECS service. It will take care of your containers. To be more precise, it will take care of your tasks that run in the ECS cluster. One task can contain one or multiple Docker containers. Three resources are needed:

  • A Task Definition that describes the Docker containers (similar to a Docker Compose file, or a Kubernetes Deployment)
  • An IAM Role for your container, so you don’t need to pass in static credentials when you want to interact with AWS from within your container. If you don’t want to make AWS API calls from your container the role is not needed.
  • The ECS Service itself.

To make things parameterizable, you also need to add a few parameters to the Parameters section:

infrastructure/ecs.ymlGitHub
# Where does this Docker image comes from? It will be created in the pipeline!
Image:
Description: 'The image to use for a container, which is passed directly to the Docker daemon. You can use images in the Docker Hub registry or specify other repositories (repository-url/image:tag).'
Type: String
DesiredCount:
Description: 'The number of simultaneous tasks, which you specify by using the TaskDefinition property, that you want to run on the cluster.'
Type: Number
Default: 2
ConstraintDescription: 'Must be >= 1'
MinValue: 1
MaxCapacity:
Description: 'The maximum number of simultaneous tasks, that you want to run on the cluster.'
Type: Number
Default: 4
ConstraintDescription: 'Must be >= 1'
MinValue: 1
MinCapacity:
Description: 'The minimum number of simultaneous tasks, that you want to run on the cluster.'
Type: Number
Default: 2
ConstraintDescription: 'Must be >= 1'
MinValue: 1

Now you can describe the resources in the CloudFormation template.

infrastructure/ecs.ymlGitHub
TaskDefinition:
Type: 'AWS::ECS::TaskDefinition'
Properties:
Family: !Ref 'AWS::StackName'
NetworkMode: bridge
ContainerDefinitions:
- Name: main
Image: !Ref Image # This is where the Docker image is configured
Memory: 128
PortMappings:
- ContainerPort: 3000 # The image exposes the app on port 3000
Protocol: tcp
Essential: true
LogConfiguration:
LogDriver: awslogs
Options:
'awslogs-region': !Ref 'AWS::Region'
'awslogs-group':
'Fn::ImportValue': !Sub '${ParentClusterStack}-LogGroup'
'awslogs-stream-prefix': !Ref 'AWS::StackName'
# The role is using the managed policy AmazonEC2ContainerServiceRole
ServiceRole:
Type: 'AWS::IAM::Role'
Properties:
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole'
AssumeRolePolicyDocument:
Version: '2008-10-17'
Statement:
- Effect: Allow
Principal:
Service: 'ecs.amazonaws.com'
Action: 'sts:AssumeRole'
Service:
Type: 'AWS::ECS::Service'
DependsOn: HttpListener
Properties:
Cluster:
'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
DeploymentConfiguration: # This is the configuration for the rolling update
MaximumPercent: 200
MinimumHealthyPercent: 50
DesiredCount: !Ref DesiredCount
LoadBalancers:
- ContainerName: main
ContainerPort: 3000 # The image exposes the app on port 3000
TargetGroupArn: !Ref DefaultTargetGroup
Role: !GetAtt 'ServiceRole.Arn'
TaskDefinition: !Ref TaskDefinition

Let’s recap what you implemented: A Task definition to define the containers that are managed by the service, an IAM role that is accessible inside the containers, and the ECS service that used the task definition to launch tasks (a bunch of containers) in the cluster. Logs from the containers are already shipped to CLoudWatch Logs by the awslogs log driver and are visible in the Log Group that is part of the ECS cluster template.

Two things are missing:

  1. Scalability of containers
  2. Alerting if containers have issues

Let’s tackle those issues step by step.

Auto scaling of containers

If you auto scale the number of containers, your ECS cluster must be able to auto scale as well. if you use our free template on GitHub, the cluster will auto scale as well

Auto Scaling works similar compared to the EC2 based approach. To scale based on the load you need to add:

  • Scaling Policies to define what should happen if the system should scale up/down
  • CloudWatch Alarms to trigger a Scaling Policy based on a metric such as CPU utilization
  • Additionally, you need a so-called Scalable Target. The Scalable Target can be an ECS service, but it could also be a Spot Fleet or an EMR Instance Group.

Again, you have to add those resources to the Resources section of your template:

infrastructure/ecs.ymlGitHub
ScalableTargetRole: # based on http://docs.aws.amazon.com/AmazonECS/latest/developerguide/autoscale_IAM_role.html
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: 'application-autoscaling.amazonaws.com'
Action: 'sts:AssumeRole'
Path: '/'
Policies:
- PolicyName: ecs
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- 'ecs:DescribeServices'
- 'ecs:UpdateService'
Resource: '*'
- PolicyName: cloudwatch
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- 'cloudwatch:DescribeAlarms'
Resource: '*'
ScalableTarget:
Type: 'AWS::ApplicationAutoScaling::ScalableTarget'
Properties:
MaxCapacity: !Ref MaxCapacity
MinCapacity: !Ref MinCapacity
ResourceId: !Sub
- 'service/${Cluster}/${Service}'
- Cluster:
'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
Service: !GetAtt 'Service.Name'
RoleARN: !GetAtt 'ScalableTargetRole.Arn'
ScalableDimension: 'ecs:service:DesiredCount'
ServiceNamespace: ecs
ScaleUpPolicy:
Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
Properties:
PolicyName: !Sub '${AWS::StackName}-scale-up'
PolicyType: StepScaling
ScalingTargetId: !Ref ScalableTarget
StepScalingPolicyConfiguration:
AdjustmentType: PercentChangeInCapacity
Cooldown: 300
MinAdjustmentMagnitude: 1
StepAdjustments:
- MetricIntervalLowerBound: 0
ScalingAdjustment: 25
ScaleDownPolicy:
Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
Properties:
PolicyName: !Sub '${AWS::StackName}-scale-down'
PolicyType: StepScaling
ScalingTargetId: !Ref ScalableTarget
StepScalingPolicyConfiguration:
AdjustmentType: PercentChangeInCapacity
Cooldown: 300
MinAdjustmentMagnitude: 1
StepAdjustments:
- MetricIntervalLowerBound: 0
ScalingAdjustment: -25
CPUUtilizationHighAlarm:
Type: 'AWS::CloudWatch::Alarm'
Properties:
AlarmDescription: 'Service is running out of CPU'
Namespace: 'AWS/ECS'
Dimensions:
- Name: ClusterName
Value:
'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
- Name: ServiceName
Value: !GetAtt 'Service.Name'
MetricName: CPUUtilization
ComparisonOperator: GreaterThanThreshold
Statistic: Average
Period: 60
EvaluationPeriods: 1
Threshold: 70
AlarmActions:
- !Ref ScaleUpPolicy
CPUUtilizationLowAlarm:
Type: 'AWS::CloudWatch::Alarm'
Properties:
AlarmDescription: 'Service is wasting CPU'
Namespace: 'AWS/ECS'
Dimensions:
- Name: ClusterName
Value:
'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
- Name: ServiceName
Value: !GetAtt 'Service.Name'
MetricName: CPUUtilization
ComparisonOperator: LessThanThreshold
Statistic: Average
Period: 60
EvaluationPeriods: 1
Threshold: 30
AlarmActions:
- !Ref ScaleDownPolicy

The number of tasks is now increased if the CPU utilization of the service goes above 70%, while the number of tasks is decreased if the CPU utilization falls below 30%.

Monitoring

Last but not least, you have to add CloudWatch Alarms to get alerted if something is wrong with your service. In the Resources section:

infrastructure/ecs.ymlGitHub
# Sends an alert if the average CPU load of the past 5 minutes is higher than 85%
CPUTooHighAlarm:
Type: 'AWS::CloudWatch::Alarm'
Properties:
AlarmDescription: 'Service is running out of CPU'
Namespace: 'AWS/ECS'
Dimensions:
- Name: ClusterName
Value:
'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
- Name: ServiceName
Value: !GetAtt 'Service.Name'
MetricName: CPUUtilization
ComparisonOperator: GreaterThanThreshold
Statistic: Average
Period: 300
EvaluationPeriods: 1
Threshold: 85
AlarmActions:
- !Ref Alerts

The infrastructure is ready now. Read the next part of the series to learn how to setup the CI/CD pipeline to deploy the ECS based app.

Series

AWS Velocity Cover

  1. Set the assembly line up
  2. Local development environment
  3. CI/CD Pipeline as Code
  4. Running your application
  5. EC2 based app
    a. Infrastructure
    b. CI/CD Pipeline
  6. Containerized ECS based app
    a. Infrastructure (you are here)
    b. CI/CD Pipeline
  7. Serverless app
  8. Summary

You can find the source code on GitHub.

Share on FacebookShare on TwitterShare on LinkedinShare on Pinterest

Filed Under: CLOUD

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Archives

  • January 2021
  • December 2020
  • November 2020
  • October 2020
  • September 2020
  • August 2020
  • July 2020
  • June 2020
  • May 2020
  • April 2020
  • March 2020
  • February 2020
  • January 2020
  • December 2019
  • November 2019
  • October 2019
  • September 2019
  • August 2019
  • July 2019
  • June 2019
  • May 2019
  • April 2019
  • March 2019
  • February 2019
  • January 2019
  • December 2018
  • November 2018
  • October 2018
  • September 2018
  • August 2018
  • July 2018
  • June 2018
  • May 2018
  • April 2018
  • March 2018
  • February 2018
  • January 2018
  • December 2017
  • November 2017
  • October 2017
  • September 2017
  • August 2017
  • July 2017
  • June 2017
  • May 2017
  • April 2017
  • March 2017
  • February 2017
  • January 2017
  • December 2016
  • November 2016
  • October 2016
  • September 2016
  • August 2016
  • July 2016
  • June 2016
  • May 2016
  • April 2016
  • March 2016

Recent Posts

  • Featured on Tech Breakfast Podcast on Jan 15 2021 “Paul Braren: Eyes – Tesla Recall – Tedlexa – Mushrooms – Intel – EVs – Boeing”
  • CES 2021 wrap up: How enterprise tech makes all those smart toilets and robots possible
  • CES 2021 roundup: Tech trends business professionals should care about
  • VMblog 2021 Industry Experts Video Predictions Series – Episode 3
  • Entrust Acquires HyTrust, Expanding Encryption, Key Management and Security Posture Management for Virtualized and Multi-Cloud Environments

Recent Comments

  • +905443535397 on Announcing Cognitive Search: Azure Search + cognitive capabilities

Categories

  • Artificial intelligence
  • BIG Data & Analytics
  • BlockChain
  • CLOUD
  • Data Center
  • IOT
  • Machine Learning
  • SECURITY
  • Storage
  • Uncategorized
  • Virtualization

Categories

  • Artificial intelligence (39)
  • BIG Data & Analytics (33)
  • BlockChain (305)
  • CLOUD (1,568)
  • Data Center (10)
  • IOT (1,851)
  • Machine Learning (143)
  • SECURITY (291)
  • Storage (20)
  • Uncategorized (62)
  • Virtualization (755)

Subscribe Our Newsletter

0% Complete

Copyright © 2021 · News Pro Theme on Genesis Framework · WordPress · Log in