We have created the base stack, now let’s consider our monolithic application that it is actually a micro-service, so let’s deploy it to our previously created infrastructure.
After all the base resources are ready and the base stack has been created and completed, we are ready to start writing separate stacks for each micro-service that we want to deploy on the existing instances.
For the moment we consider the monolithic application to be one of the micro-services. We are going to deploy it first, and after that, we are going to create similar tasks and separate the monolithic into micro-services.
So let’s start by first describing the stack which starts our big application on Amazon ECS:
Parameters
1. BaseStackName parameter is useful in the stack for using the exported values from the base stack:
BaseStackName:
Description: Please provide the base stack name
Type: String
Default: tenant1-base-name
2. ServiceName:
ServiceName:
Description: The service name
Type: String
Default: tenant1-service-name
3. TargetGroupName which should be of maximum 32 characters (bad design from Amazon in my opinion):
TargetGroupName:
Description: The target group name (maximum 32 characters)
Type: String
Default: tn1-app-name
4. And, finally, the complete name (including registry URL) of the docker image from Amazon ECR. We will use this to deploy new version of the application by updating the stack and providing different value for the Image parameter:
Image:
Description: The image on ECR to be used
Type: String
Service
In our microservice stack, the first resource we define is the Service, even though it is not the first which will be created (e.g. it references ServiceRole so the ServiceRole presented later will be created at first).
The service resource looks like this:
Service:
Type: AWS::ECS::Service
DependsOn:
# A service can only be created if the target group has an ALB associated
# e.g. error: The target group with targetGroupArn [ARN] does not have
# an associated load balancer.
# (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException;
# Request ID: [REQUEST_ID]])
# This rule makes sure the Target Group is linked to the ALB
- Listener80RuleDefault
Properties:
Cluster:
Fn::ImportValue: !Sub "${BaseStackName}-Cluster"
Role: !Ref ServiceRole
DesiredCount: 2
TaskDefinition: !Ref TaskDefinition
LoadBalancers:
- ContainerName: !Ref ServiceName
ContainerPort: 80
TargetGroupArn: !Ref TargetGroup
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 100
# PlacementConstraints:
# - Type: distinctInstance
As you can see here, we link the service with the cluster by using Fn::ImportValue: !Sub “${BaseStackName}-Cluster” exported value from the base stack.
Another important part here is the fake listener rule (Listener80RuleDefault) we are creating in the Application Load Balancer so that the service can be successfully created, even though not all the listeners have been created/updated.
We have done this for one big reason: we figured out in production that, when we make update to the target group, it is being scheduled for replacement. If the new target group is created and listener rules are changed, then requests are routed to the new target group even before the Service is created and new containers are started. So the customer receives 503 bad gateway until the new service is created. With this fake rule, we make sure that listeners are changed ONLY AFTER the new service is done creating.
Task definition
The task definition is associated to a Service and describes the containers that are going to be created. We may start as many tasks as we want with the same task definition in the same service.
Deploying a new version of the application basically means creating a new task definition and replacing the old one with it in the Service.
The task definition for the monolithic application looks like this:
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Family: "tenant1-app-name-service-production"
ContainerDefinitions:
- Name: !Ref ServiceName
Cpu: '1536'
Essential: true
Image: !Ref Image
# Since we are forced to specify Memory due to
# https://github.com/aws/amazon-ecs-agent/issues/1144
Memory: 6144
# MemoryReservation: 1024
PortMappings:
- ContainerPort: 80
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref AWS::StackName
awslogs-region: !Ref AWS::Region
You may wonder why so many CPU credits and so much memory for a single container? Well, it is because we are deploying the monolithic application which is a huge app, containing a lot of integrated products (as dlls) from different teams. Also, we also had a lot of traffic and couldn’t afford to increase the response latency for our current customers.
Of course, once we split each part of the application in a micro-service, we will reserve less CPU and less Memory for each of them.
CloudWatchLogsGroup
The next resource is a logs group on Amazon Cloud Watch where we can watch logs written by the docker in console. If we start the application with a command like dotnet Tenant1App.dll, then all the console logs will be sent to this LogsGroup, and we will be able to watch and filter through them from the AWS Console UI.
CloudWatchLogsGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Ref AWS::StackName
RetentionInDays: 365
We named the logs group as the name of the current Stack, and if you look at the previous resource (TaskDefinition) you may see that we have specified the same name in the LogConfiguration parameter for the container definition.
Service Scaling
We have created scaling on instances based on CPU reservation left in the base stack. Now it is time to create scaling on the service so we can increase/decrease the number of containers with the same application in production.
For this we need 3 resources: a ScalableTarget and 2 policies for scaling up and down.
We define them like this:
ServiceScalingTarget:
Type: AWS::ApplicationAutoScaling::ScalableTarget
DependsOn: Service
Properties:
MaxCapacity: 5
MinCapacity: 1
ResourceId: !Join
- ''
- - 'service/'
- Fn::ImportValue: !Sub "${BaseStackName}-Cluster"
- '/'
- !GetAtt [Service, Name]
# ['', [service/, Fn::ImportValue: !Sub "${BaseStackName}-Cluster", /,
# !GetAtt [Service, Name]]]
RoleARN: !GetAtt [AutoscalingRole, Arn]
ScalableDimension: ecs:service:DesiredCount
ServiceNamespace: ecs
ScaleUpWhenCPUUtilizationIsHighPolicy:
Type: AWS::ApplicationAutoScaling::ScalingPolicy
Properties:
PolicyName: 'ScaleUpWhenCPUUtilizationIsHigh'
PolicyType: StepScaling
ScalingTargetId: !Ref ServiceScalingTarget
StepScalingPolicyConfiguration:
AdjustmentType: PercentChangeInCapacity
Cooldown: 300
MetricAggregationType: Average
MinAdjustmentMagnitude: 1
StepAdjustments:
- MetricIntervalLowerBound: 0
ScalingAdjustment: 25
ScaleDownWhenCPUUtilizationIsLowPolicy:
Type: AWS::ApplicationAutoScaling::ScalingPolicy
Properties:
PolicyName: 'ScaleDownWhenCPUUtilizationIsLow'
PolicyType: StepScaling
ScalingTargetId: !Ref 'ServiceScalingTarget'
StepScalingPolicyConfiguration:
AdjustmentType: PercentChangeInCapacity
Cooldown: 300
MetricAggregationType: Average
MinAdjustmentMagnitude: 1
StepAdjustments:
- MetricIntervalUpperBound: 0
ScalingAdjustment: -25
We are creating a scalable target and assigning it to the Service, after that we are creating 2 policies and assigning them to the scalable target. The policies says something like this: if CPU utilization is high, then raise the number of containers with 25%, otherwise, if CPU utilization is low, then reduce the number of containers with 25%.
We will later define the alarms which will trigger this policies.
The Target Group
We finally came to creating the target group. A target group can be seen as the connection between the Load Balancer and the containers.
We can say something like: for this domain, go to this target group with the request, but another domain, go to another target group.
One target group is associated with one service and route requests to the multiple containers started by placing tasks in the service.
TargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: !Ref TargetGroupName
VpcId:
Fn::ImportValue: !Sub "${BaseStackName}-VpcId"
Port: 80
Protocol: HTTP
Matcher:
HttpCode: 200-299
HealthCheckIntervalSeconds: 30
HealthCheckPath: /
HealthCheckProtocol: HTTP
HealthCheckTimeoutSeconds: 29
HealthyThresholdCount: 2
UnhealthyThresholdCount: 10
TargetGroupAttributes:
- Key: deregistration_delay.timeout_seconds
Value: 120
Tags:
- Key: Service
Value: !Ref ServiceName
# - Key: StackVersion
# Value: !Ref StackVersion
Some important parameters here: the health check path, which should always work, otherwise the container will not be healthy and therefore requests will not be routed to it. Be careful, if you have no container in service, that means 503 Bad Gateway response for the customer.
Through the de-registration delay you specify how many seconds to wait for the current request to finish when the container is scheduler for deleting/replacing. Basically when a container is about to be deleted, no new requests will be routed to it and the container will continue to run for 120s so running requests have time to finish.
Listener rules for the Listener on port 80
In the base stack we have created 2 listeners: one for port 80 and one for port 443. Now it is the time to create listener rules, so requests are routed to target groups based on the domain.
# NOTE: this default rule is defined so the TargetGroup is
# linked with the ALB and the Service can be created
# This fixes the problem with replacing the target group when
# we make changes to it so the flow becomes like this:
# new target group is created - target group is linked with ALB
# through the Listener80RuleDefault rule - new service is created
# - after the service is created the rest of the listeners are
# created/updated, thus, we will route requests to the new target group
# only when the service is ready! - we avoid 503 gateway errors with this
Listener80RuleDefault:
Type: AWS::ElasticLoadBalancingV2::ListenerRule
Properties:
ListenerArn:
Fn::ImportValue: !Sub "${BaseStackName}-Listener80"
Priority: 2
Conditions:
- Field: host-header
Values:
- tenant1-default.domain.com
Actions:
- TargetGroupArn: !Ref TargetGroup
Type: forward
Listener80Rule1:
Type: AWS::ElasticLoadBalancingV2::ListenerRule
DependsOn:
- Service
Properties:
ListenerArn:
Fn::ImportValue: !Sub "${BaseStackName}-Listener80"
Priority: 100
Conditions:
- Field: host-header
Values:
- tenant1.domain.cloud
Actions:
- TargetGroupArn: !Ref TargetGroup
Type: forward
When you define listeners, you should be careful on the Priority field. If there is already a rule with the same priority on the current listener, your stack will fail to be created.
Listeners for port 443
Listener443Rule1:
Type: AWS::ElasticLoadBalancingV2::ListenerRule
DependsOn:
- Service
Properties:
ListenerArn:
Fn::ImportValue: !Sub "${BaseStackName}-Listener443"
Priority: 100
Conditions:
- Field: host-header
Values:
- tenant.domain.com
Actions:
- TargetGroupArn: !Ref TargetGroup
Type: forward
Roles
Even though we used them already in Service and in AutoScaling, the next resources in our stack are the roles. Since the above resources have pointers to this roles, of course the roles will be created first, even if we define them later in the stack.
1. Service Role:
# This IAM Role grants the service access to register/unregister with the
# Application Load Balancer (ALB). It is based on the default documented here:
# http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_IAM_role.html
ServiceRole:
Type: AWS::IAM::Role
Properties:
# RoleName: !Sub ecs-service-${AWS::StackName}
Path: /
AssumeRolePolicyDocument: |
{
"Statement": [{
"Effect": "Allow",
"Principal": { "Service": [ "ecs.amazonaws.com" ]},
"Action": [ "sts:AssumeRole" ]
}]
}
Policies:
- PolicyName: !Sub ecs-service-${AWS::StackName}
PolicyDocument:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:Describe*",
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
"elasticloadbalancing:Describe*",
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
"elasticloadbalancing:DeregisterTargets",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:RegisterTargets"
],
"Resource": "*"
}]
}
2. AutoScaling Role:
AutoscalingRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: [application-autoscaling.amazonaws.com]
Action: ['sts:AssumeRole']
Path: /
Policies:
- PolicyName: service-autoscaling
PolicyDocument:
Statement:
- Effect: Allow
Action: ['application-autoscaling:*', 'cloudwatch:DescribeAlarms',
'cloudwatch:PutMetricAlarm', 'cloudwatch:GetMetricStatistics',
'ecs:DescribeServices', 'ecs:UpdateService']
Resource: '*'
Alarms
In the last part of our stack, we are defining the alarms which, besides sending emails and SMSs, it also triggers the policies for scaling the application service.
CPUUtilizationIsHighAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: 'Containers CPU Utilization is High'
ActionsEnabled: true
ComparisonOperator: GreaterThanOrEqualToThreshold
EvaluationPeriods: 1
MetricName: CPUUtilization
Namespace: 'AWS/ECS'
Period: 300
Statistic: Average
Threshold: 75
AlarmActions:
- !Ref 'ScaleUpWhenCPUUtilizationIsHighPolicy'
- 'arn:aws:sns:REGION:ACCOUNT_ID:SNS_NAME'
Dimensions:
- Name: ClusterName
Value:
Fn::ImportValue: !Sub "${BaseStackName}-Cluster"
- Name: ServiceName
Value: !GetAtt 'Service.Name'
CPUUtilizationIsLowAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: 'Service is wasting CPU'
ActionsEnabled: true
ComparisonOperator: LessThanOrEqualToThreshold
EvaluationPeriods: 1
MetricName: CPUUtilization
Namespace: 'AWS/ECS'
Period: 300
Statistic: Average
Threshold: 25
AlarmActions:
- !Ref 'ScaleDownWhenCPUUtilizationIsLowPolicy'
Dimensions:
- Name: ClusterName
Value:
Fn::ImportValue: !Sub "${BaseStackName}-Cluster"
- Name: ServiceName
Value: !GetAtt 'Service.Name'
MemoryUtilizationIsHighAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: 'Containers MEMORY Utilization is High'
ActionsEnabled: true
ComparisonOperator: GreaterThanOrEqualToThreshold
EvaluationPeriods: 2
MetricName: MemoryUtilization
Namespace: 'AWS/ECS'
Period: 60
Statistic: Average
Threshold: 90
AlarmActions:
- 'arn:aws:sns:REGION:ACCOUNT_ID:SNS_NAME'
Dimensions:
- Name: ClusterName
Value:
Fn::ImportValue: !Sub "${BaseStackName}-Cluster"
- Name: ServiceName
Value: !GetAtt 'Service.Name'
Finally, we now have 2 stacks:
1. The base stack which creates the base resources needed for running windows containers on windows instances on Amazon ECS (Elastic Container Service).
2. The stack which creates all we need for having the monolithic in production as a docker container on the created ECS cluster.
The next big step in our path is to start separating the monolithic into microservices and create separate stacks for each of them.
As you can see, we have assigned priority 100 to the listener rules routing requests to the monolithic, so that gives us the possibility to route requests for a specific product (e.g. tenant1.domain.com/product1) to another microservice in the future, by using smaller priority in their listener rule.
In the next article, we will use the same stack, but this time we are going to deploy a smaller part of the monolithic as a microservice.