icon
Have any questions?
[email protected]

Transforming a monolithic application to a micro-services oriented architecture – Part 4 – Stack for the monolith

Create separate stack for the monolithic

Software Development Outsourcing
Development

Transforming a monolithic application to a micro-services oriented architecture – Part 4 – Stack for the monolith

by February 18, 2019

We have created the base stack, now let’s consider our monolithic application that it is actually a micro-service, so let’s deploy it to our previously created infrastructure.

After all the base resources are ready and the base stack has been created and completed, we are ready to start writing separate stacks for each micro-service that we want to deploy on the existing instances.

For the moment we consider the monolithic application to be one of the micro-services. We are going to deploy it first, and after that, we are going to create similar tasks and separate the monolithic into micro-services.

So let’s start by first describing the stack which starts our big application on Amazon ECS:

Parameters

1. BaseStackName parameter is useful in the stack for using the exported values from the base stack:

BaseStackName:
  Description: Please provide the base stack name
  Type: String
  Default: tenant1-base-name

2. ServiceName:

ServiceName: 
  Description: The service name
  Type: String
  Default: tenant1-service-name

3. TargetGroupName which should be of maximum 32 characters (bad design from Amazon in my opinion):

TargetGroupName:
  Description: The target group name (maximum 32 characters)
  Type: String
  Default: tn1-app-name

4. And, finally, the complete name (including registry URL) of the docker image from Amazon ECR. We will use this to deploy new version of the application by updating the stack and providing different value for the Image parameter:

Image: 
  Description: The image on ECR to be used
  Type: String

Service

In our microservice stack, the first resource we define is the Service, even though it is not the first which will be created (e.g. it references ServiceRole so the ServiceRole presented later will be created at first).

The service resource looks like this:

Service: 
  Type: AWS::ECS::Service
  DependsOn:
    # A service can only be created if the target group has an ALB associated
    # e.g. error: The target group with targetGroupArn [ARN] does not have 
    # an associated load balancer. 
    # (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException;
    # Request ID: [REQUEST_ID]])
    # This rule makes sure the Target Group is linked to the ALB
    - Listener80RuleDefault
  Properties: 
    Cluster: 
      Fn::ImportValue: !Sub "${BaseStackName}-Cluster"
    Role: !Ref ServiceRole
    DesiredCount: 2
    TaskDefinition: !Ref TaskDefinition
    LoadBalancers: 
      - ContainerName: !Ref ServiceName
        ContainerPort: 80
        TargetGroupArn: !Ref TargetGroup
    DeploymentConfiguration:
      MaximumPercent: 200
      MinimumHealthyPercent: 100
    # PlacementConstraints:
    #    - Type: distinctInstance

As you can see here, we link the service with the cluster by using Fn::ImportValue: !Sub “${BaseStackName}-Cluster” exported value from the base stack.

Another important part here is the fake listener rule (Listener80RuleDefault) we are creating in the Application Load Balancer so that the service can be successfully created, even though not all the listeners have been created/updated.

We have done this for one big reason: we figured out in production that, when we make update to the target group, it is being scheduled for replacement. If the new target group is created and listener rules are changed, then requests are routed to the new target group even before the Service is created and new containers are started. So the customer receives 503 bad gateway until the new service is created. With this fake rule, we make sure that listeners are changed ONLY AFTER the new service is done creating.

Task definition

The task definition is associated to a Service and describes the containers that are going to be created. We may start as many tasks as we want with the same task definition in the same service.

Deploying a new version of the application basically means creating a new task definition and replacing the old one with it in the Service.

The task definition for the monolithic application looks like this:

TaskDefinition:
  Type: AWS::ECS::TaskDefinition
  Properties:
    Family: "tenant1-app-name-service-production"
    ContainerDefinitions:
      - Name: !Ref ServiceName
        Cpu: '1536'
        Essential: true
        Image: !Ref Image
        # Since we are forced to specify Memory due to
        # https://github.com/aws/amazon-ecs-agent/issues/1144
        Memory: 6144
        # MemoryReservation: 1024
        PortMappings:
          - ContainerPort: 80
        LogConfiguration:
          LogDriver: awslogs
          Options:
            awslogs-group: !Ref AWS::StackName
            awslogs-region: !Ref AWS::Region

You may wonder why so many CPU credits and so much memory for a single container? Well, it is because we are deploying the monolithic application which is a huge app, containing a lot of integrated products (as dlls) from different teams. Also, we also had a lot of traffic and couldn’t afford to increase the response latency for our current customers.

Of course, once we split each part of the application in a micro-service, we will reserve less CPU and less Memory for each of them.

CloudWatchLogsGroup

The next resource is a logs group on Amazon Cloud Watch where we can watch logs written by the docker in console. If we start the application with a command like dotnet Tenant1App.dll, then all the console logs will be sent to this LogsGroup, and we will be able to watch and filter through them from the AWS Console UI.

CloudWatchLogsGroup:
  Type: AWS::Logs::LogGroup
  Properties: 
    LogGroupName: !Ref AWS::StackName
    RetentionInDays: 365

We named the logs group as the name of the current Stack, and if you look at the previous resource (TaskDefinition) you may see that we have specified the same name in the LogConfiguration parameter for the container definition.

Service Scaling

We have created scaling on instances based on CPU reservation left in the base stack. Now it is time to create scaling on the service so we can increase/decrease the number of containers with the same application in production.

For this we need 3 resources: a ScalableTarget and 2 policies for scaling up and down.

We define them like this:

ServiceScalingTarget:
  Type: AWS::ApplicationAutoScaling::ScalableTarget
  DependsOn: Service
  Properties:
    MaxCapacity: 5
    MinCapacity: 1
    ResourceId: !Join 
      - ''
      - - 'service/'
        - Fn::ImportValue: !Sub "${BaseStackName}-Cluster"
        - '/'
        - !GetAtt [Service, Name]
      # ['', [service/, Fn::ImportValue: !Sub "${BaseStackName}-Cluster", /, 
      # !GetAtt [Service, Name]]]
    RoleARN: !GetAtt [AutoscalingRole, Arn]
    ScalableDimension: ecs:service:DesiredCount
    ServiceNamespace: ecs

ScaleUpWhenCPUUtilizationIsHighPolicy:
  Type: AWS::ApplicationAutoScaling::ScalingPolicy
  Properties:
    PolicyName: 'ScaleUpWhenCPUUtilizationIsHigh'
    PolicyType: StepScaling
    ScalingTargetId: !Ref ServiceScalingTarget
    StepScalingPolicyConfiguration:
      AdjustmentType: PercentChangeInCapacity
      Cooldown: 300
      MetricAggregationType: Average
      MinAdjustmentMagnitude: 1
      StepAdjustments:
        - MetricIntervalLowerBound: 0
          ScalingAdjustment: 25

ScaleDownWhenCPUUtilizationIsLowPolicy:
  Type: AWS::ApplicationAutoScaling::ScalingPolicy
  Properties:
    PolicyName: 'ScaleDownWhenCPUUtilizationIsLow'
    PolicyType: StepScaling
    ScalingTargetId: !Ref 'ServiceScalingTarget'
    StepScalingPolicyConfiguration:
      AdjustmentType: PercentChangeInCapacity
      Cooldown: 300
      MetricAggregationType: Average
      MinAdjustmentMagnitude: 1
      StepAdjustments:
        - MetricIntervalUpperBound: 0
          ScalingAdjustment: -25

We are creating a scalable target and assigning it to the Service, after that we are creating 2 policies and assigning them to the scalable target. The policies says something like this: if CPU utilization is high, then raise the number of containers with 25%, otherwise, if CPU utilization is low, then reduce the number of containers with 25%.

We will later define the alarms which will trigger this policies.

The Target Group

We finally came to creating the target group. A target group can be seen as the connection between the Load Balancer and the containers.

We can say something like: for this domain, go to this target group with the request, but another domain, go to another target group.

One target group is associated with one service and route requests to the multiple containers started by placing tasks in the service.

TargetGroup:
  Type: AWS::ElasticLoadBalancingV2::TargetGroup
  Properties:
    Name: !Ref TargetGroupName
    VpcId: 
      Fn::ImportValue: !Sub "${BaseStackName}-VpcId"
    Port: 80
    Protocol: HTTP
    Matcher: 
      HttpCode: 200-299
    HealthCheckIntervalSeconds: 30
    HealthCheckPath: /
    HealthCheckProtocol: HTTP
    HealthCheckTimeoutSeconds: 29
    HealthyThresholdCount: 2
    UnhealthyThresholdCount: 10
    TargetGroupAttributes:
      - Key: deregistration_delay.timeout_seconds
        Value: 120
    Tags:
      - Key: Service
        Value: !Ref ServiceName
      # - Key: StackVersion
      #  Value: !Ref StackVersion

Some important parameters here: the health check path, which should always work, otherwise the container will not be healthy and therefore requests will not be routed to it. Be careful, if you have no container in service, that means 503 Bad Gateway response for the customer.

Through the de-registration delay you specify how many seconds to wait for the current request to finish when the container is scheduler for deleting/replacing. Basically when a container is about to be deleted, no new requests will be routed to it and the container will continue to run for 120s so running requests have time to finish.

Listener rules for the Listener on port 80

In the base stack we have created 2 listeners: one for port 80 and one for port 443. Now it is the time to create listener rules, so requests are routed to target groups based on the domain.

# NOTE: this default rule is defined so the TargetGroup is
# linked with the ALB and the Service can be created
# This fixes the problem with replacing the target group when 
# we make changes to it so the flow becomes like this:
# new target group is created - target group is linked with ALB 
# through the Listener80RuleDefault rule - new service is created
# - after the service is created the rest of the listeners are 
# created/updated, thus, we will route requests to the new target group 
# only when the service is ready! - we avoid 503 gateway errors with this
Listener80RuleDefault:
  Type: AWS::ElasticLoadBalancingV2::ListenerRule
  Properties:
    ListenerArn:
      Fn::ImportValue: !Sub "${BaseStackName}-Listener80"
    Priority: 2
    Conditions:
      - Field: host-header
        Values: 
          - tenant1-default.domain.com
    Actions:
      - TargetGroupArn: !Ref TargetGroup
        Type: forward

Listener80Rule1:
  Type: AWS::ElasticLoadBalancingV2::ListenerRule
  DependsOn: 
    - Service
  Properties:
    ListenerArn: 
      Fn::ImportValue: !Sub "${BaseStackName}-Listener80"
    Priority: 100
    Conditions:
      - Field: host-header
        Values: 
          - tenant1.domain.cloud
    Actions:
      - TargetGroupArn: !Ref TargetGroup
        Type: forward

When you define listeners, you should be careful on the Priority field. If there is already a rule with the same priority on the current listener, your stack will fail to be created.

Listeners for port 443

Listener443Rule1:
  Type: AWS::ElasticLoadBalancingV2::ListenerRule
  DependsOn: 
    - Service
  Properties:
    ListenerArn: 
      Fn::ImportValue: !Sub "${BaseStackName}-Listener443"
    Priority: 100
    Conditions:
      - Field: host-header
        Values: 
          - tenant.domain.com
    Actions:
      - TargetGroupArn: !Ref TargetGroup
        Type: forward

Roles

Even though we used them already in Service and in AutoScaling, the next resources in our stack are the roles. Since the above resources have pointers to this roles, of course the roles will be created first, even if we define them later in the stack.

1. Service Role:

# This IAM Role grants the service access to register/unregister with the 
# Application Load Balancer (ALB). It is based on the default documented here:
# http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_IAM_role.html
ServiceRole: 
  Type: AWS::IAM::Role
  Properties: 
    # RoleName: !Sub ecs-service-${AWS::StackName}
    Path: /
    AssumeRolePolicyDocument: |
      {
        "Statement": [{
          "Effect": "Allow",
          "Principal": { "Service": [ "ecs.amazonaws.com" ]},
          "Action": [ "sts:AssumeRole" ]
        }]
      }
    Policies: 
      - PolicyName: !Sub ecs-service-${AWS::StackName}
        PolicyDocument: 
          {
            "Version": "2012-10-17",
            "Statement": [{
              "Effect": "Allow",
              "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:Describe*",
                "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
                "elasticloadbalancing:Describe*",
                "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                "elasticloadbalancing:DeregisterTargets",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:RegisterTargets"
              ],
              "Resource": "*"
            }]
          }

2. AutoScaling Role:

AutoscalingRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Statement:
      - Effect: Allow
        Principal:
          Service: [application-autoscaling.amazonaws.com]
        Action: ['sts:AssumeRole']
    Path: /
    Policies:
    - PolicyName: service-autoscaling
      PolicyDocument:
        Statement:
        - Effect: Allow
          Action: ['application-autoscaling:*', 'cloudwatch:DescribeAlarms', 
            'cloudwatch:PutMetricAlarm', 'cloudwatch:GetMetricStatistics',
            'ecs:DescribeServices', 'ecs:UpdateService']
          Resource: '*'

Alarms

In the last part of our stack, we are defining the alarms which, besides sending emails and SMSs, it also triggers the policies for scaling the application service. 

CPUUtilizationIsHighAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmDescription: 'Containers CPU Utilization is High'
    ActionsEnabled: true
    ComparisonOperator: GreaterThanOrEqualToThreshold
    EvaluationPeriods: 1
    MetricName: CPUUtilization
    Namespace: 'AWS/ECS'
    Period: 300
    Statistic: Average
    Threshold: 75
    AlarmActions: 
      - !Ref 'ScaleUpWhenCPUUtilizationIsHighPolicy'
      - 'arn:aws:sns:REGION:ACCOUNT_ID:SNS_NAME'
    Dimensions:
      - Name: ClusterName
        Value: 
          Fn::ImportValue: !Sub "${BaseStackName}-Cluster"
      - Name: ServiceName
        Value: !GetAtt 'Service.Name'

CPUUtilizationIsLowAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmDescription: 'Service is wasting CPU'
    ActionsEnabled: true
    ComparisonOperator: LessThanOrEqualToThreshold
    EvaluationPeriods: 1
    MetricName: CPUUtilization
    Namespace: 'AWS/ECS'
    Period: 300
    Statistic: Average
    Threshold: 25
    AlarmActions: 
      - !Ref 'ScaleDownWhenCPUUtilizationIsLowPolicy'
    Dimensions:
      - Name: ClusterName
        Value: 
          Fn::ImportValue: !Sub "${BaseStackName}-Cluster"
      - Name: ServiceName
        Value: !GetAtt 'Service.Name'

MemoryUtilizationIsHighAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmDescription: 'Containers MEMORY Utilization is High'
    ActionsEnabled: true
    ComparisonOperator: GreaterThanOrEqualToThreshold
    EvaluationPeriods: 2
    MetricName: MemoryUtilization
    Namespace: 'AWS/ECS'
    Period: 60
    Statistic: Average
    Threshold: 90
    AlarmActions: 
      - 'arn:aws:sns:REGION:ACCOUNT_ID:SNS_NAME'
    Dimensions:
      - Name: ClusterName
        Value: 
          Fn::ImportValue: !Sub "${BaseStackName}-Cluster"
      - Name: ServiceName
        Value: !GetAtt 'Service.Name'

Finally, we now have 2 stacks:

1. The base stack which creates the base resources needed for running windows containers on windows instances on Amazon ECS (Elastic Container Service).

2. The stack which creates all we need for having the monolithic in production as a docker container on the created ECS cluster.

The next big step in our path is to start separating the monolithic into microservices and create separate stacks for each of them.

As you can see, we have assigned priority 100 to the listener rules routing requests to the monolithic, so that gives us the possibility to route requests for a specific product (e.g. tenant1.domain.com/product1) to another microservice in the future, by using smaller priority in their listener rule.

In the next article, we will use the same stack, but this time we are going to deploy a smaller part of the monolithic as a microservice.

Tags: