In this article I am going to present to you how our current infrastructure was looking like. Beside presenting it, I will also share with you scripts and code that we used to use.
At this moment we have a big application targeting .NET 4.6.1 (as the framework to rely on). We used to have it on .NET 4.5 but in time we have upgraded it to .NET 4.6.1 so we can use the new features that came up with this framework.
The application is multi-tenant (different databases and different deploys of the code in different places) but, based on the tenant we want to deploy, different parts of the application from different teams are going to be used (this is decided at build time, based on a value which is either passed as a parameter to the msbuild command, or written into a text file in the solution).
I have explained in another article how to build different projects based on a value written in a text file: TODO: Choose projects (csprojs) to build based on a value written in a text file near the solution.
We have defined different build configurations depending on the environment (LOCAL/QA/STAGE/PRODUCTION) and the TENANT we want to deploy.
We were deploying the application directly to the IIS websites (the feature was enabled on our Amazon EC2 instances), by using custom defined msbuild files with the .build extension.
We used to have 2 machines:
- QA machine – quality assurance machine were we do the first deploys of our application and on which we run the unit tests and the manual tests;
- STAGE machine – the “most-close to production” machine (which we use to clone it for production) were we deploy STAGE and PRODUCTION tenants; we usually keep this machine closed and only open it for tests or for production deployment;
The build files we were using for deploying are:
<?xml version="1.0"?>
<Project DefaultTargets="Publish"
ToolsVersion="4.0"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<Configuration>QA</Configuration>
<SrcDir>.\..\src</SrcDir>
<SolutionDir>.\..\src</SolutionDir>
<PublishDir>.\..\publish\$(Tenant)\$(Configuration)\Service</PublishDir>
<AllowedReferenceRelatedFileExtensions>
.pdb;.xml;.manifest
</AllowedReferenceRelatedFileExtensions>
<Tenant>Tenant1</Tenant>
<EnableMSDeployBackup>True</EnableMSDeployBackup>
<MSDeployPublishMethod>WMSVC</MSDeployPublishMethod>
<LastUsedPlatform>Any CPU</LastUsedPlatform>
<ExcludeApp_Data>False</ExcludeApp_Data>
<SkipExtraFilesOnServer>False</SkipExtraFilesOnServer>
<DeployTarget>MSDeployPublish</DeployTarget>
<VisualStudioVersion>14.0</VisualStudioVersion>
<_SavePWD>True</_SavePWD>
</PropertyGroup>
<Choose>
<When Condition=" '$(Configuration)'=='QA' AND '$(Tenant)'=='Tenant1' ">
<PropertyGroup>
<MSDeployServiceURL>IP_OF_THE_QA_MACHINE</MSDeployServiceURL>
<DeployIisAppPath>NAME_OF_THE_IIS_APP</DeployIisAppPath>
</PropertyGroup>
</When>
<When Condition=" '$(Configuration)'=='Stage' AND '$(Tenant)'=='Tenant1' ">
<PropertyGroup>
<MSDeployServiceURL>IP_OF_THE_STAGE_MACHINE</MSDeployServiceURL>
<DeployIisAppPath>NAME_OF_THE_IIS_APP</DeployIisAppPath>
</PropertyGroup>
</When>
<When Condition=" '$(Configuration)'=='Prod' AND '$(Tenant)'=='Tenant1' ">
<PropertyGroup>
<MSDeployServiceURL>IP_OF_THE_PROD_MACHINE</MSDeployServiceURL>
<DeployIisAppPath>NAME_OF_THE_IIS_APP</DeployIisAppPath>
</PropertyGroup>
</When>
<When Condition=" '$(Configuration)'=='QA' AND '$(Tenant)'=='Tenant2' ">
<PropertyGroup>
<MSDeployServiceURL>IP_OF_THE_QA_MACHINE</MSDeployServiceURL>
<DeployIisAppPath>NAME_OF_THE_IIS_APP</DeployIisAppPath>
</PropertyGroup>
</When>
<When Condition=" '$(Configuration)'=='Stage' AND '$(Tenant)'=='Tenant2' ">
<PropertyGroup>
<MSDeployServiceURL>
IP_OF_THE_STAGE_MACHINE
</MSDeployServiceURL>
<DeployIisAppPath>NAME_OF_THE_IIS_APP</DeployIisAppPath>
</PropertyGroup>
</When>
<When Condition=" '$(Configuration)'=='Prod' AND '$(Tenant)'=='Tenant2' ">
<PropertyGroup>
<MSDeployServiceURL>IP_OF_THE_PROD_MACHINE</MSDeployServiceURL>
<DeployIisAppPath>NAME_OF_THE_IIS_APP</DeployIisAppPath>
</PropertyGroup>
</When>
</Choose>
<!-- Items -->
<ItemGroup>
<OutDir Include="$(PublishDir)" />
<Projects Include="$(SrcDir)\PATH_TO_PROJECT\MY_PROJECT.csproj" />
</ItemGroup>
<Target Name="Clean" Condition="Exists(@(OutDir))">
<RemoveDir Directories="$(OutDir)" />
</Target>
<Target Name="Init" DependsOnTargets="Clean">
<MakeDir Directories="$(OutDir)" />
</Target>
<Target Name="Compile" DependsOnTargets="Init">
<MSBuild Projects="@(Projects)"
Targets="Rebuild"
Properties="OutDir=%(OutDir.FullPath);
Configuration=$(Configuration);
VisualStudioVersion=$(VisualStudioVersion)" />
</Target>
<Target Name="Publish" DependsOnTargets="Compile">
<MSBuild Projects="@(Projects)"
Targets="Rebuild"
Properties="Configuration=$(Configuration);
DeployOnBuild=True;
SkipExtraFilesOnServer=$(SkipExtraFilesOnServer);
AllowUntrustedCertificate=True;
MSDeployServiceURL=$(MSDeployServiceURL);
DeployIisAppPath=$(DeployIisAppPath);
UserName=$(UserName);
Password=$(PublishPassword);
DeployTarget=$(DeployTarget);
MSDeployPublishMethod=$(MSDeployPublishMethod);
_SavePWD=$(_SavePWD);
VisualStudioVersion=$(VisualStudioVersion)" />
</Target>
</Project>and for running the unit tests is this one:
<?xml version="1.0"?>
<Project DefaultTargets="Publish"
ToolsVersion="15.0"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<!-- Properties -->
<Configuration>QA</Configuration>
<SrcDir>.\..\src</SrcDir>
<SolutionDir>.\..\src</SolutionDir>
<Tenant>Tenant1</Tenant>
<PublishDir>
.\..\publish\$(Tenant)\$(Configuration)\Tests
</PublishDir>
<TestResultsDir>$(PublishDir)\_TestResults</TestResultsDir>
<VisualStudioVersion>15.0</VisualStudioVersion>
<TestsCategory></TestsCategory>
</PropertyGroup>
<!-- Items -->
<ItemGroup>
<OutDir Include="$(PublishDir)" />
<TestsProject1
Include="$(SrcDir)\Tests\Tests.$(Tenant)\Tests.$(Tenant).csproj" />
<TestsAssemblies1 Include="$(PublishDir)\Tests.$(Tenant).dll" />
<TestsResults1 Include="$(TestResultsDir)\Tests.$(Tenant).trx"/>
<TestsProject2 Include="$(SrcDir)\Tests\Tests2\Tests2.csproj" />
<TestsAssemblies2 Include="$(PublishDir)\Tests2.dll" />
<TestsResults2 Include="$(TestResultsDir)\Tests2.trx"/>
</ItemGroup>
<Target Name="Clean" Condition="Exists(@(OutDir))">
<RemoveDir Directories="$(OutDir)" />
<RemoveDir Directories="$(TestResultsDir)"/>
</Target>
<Target Name="Init" DependsOnTargets="Clean">
<MakeDir Directories="$(TestResultsDir)" />
<MakeDir Directories="$(OutDir)" />
</Target>
<Target Name="Tests" DependsOnTargets="Init">
<MSBuild Projects="@(TestsProject1)"
Targets="Rebuild"
Properties="OutDir=%(OutDir.FullPath);
Configuration=$(Configuration);
VisualStudioVersion=$(VisualStudioVersion)" />
<MSBuild Projects="@(TestsProject2)"
Targets="Rebuild"
Properties="OutDir=%(OutDir.FullPath);
Configuration=$(Configuration);
VisualStudioVersion=$(VisualStudioVersion)" />
<Exec IgnoreExitCode="true"
Command='mstest /testcontainer:@(TestsAssemblies1)
/testcontainer:@(TestsAssemblies2)
/category:"$(TestsCategory)"
/resultsfile:@(TestsResults)' />
</Target>
</Project>QA deploy and unit tests
To deploy our application to QA, it was enough to run the following commands:
- Write the tenant name in a text file (that way we decide what projects to build):
echo Tenant1 > src/.config/tenant.txt- Run the MsBuild command:
msbuild build1.build
/p:VisualStudioVersion=15.0
/toolsversion:15.0
/p:Tenant=Tenant1
/p:Configuration=QA
/p:UserName=QA.UserName
/p:PublishPassword=QA.PublishPasswordWe have to make sure that Web Deploy feature is installed on IIS and that the user that we specify in the command has access to make deployments on the IIS website.
- Finally, to run the unit tests we are using the following command:
msbuild build2.build
/p:VisualStudioVersion=15.0
/toolsversion:15.0
/p:Tenant=Tenant1
/target:Tests
/p:Configuration=QA
/p:TestsCategory=TestsCategorySTAGE deploy and unit tests
To deploy to the STAGE machine, we first need to start it up, since we were usually keeping it close to reduce the costs.
- To start the machine and put the script/job on pause until it is opened, we were using the following power-shell script:
aws ec2 start-instances
--instance-ids ID_OF_THE_INSTANCE --region us-west-2powershell -command "Start-Sleep -s 300"aws ec2 describe-instance-status
--instance-id ID_OF_THE_INSTANCE --region us-west-2- Write the tenant name in a text file (that way we decide what projects to build):
echo Tenant1 > src/.config/tenant.txt- After that, we are running same command as above but with different environment:
msbuild build1.build
/p:VisualStudioVersion=15.0
/toolsversion:15.0
/target:Publish
/p:Configuration=Stage
/p:Tenant=Tenant1
/p:UserName=STAGE.UserName
/p:PublishPassword=STAGE.PublishPassword- Then, running same tests but using different build configuration:
msbuild build2.build
/p:VisualStudioVersion=15.0
/toolsversion:15.0
/p:Tenant=Tenant1
/target:Tests
/p:Configuration=STAGE
/p:TestsCategory=TestsCategory- Finally, after all tests are run, we stop the instance:
aws ec2 stop-instances
--instance-ids ID_OF_THE_INSTANCE --region us-west-2PRODUCTION deploy
For the PRODUCTION deploy, things were a bit more complicated. We first do the same steps as the STAGE deploy (since we are deploying to the same machine).
- We first start the same machine as for the STAGE deployment:
aws ec2 start-instances
--instance-ids ID_OF_THE_INSTANCE --region us-west-2powershell -command "Start-Sleep -s 300"aws ec2 describe-instance-status
--instance-id ID_OF_THE_INSTANCE --region us-west-2- Write the tenant name in a text file (that way we decide what projects to build)
echo Tenant1 > src/.config/tenant.txt- Restore nuget packages:
"src/.nuget/NuGet.exe" restore src/OurProjectSolution.sln- Deploy to the machine but on a different IIS website:
msbuild build1.build
/p:VisualStudioVersion=15.0
/toolsversion:15.0
/target:Publish
/p:Configuration=Production
/p:Tenant=Tenant1
/p:UserName=STAGE.UserName
/p:PublishPassword=STAGE.PublishPassword- Run power-shell script to perform some final tests before the deployment to production (we have assigned extra domains to the IIS website besides the production ones so we can run this tests):
powershell .\build\awsprime.ps1$ErrorActionPreference = "Stop"
$serverName = "IP_OF_THE_SERVER";
function GetCredentials(){
$username = $env:StageUser
$password = $env:StagePassword
$secstr = New-Object -TypeName System.Security.SecureString
$password.ToCharArray() | ForEach-Object {$secstr.AppendChar($_)}
$cred = new-object -typename System.Management.Automation.PSCredential
-argumentlist $username, $secstr;
return $cred;
}
function RestartIIS(){
Try
{
Write-Host " ";
Write-Host "-----------------------------------------------------";
Write-Host "Attempting to restart IIS";
Write-Host "-----------------------------------------------------";
$start=Get-Date;
$cred = GetCredentials;
Invoke-Command -ComputerName $serverName
-ScriptBlock { iisreset } -Authentication Default -Credential $cred
# Invoke-Command -ScriptBlock { iisreset }
$elapsedTime = new-timespan $start $(get-date);
Write-Host "Sucessfully completed script.";
Write-Host "Time taken to run script: " $elapsedTime;
return 1;
}
Catch
{
Write-Host "There was an error running this powershell script";
Write-Host $_.Exception.Message;
return 0;
}
Finally
{
Write-Host "-----------------------------------------------------";
}
}
function HitPages() {
Try
{
Write-Host " ";
Write-Host "-----------------------------------------------------";
Write-Host "Begin hitting pages on the site to pre-load the site";
Write-Host "-----------------------------------------------------";
$start=Get-Date;
$urls = "http://stage.tenant1.com",
"http://stage.tenant1.com/swagger/ui/index";
foreach($page in $urls){
$url = $page;
Write-Host "Page we will be targeting: " + $url;
$pageStart=Get-Date;
Try
{
$Request = [System.Net.HttpWebRequest]::CreateHttp($url);
$Request.Timeout = 600000;
$Request.Method = 'GET';
$response = $Request.GetResponse();
$response.Close();
$elapsedTime = new-timespan $pageStart $(get-date);
Write-Host "Time taken to load the page: " + $elapsedTime;
}
Catch
{
Write-Host "There was an error trying to his the page: " + $url;
Write-Host $_.Exception.Message;
return 0;
}
Write-Host " ";
}
$elapsedTime = new-timespan $start $(get-date);
Write-Host "Time taken to load the pages: " + $elapsedTime;
return 1;
}
Catch
{
Write-Host "There was an error running this powershell script";
Write-Host $_.Exception.Message;
return 0;
}
Finally
{
Write-Host "-----------------------------------------------------";
}
}
$result = RestartIIS;
Write-Host $result;
if (($?) -and ($result)) {
$result = HitPages;
if (($?) -and ($result)) {
exit 0;
}
else {
exit 1;
}
}
else {
exit 1;
}- And the most important and complex step, we have constructed an MSbuild Task in C# which does the final deploy on Amazon:
The build file:
<?xml version="1.0"?>
<Project DefaultTargets="AwsPublish"
ToolsVersion="4.0"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<UsingTask
AssemblyFile="AWSDeploy.MSBuild.Extensions.dll"
TaskName="AWSDeploy"/>
<PropertyGroup>
<ReleaseNumber>r1.1.6.5</ReleaseNumber>
<BuildNumber>2-test</BuildNumber>
<ImageName>api-prod-ami</ImageName>
<InstanceId>IF_OF_THE_INSTANCE</InstanceId>
<InstanceType>c1.medium</InstanceType>
<IAMRole>IAM_ROLE</IAMRole>
<SecurityGroup>AWS_SECURITY_GROUP</SecurityGroup>
<Key>KEY_PEM_FOR_INSTANCES</Key>
<GracePeriod>5</GracePeriod>
<ConfigurationName>AWS_CONFIGURATION_NAME</ConfigurationName>
<GroupName>AWS_ASG_GROUP_NAME</GroupName>
<MinSize>1</MinSize>
<MaxSize>3</MaxSize>
<DesiredCapacity>2</DesiredCapacity>
</PropertyGroup>
<Target Name="AwsPublish">
<AWSDeploy ReleaseNumber="$(ReleaseNumber)"
BuildNumber="$(BuildNumber)"
ImageName="$(ImageName)"
InstanceId="$(InstanceId)"
InstanceType="$(InstanceType)"
IAMRole="$(IAMRole)"
SecurityGroup="$(SecurityGroup)"
Key="$(Key)"
GracePeriod="$(GracePeriod)"
ConfigurationName="$(ConfigurationName)"
GroupName="$(GroupName)"
MinSize="$(MinSize)"
MaxSize="$(MaxSize)"
DesiredCapacity="$(DesiredCapacity)" />
</Target>
<Target Name="Debug">
<Message Text="$(TestResultsDir)" />
</Target>
</Project>- To run the task, we are simply calling the following command with the .build file which will actually run the compiled dll MSBuild Task:
msbuild build3.build
/p:BuildNumber=$BUILD_NUMBER
/p:DesiredCapacity=4
/p:GracePeriod=10
/p:GroupName=ASG_GROUP_NAME
/p:InstanceId=INSTANCE_ID_TO_CLONE
/p:InstanceType=t2.large
/p:MaxSize=5 /p:MinSize=2
/p:ReleaseNumber=r1.1.7.2
/p:Tenant=Tenant1In short, the task does the following:
1. Checks that the instance provided is present on the AWS account;
2. Creates an image from the instance on which we deployed the production code;
3. Waits and checks that the image is successfully created;
4. Creates new launch configuration with the newly created image;
5. Updates the Auto Scaling Group and associate to it the above created launch configuration;
If you want to find out more information about how the task is declared and how to do deploys by replacing instances you can check the following article:
TODO: EC2 scalable infrastructure on Amazon
