Deployment Automation using Powershell DSC

Immutable Servers using Desired State Configuration

At JUST EAT, when we upgrade our platform services within our AWS hosted environment, we don’t just update the package on the instances, we replace the old servers with new ones built from scratch. This is an extension of the phoenix server approach. The longer a server has been provisioned and running, the more likely it is in an unknown state, creating a problem known as Snowflake servers.
We continuously deploy our components many times a day replacing almost every part of our platform infrastructure. Our continuous deployment strategy encompasses repeatable environments and extremely fast deployment times through pre-built static images (Amazon images – AMIs) and service orchestration times through decentralised orchestration tools (AWS Cloudformation). It’s difficult to guarantee repeatable deployments of the same server due to dependencies on packages, network availability and changes in the environment descriptions hence we achieve predictability using machine images that have been pre-built and tested within a Continuous Integration pipeline. Using version controlled recipes to define server configurations is an important part of continuous delivery.
Desired State Configuration (DSC) is a new management model for Powershell that allows us to deploy and manage configuration data for software services and environments, in turn allowing us to configure how to bring new and existing machines into compliance. DSC provides a set of Windows Powershell language extensions and resources that can be used to specify how you want your server to be configured allowing us to create Pheonix servers that can be recreated reliably from scratch.
DSC is the layer that accepts a configuration in the form of a static vendor neutral Management Object Format (MOF) file, a plain-text file in a format developed by the Distributed Management Task Force (DMTF), and implements it. The extensions in Powershell merely provide a Domain Specific Language (DSL) to define a configuration and generate the final MOF which is then executed by DSC. You could potentially write your own administrative interface to DSC. Moreover, since the MOF is based on an open standard, you could implement your own agent that provides the local and native resources to manage an OS. Hence, DSC is an open, cross platform solution to configuration management. The DSC agent on Windows is called the Local Configuration Manager (LCM) and is easily configured via a DSC configuration. You can use DSC to build a LINUX machine using the Open Management Infrastructure. This post aims to introduce DSC and is an introduction to a series of posts that will dive into advanced features of DSC including the different modes in which DSC operates (push and pull), the LCM and building LINUX AMIs. In order to generate machine images (AMIs) in AWS, we will be pushing a configuration to a remote instance and asking DSC to evaluate it.

Generating an AMI for EC2 in AWS

An Amazon Machine Image (AMI) defines the programs and settings that will be applied when you launch an EC2 instance. We dynamically install and configure our services on EC2 at instance launch time, pulling packages from S3, deploying files and ensuring services are started using CloudFormation. The AMI has the necessary OS features and tools pre-installed in order to speed up stack creation.
In order to create an AMI, we are going to require AWS Tools for Powershell installed. The following steps allow you to create an EC2 instance to create an image from, install features using DSC and finally create an image from it using just Powershell.
1) Generate a key-pair and a security group for managing access to the instance whilst you are building it. You will need to open open up ports for WinRM.

$keypair = New-EC2KeyPair -KeyName $name -ProfileName $profile
"$($keypair.KeyMaterial)" | Out-File -Encoding ascii -Filepath $keyfile
"KeyName: $($keypair.KeyName)" | Out-File -encoding ascii -Filepath $keyfile -Append
"KeyFingerprint: $($keypair.KeyFingerprint)" | Out-File -Encoding ascii -Filepath $keyfile -Append
$groupid = New-EC2SecurityGroup $name -Description $description -ProfileName $profile
$publicIpAddress = <substitute with your public ip address>
$ipRanges = @("$publicIpAddress/32")
Grant-EC2SecurityGroupIngress -GroupName $name -ProfileName $profile -IpPermissions @{IpProtocol = "tcp"; FromPort = 5985; ToPort = 5985; IpRanges = $ipRanges}
Grant-EC2SecurityGroupIngress -GroupName $name -ProfileName $profile -IpPermissions @{IpProtocol = "tcp"; FromPort = 5986; ToPort = 5986; IpRanges = $ipRanges}

2) Find the standard Windows 2012 R2 image from the Amazon store and create an EC2 instance from it and set its attached EBS volumes in the exact way you want them created in the final image. Set the key-pair and security group to the ones created in step 1. Set the user data block to the following:

$newInstances = Get-EC2Image -ProfileName $profile -Filters @{Name = "name"; Values = "$imagePrefix*"}
$imageid = $newInstances[$newInstances.Length-1].ImageId
$imagename = $newInstances[$newInstances.Length-1].Name
$userdata = @"
<powershell>
Set-ExecutionPolicy Unrestricted -Force
Enable-NetFirewallRule FPS-ICMP4-ERQ-In
Set-NetFirewallRule -Name WINRM-HTTP-In-TCP-PUBLIC -RemoteAddress Any
New-NetFirewallRule -Name "WinRM80" -DisplayName "WinRM80" -Protocol TCP -LocalPort 80
New-NetFirewallRule -Name "WinRM443" -DisplayName "WinRM443" -Protocol TCP -LocalPort 443
Enable-PSRemoting -Force
Restart-Service winners
</powershell>
"@
$userdataBase64Encoded = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($userdata))
$parameters = @{
            ImageId = $imageid
            MinCount = 1
            MaxCount = 1
            InstanceType = $instanceType
            KeyName = $keyPairName
            SecurityGroupIds = $securityGroupId # created in step (1)
            UserData = $userdataBase64Encoded
            ProfileName = $profile
            Region = $region
            BlockDeviceMapping = $blockDeviceMapping # see docs..
        }
$newInstances = New-EC2Instance @parameters
Please note that on Windows Server 2012 R2, WinRM is enabled by default, the firewall needs to allow remote management via WinRM.

3) Extract the DNS name from the API and use the key-pair to extract the administrator password as shown below. You can now configure the machine using DSC.

$newInstances = Get-EC2Instance -ProfileName $profile -Filter @{Name = "instance-id"; Values = $instanceid
$publicDNSName = $newInstances.Instances[0].PublicDnsName
$keyfile = Get-KeyFile -KeyPairName $keyPairName -Profile $profile -Folder $folder
$password = Get-EC2PasswordData -InstanceId $instanceid -PemFile $keyfile -Decrypt -ProfileName $profile
$credential = Create-CredentialObject -Username "Administrator" -Password $password

Executing a DSC configuration against an EC2 instance

The following is an example configuration we use at JUST EAT to install the web server features on server core:

Configuration SomeConfiguration
{
  param ([string[]]$computerName = 'localhost')
  Node $computerName
  {
    WindowsFeature Web-AppInit { Ensure = 'Present'; Name = 'Web-AppInit' }
    WindowsFeature Web-Asp-Net45 { Ensure = 'Present'; Name = 'Web-Asp-Net45' }
    WindowsFeature Web-Http-Tracing { Ensure = 'Present'; Name = 'Web-Http-Tracing' }
    WindowsFeature Web-Mgmt-Service { Ensure = 'Present'; Name = 'Web-Mgmt-Service' }
    WindowsFeature Web-Net-Ext { Ensure = 'Present'; Name = 'Web-Net-Ext' }
    WindowsFeature Web-Server { Ensure = 'Present'; Name = 'Web-Server' }
    WindowsFeature Web-WebSockets { Ensure = 'Present'; Name = 'Web-WebSockets' }
    WindowsFeature Web-Mgmt-Compat { Ensure = 'Present'; Name = 'Web-Mgmt-Compat' }
  }
}

The keyword Configuration is an extension to the Powershell language. The keyword Node specifies the machine where the configuration will execute. WindowsFeatrue is a DSC Resource provided out-of-the-box with Windows Management Framework 4.0.
Within this config we are asking DSC to ensure the features Web-AppInit, Web-Asp-Net45, Web-Http-Tracing, Web-Mgmt-Service, Web-Net-Ext, Web-Server, Web-WebSockets and Web-Mgmt-Compat are installed on the machine with name $computerName. We set the property Ensure to Present in order to ask DSC to make sure the feature is present. We can set the value to Absent to remove a feature. You can use the Powershell commandlet Get-WindowsFeature to find the feature names.
You can see the declarative syntax of DSC at work here; we are specifying the desired state of the machine, we don’t really care how it happens. DSC will check if the feature already exists and install it if not found, which means that you can keep running the configuration above to ensure compliance.
This configuration can be extracted into a composite DSC resource, which you can use in multiple configurations. This is quite useful as you can develop a suite of behaviour tests for your own resources which can then be combined to create a robust set of configurations for your images. Creating your own resources is an important part of authoring DSC configurations and we’ll cover this in another post.
Once you have defined a configuration, generate the DSC MOF using:

SomeConfiguration -ComputerName <dns name of the EC2 instance> -OutputPath $outpath

$outpath will contain the DSC MOF file which will be pushed to the EC2 instance.
We can push the MOF to the remote machine and ask DSC to force the remote instance to evaluate the configuration using the following script:

$securepassword = ConvertTo-SecureString $password -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential ($username, $securepassword)
$cimSession = New-CimSession -ComputerName <dns anme of EC2 instance> -Credential $credential -Authentication Negotiate
Start-DscConfiguration -Verbose -Wait -Path $outpath -Force -CimSession $cimSession

And that’s it!
-Wait will wait until the configuration is applied and -Verbose will output the verbose logging in the resources to your local console. Once done, stop the EC2 instance and generate the AMI using the following powershell script:

Stop-EC2Instance -Instance $instanceId -ProfileName $profile
$ami = New-EC2Image -InstanceId $instanceId -Name $name -Description "JustBake generated AMI from configuration $name" -ProfileName $profile

Setting up a task in CI to generate an AMI

DSC is already pre-installed on Windows Server 2012 R2 and if you use just DSC to configure your AMIs, then everything is just a set of text files that live in a version control system like Git. Just clone the repo and execute the scripts on a Windows Server 2012 R2 build agent. The only requirement is that you have the AWS SDK for Windows installed.

The future of DSC

DSC was first mentioned in the Monad Manifesto in 2002 by Jeffery Snover, a distinguished engineer at Microsoft and lead architect for Windows Server and System Center. He describes the delivery of DSC as completing an eleven year journey to deploy automation to the data center in the form of Powershell.
The Windows OS, unlike LINUX is built around an enormous set of disparate APIs, meaning that getting something like Chef, Puppet or even DSC working on Windows is not only harder but can be a stressful task. Microsoft is pouring time and money into creating DSC resources that can, through a very simple and consistent interface, configure most of the OS. The next version of the Windows Management Framework, WMF 5.0, also shipped with Windows 10 will allow installing DSC resources directly from the Powershell Gallery, hence providing a mechanism to share DSC resources, guaranteeing that the coverage of the OS and applications will grow exponentially. Microsoft and other companies out there will be doing a lot of work that you don’t have to!
Microsoft recently announced Nanoserver, a purpose-built OS designed to run cloud applications and containers. Powershell DSC will be the preferred way to remotely manage the OS in the future.
Microsoft is working with Chef to create a set of Cookbooks based on Powershell DSC in order to provide enterprises with a rock-solid, native automation experience for the provision of compute and storage instances on Azure (and AWS). If you are not already using DSC to configure your Windows instances, then you are going to miss out on the work that Microsoft is doing with the community to save them and their users the work to configure the OS.

Useful links

The following book by Jez Humble and David Farley is highly recommended if you want to dig deeper into Deployment Pipelines: Continuous Delivery
http://martinfowler.com/bliki/DeploymentPipeline.html
Advanced Powershell DSC series: http://channel9.msdn.com/Series/Advanced-PowerShell-Desired-State-Configuration-DSC-and-Custom-Resources

  1. Awesome article. Thanks for sharing. As a regular user, this obviously works well for you!
    I used WinRM the last time I implemented something like this and it worked well. I’ve been considering a rewrite using Amazon’s CodeDeploy, since the appspec.yml allows arbitrary scripts to run on the BeforeInstall hook, allowing me to run local scripts like Install-WindowsFeature without the need for the firewall hole or the CI node being tied up waiting. The only struggle I see here is getting the CI chain a notification of completion to trigger further downstream work, like integration tests. Have you looked into CodeDeploy and discounted it? Presumably the structure you’ve described wasn’t the v1 – were there other methods you tried and discarded?

  2. Can someone please modify the above DSC script so it does the same thing but on Azure instead of on AWS? Thanks in advance.

Comments are closed.