Ben Gelens

Rebooting...

Wed, 26 Jan 2022 12:00:00 +0000

It has been a while since I published anything here. It has been a while since I spoke at conferences / user groups. What is going one with me?

Well mostly the same as with everybody else of course. The pandemic hasn’t been good for my community life. No more traveling, no more in person conferences and user groups. It kind of put a full stop on my contributions. But this is not the whole story.

The personal bits

I’m not one for sharing much about my private life. I’m all about technology as this is the main thing that connects me with other people and makes me happy. But sharing only this aspect of my life doesn’t reflect the whole person and therefore this time, I’m making a tiny exception. So what has been going on besides growing a beard?

I experienced some personal tragedy when I lost my mother in may 2020. She died of complications of something not covid related but unfortunately it did happen during the early days of the pandemic. It tainted her last days being hospitalized, not able to see all of her children at the same time due to visiting restrictions. When she draw her last breath, the hospital kindly made an exception and allowed all of us to be at her side. After my mothers passing, me and my brothers arranged the funeral and I acted as the family proxy for all affairs. As you can imagine, this life experience was pretty hard to deal with.

We decided to buy a new house. At the time of writing it is still under construction and we are looking forward to get the keys somewhere at the end of this year (2022).

We are really looking forward to moving over! In the mean time, we sold our previous house and moved into a temporary appartement (the 4 of us) suited for 3 people. It is a one of a kind appartement as it is build inside of a church, which is awesome but also very noisy and a bit cramped for the 4 of us. I work, sleep, live and cook in the same room. Luckily both of my children both have their own rooms so when they are done with us, they can at least have a bit of privacy :)

Looking forward to the new house, I’ve not made any plans for visiting / speaking at conferences. As soon as the keys are in our possession, we need to make the house ready asap and move over.

The professional bits

Professionally a lot has changed. I used to be a consultant doing mainly automation with PowerShell, infrastructure as code, desired state configuration, that sort of thing. I used a Windows laptop but moved to a MacBook. I worked with VMs but moved to containers. From containers I started working on Kubernetes a lot, more and more open source tooling. Pulumi, Helm, you name it.

Every now and then I contributed some code to some project using a variety of coding languages and this has been a bit of an eye opener for me. I always thought my passion was complex problem solving and integration from a infrastructure perspective, but really what I like to do most, is to code!

My employer, Wortell gave me the opportunity to work part time on our MDR (Managed Detection and Response) service. Writing code for this project from a business value perspective has been great and I learned tons of new things. Since this summer I have moved into the team full time and am now dedicating all time to programming! Learning new things every day, new patterns, new techniques, new tools, etc. etc. In short, I’m having a blast!

This transition does mean I’m kinda moving away from my goto contribution area. I still use PowerShell everyday (can’t live / work without it) but my professional interests have moved into other directions. As my experience grows, I’m looking forward to connecting with the community again and sharing knowledge. Hopefully sooner than later as I miss being connected and feeling being part of a community.

AWS Custom Config Rule backed by PowerShell

Wed, 05 Aug 2020 12:00:00 +0000

AWS Config Rules allow you to evaluate the configuration settings of AWS resources during creation, update, deletions of such resources or on a schedule. They are used to audit resources for compliance with some defined policy. They don’t prevent you from deploying in-compliant resources as this is the realm of service control policies.

A set of pre-defined (Managed) config rules exist which can easily be used, but these inbox rules might not cover the policy that needs to be evaluated. For this, AWS offers you to build your own custom config rules.

A custom config rule is backed by a AWS Lambda function that contains the logic to handle the evaluation. A lot of examples can be found that use Java, Node or Python. There is also the RDK (Rule Development Kit) that can help you develop custom rules with Lambda functions in one of the, by the toolkit, supported languages (C#, Java, Node, Python). I was not able to find any example however to use PowerShell for the Lambda function.

In this blog I’m providing an example to have the Lambda function use PowerShell as the language to do the evaluation with a sample use case to evaluate S3 bucket’s compliance with a naming convention.

Getting started

In this blog post I’m assuming AWS credentials to already be setup. Also I’m using the AWS.Tools PowerShell modules.

Before we start coding the Lambda function, we need all the AWS resources in place so we can actually see what we get as input to that function and how to process that data. The custom config rule will be deployed by using a CloudFormation template. The Lambda code will be staged in a S3 bucket so it can be referenced to during the Cloud Formation Stack deployment and updated if that code changes and the Stack gets updated.

I’m assuming the config service is already configured. If not, please follow this to setup the service.

CloudFormation Template

I’ve prepared the following CloudFormation template:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Description: Custom Config Rule S3
Parameters:
  SourceBucketName:
    Description: Bucket containing the lambda zip files
    Type: String

  NamingConventionPattern:
    Default: "*"
    Type: String

  PackageVersion:
    Type: String

Resources:
  S3ConfigRulesLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: S3ConfigRulesLambdaRolePolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - config:PutEvaluations
                Resource: "*"
                Sid: ConfigPutEvaluation

  S3ConfigRulesLambda:
    Type: AWS::Lambda::Function
    DependsOn: S3ConfigRulesLambdaRole
    Properties:
      Runtime: dotnetcore3.1
      Code:
        S3Bucket: !Ref SourceBucketName
        S3Key: s3.zip
        S3ObjectVersion: !Ref PackageVersion
      Role: !GetAtt
        - S3ConfigRulesLambdaRole
        - Arn
      Handler: s3::s3.Bootstrap::ExecuteFunction
      MemorySize: 512
      Timeout: 300

  S3ConfigRolePermission:
    Type: AWS::Lambda::Permission
    DependsOn: S3ConfigRulesLambda
    Properties:
      FunctionName: !GetAtt
        - S3ConfigRulesLambda
        - Arn
      Action: lambda:InvokeFunction
      Principal: config.amazonaws.com

  S3NamingConventionConfigRule:
    Type: AWS::Config::ConfigRule
    DependsOn: S3ConfigRolePermission
    Properties:
      ConfigRuleName: s3-naming-convention
      Description: Checks for s3 naming convention
      InputParameters:
        likePattern: !Ref NamingConventionPattern
      Scope:
        ComplianceResourceTypes:
          - "AWS::S3::Bucket"
      Source:
        Owner: CUSTOM_LAMBDA
        SourceDetails:
          - EventSource: aws.config
            MessageType: ConfigurationItemChangeNotification
        SourceIdentifier: !GetAtt
          - S3ConfigRulesLambda
          - Arn

The template will create the following resources:

IAM Role: The Lambda function will execute as this role and will therefore have the permissions associated. In this case this permission is required: Allow config:PutEvaluations for all resources.
Lambda Function: The Lambda function itself configured to have dotnet core 3.1 as the runtime (we can run PowerShell 7 with this runtime). The code is coming from a zip file that is uploaded to a S3 bucket which will be configured with versioning enabled (uploading a new version of the zip file will result in a new version, deploying the CloudFormation template again, will see the version update, triggering the Lambda to pull in the new code).
Lambda Permission: The permission will allow the config service to trigger the Lambda Function.
ConfigRule: The Config Rule is rule representing our policy. It is configured with source owner CUSTOM_LAMBDA signaling this is a custom config rule (managed config rule uses the same CloudFormation resource where this property would be set to AWS). The input parameters allow us to have the Lambda function take in a value defined here, making the Lambda code more reusable. The config rule will evaluate based on change notifications (CRUD). The handler is hardcoded with an expected value (we’ll see later), if you call your function script differently, the handler name will be different as well.

Initial PowerShell Lambda code

To start coding PowerShell Lambda functions, the developer needs to:

Install dotnet core 3.1 SDK
Install (and use) PowerShell 7
Install the AWSLambdaPSCore PowerShell module (Install-Module AWSLambdaPSCore). At the time of writing I’m using version 2.0.0.

Additionally for our function we need the following PowerShell modules installed:

AWS.Tools.Common
AWS.Tools.ConfigService

Additionally for interacting with AWS we need the following PowerShell modules:

AWS.Tools.S3
AWS.Tools.CloudFormation

At the time of writing I’m using version 4.0.6 of these modules.

From the AWSLambdaPSCore module, we’ll use 2 cmdlets:

New-AWSPowerShellLambda: (scaffolding the function code)
New-AWSPowerShellLambdaPackage: Creating the zip file (handles the dotnet restore, etc.)

Let’s create the function:

New-AWSPowerShellLambda -Template Basic -ScriptName s3

The cmdlet will create a directory named s3 at the path currently set and in the directory it will create a ps1 file named s3.

The content of s3.ps1 will look like:

# PowerShell script file to be executed as a AWS Lambda function. 
# 
# When executing in Lambda the following variables will be predefined.
#   $LambdaInput - A PSObject that contains the Lambda function input data.
#   $LambdaContext - An Amazon.Lambda.Core.ILambdaContext object that contains information about the currently running Lambda environment.
#
# The last item in the PowerShell pipeline will be returned as the result of the Lambda function.
#
# To include PowerShell modules with your Lambda function, like the AWS.Tools.S3 module, add a "#Requires" statement
# indicating the module and version. If using an AWS.Tools.* module the AWS.Tools.Common module is also required.

#Requires -Modules @{ModuleName='AWS.Tools.Common';ModuleVersion='4.0.6.0'}

# Uncomment to send the input event to CloudWatch Logs
# Write-Host (ConvertTo-Json -InputObject $LambdaInput -Compress -Depth 5)

The scaffolded ps1 file contains all the info required to start developing. One thing to know is that the Requires modules statements are used during packaging with the New-AWSPowerShellLambdaPackage cmdlet, to inject the PowerShell modules in the zip file. These modules have to be available (installed locally).

We’ll modify the content of the ps1 file a little bit so it is clean and logs the LambdaInput variable to CloudWatch.

#Requires -Modules @{ModuleName='AWS.Tools.Common'; ModuleVersion='4.0.6.0'}

# log the input to cloudwatch
Write-Host -Object (ConvertTo-Json -InputObject $LambdaInput -Compress -Depth 5)

Now let’s package the Lambda function:

New-AWSPowerShellLambdaPackage -ScriptPath './s3/s3.ps1' -OutputPackage './s3.zip'

As you can see, the New-AWSPowerShellLambdaPackage cmdlet handles a couple of things for us.

Note the LambdaHandler value. If this value is different from the CloudFormation template value, you’ll need to change the value on the CloudFormation template.

Creating the staging bucket

Let’s create the staging bucket used by the CloudFormation Template to fetch the Lambda zip file from. The script below will create the bucket if it does not yet exists, enables versioning and setup some default best practices like encryption at rest and denial of public access.

# set bucketname variable
$bucketName = 'bgstagingbucket'

# setup staging s3
$s3Bucket = Get-S3Bucket -BucketName $bucketName
$s3Bucket ??= New-S3Bucket -BucketName $bucketName -CannedACLName 'private' -ErrorAction Stop
Write-S3BucketVersioning -BucketName $bucketName -VersioningConfig_Status 'Enabled'

# Block public access for the bucket and objects
$s3PublicParams = @{
  BucketName                                          = $bucketName
  PublicAccessBlockConfiguration_BlockPublicAcl       = $true
  PublicAccessBlockConfiguration_BlockPublicPolicy    = $true
  PublicAccessBlockConfiguration_IgnorePublicAcl      = $true
  PublicAccessBlockConfiguration_RestrictPublicBucket = $true
}
Add-S3PublicAccessBlock @s3PublicParams

# Ensure Encryption
$s3EncryptionParams = @{
  BucketName                                                 = $bucketName
  ServerSideEncryptionConfiguration_ServerSideEncryptionRule = @{
    ServerSideEncryptionByDefault = @{
      ServerSideEncryptionAlgorithm = 'AES256'
    }
  }
}
Set-S3BucketEncryption @s3EncryptionParams

Now we have the staging bucket setup, let’s upload the s3.zip file created earlier.

$s3Bucket | Write-S3Object -File './s3.zip' -Force
$s3Bucket | Get-S3Object -Key s3.zip

Deploying the initial config rule

To deploy the config rule we need to fetch the current s3 object version first:

$version = (Get-S3Version -BucketName $bucketName -Prefix 's3.zip').Versions.Where{ $_.IsLatest }[0].VersionId

Now we have the version, we can deploy the CloudFormation template (note that I saved the above template to disk first):

# explicit import to make types available as we cannot rely on autoloading due to sequence of script
Import-Module -Name 'AWS.Tools.CloudFormation'

# make a list of parameter values for cloud formation deployment
$cfnParams = [System.Collections.Generic.List[Amazon.CloudFormation.Model.Parameter]]::new()

# the source bucket name
$cfnParams.Add([Amazon.CloudFormation.Model.Parameter]@{
  ParameterKey   = 'SourceBucketName'
  ParameterValue = $bucketName
})

# the naming convention buckets should adhere to. In my case prefixed with bg is allowed
$cfnParams.Add([Amazon.CloudFormation.Model.Parameter]@{
  ParameterKey   = 'NamingConventionPattern'
  ParameterValue = 'bg*'
})

# the version fetched earlier
$cfnParams.Add([Amazon.CloudFormation.Model.Parameter]@{
  ParameterKey   = 'PackageVersion'
  ParameterValue = $version
})

$deployArgs = @{
  StackName   = "config-rule-s3-naming"
  Parameter   = $cfnParams
  Capability  = 'CAPABILITY_IAM', 'CAPABILITY_AUTO_EXPAND'
  TemplateBody = (Get-Content -Path ./configrule.yaml -Raw)
}

# deploy cloudformation template and wait for maximum 15 minutes
New-CFNStack @deployArgs | Wait-CFNStack -Timeout (15 * 60)

Note the capabilities set are needed for CloudFormation to be allowed to handle certain tasks. For more info see the API documention

Investigating the deployed stack in the console:

Navigating to the config service, we see the newly deployed custom config rule with No results available.

Iterating the config rule

Now everything is in place, we need to develop the logic for the Lambda function. To do this, we’ll need some logging information. Let’s create an s3 bucket that violates our policy and one that complies with it. Note that I set the policy to allow s3 buckets with bg as the prefix (see above). As my stating S3 bucket is already compliant, we’ll need a S3 bucket that is not compliant to see both results.

Creating in-compliant S3 bucket

New-S3Bucket -BucketName 'notcompliantbgbucket'

Evaluate logging

Let’s start a manually triggered evaluation (this is handy when developing iteratively):

Start-CFGConfigRulesEvaluation -ConfigRuleName 's3-naming-convention'

In the console, navigate to the CloudWatch service, select log groups and search for the log group associated with your Lambda function.

Once you click on the log group, you’ll see different log streams. Just click the most recent one:

Inspect one of the log messages with the [Information] header. Now we can see what information is available as part of the LambdaInput.

Adjusting the Lambda code

Now we know what is being send, we can adjust the Lambda code so it will evaluate what we desire and is able to report back to the config service.

#Requires -Modules @{ModuleName='AWS.Tools.Common'; ModuleVersion='4.0.6.0'}
#Requires -Modules @{ModuleName='AWS.Tools.ConfigService'; ModuleVersion='4.0.6.0'}

# log the input to cloudwatch
Write-Host -Object (ConvertTo-Json -InputObject $LambdaInput -Compress -Depth 5)

$invokingEventDeserialized = $LambdaInput.invokingEvent | ConvertFrom-Json -Depth 20
$bucketName = $invokingEventDeserialized.configurationItem.resourceName
$ruleParameters = $LambdaInput.ruleParameters | ConvertFrom-Json
$likePattern = $ruleParameters.likePattern

Write-Host -Object "Evaluating BucketName: $bucketName against naming pattern: $likePattern"

$complianceType = if ($LambdaInput.eventLeftScope) {
  'NOT_APPLICABLE'
} elseif ($bucketName -notlike $likePattern) {
  'NON_COMPLIANT'
} else {
  'COMPLIANT'
}

$evalResult = [Amazon.ConfigService.Model.Evaluation]@{
  ComplianceResourceId   = $invokingEventDeserialized.configurationItem.resourceId
  ComplianceResourceType = $invokingEventDeserialized.configurationItem.resourceType
  ComplianceType         = $complianceType
  OrderingTimestamp      = [datetime]::UtcNow
}

Write-Host -Object "EvalResult:`n$($evalResult | ConvertTo-Json)"
Write-CFGEvaluation -Evaluation $evalResult -ResultToken $LambdaInput.resultToken

A few things to note:

The AWS.Tools.ConfigService Module has a cmdlet Write-CFGEvaluation which is capable of handling the reporting to the config service (no need to write your own). By including it as a Required Module, the New-AWSPowerShellLambdaPackage will includes it in the package).
When eventLeftScope is true, a resource was deleted. No need to evaluate a deleted resource.

As you can see, the end result is pretty lean which is one of the reasons I think PowerShell should be used a little bit more for Custom Config Rules.

Let’s republish the Lambda function code:

Generate new zip file:

New-AWSPowerShellLambdaPackage -ScriptPath './s3/s3.ps1' -OutputPackage './s3.zip'
$s3Bucket | Write-S3Object -File './s3.zip' -Force

Update CloudFormation Stack:

$version = (Get-S3Version -BucketName $bucketName -Prefix 's3.zip').Versions.Where{ $_.IsLatest }[0].VersionId

$cfnParams = [System.Collections.Generic.List[Amazon.CloudFormation.Model.Parameter]]::new()

# the source bucket name
$cfnParams.Add([Amazon.CloudFormation.Model.Parameter]@{
  ParameterKey   = 'SourceBucketName'
  ParameterValue = $bucketName
})

# the naming convention buckets should adhere to. In my case prefixed with bg is allowed
$cfnParams.Add([Amazon.CloudFormation.Model.Parameter]@{
  ParameterKey   = 'NamingConventionPattern'
  ParameterValue = 'bg*'
})

# the version fetched earlier
$cfnParams.Add([Amazon.CloudFormation.Model.Parameter]@{
  ParameterKey   = 'PackageVersion'
  ParameterValue = $version
})

$deployArgs = @{
  StackName   = "config-rule-s3-naming"
  Parameter   = $cfnParams
  Capability  = 'CAPABILITY_IAM', 'CAPABILITY_AUTO_EXPAND'
  TemplateBody = (Get-Content -Path ./configrule.yaml -Raw)
}

try {
  Update-CFNStack @deployArgs -Force | Wait-CFNStack -Timeout (15 * 60)
} catch {
  if ($_.Exception.Message -eq 'No updates are to be performed.') {
    Write-Host -Object 'Stack exist, no updates required'
  } else {
    throw $_
  }
}

End result

Now everything is in place, let’s trigger a re-evaluation:

Start-CFGConfigRulesEvaluation -ConfigRuleName 's3-naming-convention'

In the console, we can navigate to the config service. Go to the rules, select the s3-naming-convention rule and check for resources that are compliant:

and noncompliant:

Azure DevOps Agents as Container Instances

Wed, 21 Aug 2019 12:00:00 +0000

The Microsoft provided hosted build agents for Azure DevOps might not suite all requirements. E.g. the Az PowerShell modules on the images provided by Microsoft are lagging behind. To compensate, in general, pipelines spend a lot of time installing dependencies to complete the job at hand. Having a custom build agent can resolves these issues as the dependencies are installed at image creation and available from there on, thus these images beter suite your needs (build for purpose).

You can host build agents on any compute platform. For this solution we build docker images and host them on Azure Container Instances as it is relatively easy to create container images containing all the requirements compared to VMs. It is also far easier creating new versions as the creation of the image is fully automated.

Create Agent Pool

To create an Agent Pool please follow the documentation.

Build container image

For this blog we’ll create images for both Windows and Linux.

Linux

The image run commands are based on the Linux Azure DevOps agent guide.

The Linux image is based on mcr.microsoft.com/powershell:6.2.0-ubuntu-18.04 which is an Ubuntu 18.04 image with PowerShell Core 6.2 pre-installed managed by Microsoft (see DockerHub for more info).

Safe the following Dockerfile as Linux.Dockerfile:

# escape=`

FROM mcr.microsoft.com/powershell:6.2.0-ubuntu-18.04

ARG agentversion=2.155.1
ENV agentversion=${agentversion}

SHELL [ "pwsh", "-NoProfile", "-Command" ]

ENV AGENT_ALLOW_RUNASROOT 1

WORKDIR /agent

RUN $ProgressPreference = 'SilentlyContinue' ; `
    Invoke-WebRequest -Uri "https://vstsagentpackage.azureedge.net/agent/$env:agentversion/vsts-agent-linux-x64-$env:agentversion.tar.gz" -OutFile "./vsts-agent-linux-x64-$env:agentversion.tar.gz" -UseBasicParsing ; `
    tar  zxvf "./vsts-agent-linux-x64-$env:agentversion.tar.gz" ; `
    rm "./vsts-agent-linux-x64-$env:agentversion.tar.gz" ; `
    ./bin/installdependencies.sh ; `
    Install-Module Az -Force -Scope AllUsers ; `
    Install-Module Pester -Force -Scope AllUsers ; `
    Install-Module PSScriptAnalyzer -Force -Scope AllUsers ; `
    apt install git -y ; `
    Invoke-WebRequest -Uri https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb -OutFile ./packages-microsoft-prod.deb -UseBasicParsing ; `
    dpkg -i ./packages-microsoft-prod.deb ; `
    rm ./packages-microsoft-prod.deb ; `
    apt-get install apt-transport-https -y ; `
    apt-get update ; `
    apt-get install dotnet-sdk-2.2 -y ; `
    apt-get install zip -y

ENTRYPOINT [ "/bin/bash", "-c", "./config.sh --unattended --replace && ./run.sh" ]

As you can see, I preselected some things to install like the Az modules, Pester and PSScriptAnalyzer. Also, the dotnet sdk is installed in case you need it (e.g. deploy Azure Functions). Now to build the image:

docker build --force-rm -t azd-ubuntu1804-pwsh6.2.0 -f Linux.Dockerfile .

Note that the Agent Version downloaded is passed in via arguments with a default of 2.155.1 which is the latest at the time of writing. You can overwrite by adding the --build-arg argument to docker build. E.g: docker build --force-rm -t azd-ubuntu1804-pwsh6.2.0 --build-arg agentversion=2.150.3 -f ./Linux.Dockerfile . to make use of agent version 2.150.3.

Windows

The Windows image is based on mcr.microsoft.com/powershell:6.2.0-windowsservercore-1809 which is a Windows Server 2019 / 1809 image with PowerShell Core 6.2 pre-installed managed by Microsoft (see DockerHub for more info).

Safe the following Dockerfile as Windows.Dockerfile:

# escape=`

FROM mcr.microsoft.com/powershell:6.2.0-windowsservercore-1809

ARG agentversion=2.155.1
ENV agentversion=${agentversion}

SHELL [ "pwsh", "-NoProfile", "-Command" ]

WORKDIR /agent

COPY [ "DnsFix.ps1", "./DnsFix.ps1" ]

RUN $ProgressPreference = 'SilentlyContinue' ; `
    Invoke-WebRequest -Uri "https://vstsagentpackage.azureedge.net/agent/$env:agentversion/vsts-agent-win-x64-$env:agentversion.zip" -OutFile "./vsts-agent-win-x64-$env:agentversion.zip" -UseBasicParsing ; `
    Expand-Archive -Path "./vsts-agent-win-x64-$env:agentversion.zip" -DestinationPath . ; `
    Remove-Item "./vsts-agent-win-x64-$env:agentversion.zip" ; `
    New-Item -ItemType Directory -Name Modules -Path c:\ -Force ; `
    Save-Module Az -Path c:\Modules ; `
    Save-Module Pester -Path c:\Modules ; `
    Save-Module PSScriptAnalyzer -Path c:\Modules ; `
    Invoke-WebRequest -Uri https://download.visualstudio.microsoft.com/download/pr/3c43f486-2799-4454-851c-fa7a9fb73633/673099a9fe6f1cac62dd68da37ddbc1a/dotnet-sdk-2.2.203-win-x64.exe -OutFile ./dotnet-sdk-2.2.203-win-x64.exe ; `
    Start-Process ./dotnet-sdk-2.2.203-win-x64.exe -ArgumentList '-q' -Wait ; `
    Remove-Item ./dotnet-sdk-2.2.203-win-x64.exe ; `
    Invoke-WebRequest -Uri https://github.com/git-for-windows/git/releases/download/v2.21.0.windows.1/Git-2.21.0-64-bit.exe -OutFile ./Git-2.21.0-64-bit.exe ; `
    Start-Process ./Git-2.21.0-64-bit.exe -ArgumentList ' /VERYSILENT /NORESTART /SUPPRESSMSGBOXES /NOCANCEL /SP-' -Wait ; `
    Remove-Item ./Git-2.21.0-64-bit.exe ; `
    $machinePath = [environment]::GetEnvironmentVariable('path', [System.EnvironmentVariableTarget]::Machine) ; `
    $newMachinePath = 'C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\bin;' + $machinePath ; `
    [environment]::SetEnvironmentVariable('path', $newMachinePath, [System.EnvironmentVariableTarget]::Machine) ; `
    $config = cat $PSHOME/powershell.config.json | ConvertFrom-Json -AsHashtable ; `
    $config.Add('PSModulePath','%ProgramFiles%\PowerShell\Modules;%ProgramFiles%\powershell\latest\Modules;%windir%\system32\WindowsPowerShell\v1.0\Modules;C:\Modules') ; `
    $config | ConvertTo-Json | Out-File $PSHOME/powershell.config.json -Force ; `
    $machineV5ModPath = [environment]::GetEnvironmentVariable('PSModulePath', [System.EnvironmentVariableTarget]::Machine) ; `
    $newMachineV5ModPath = 'c:\Modules;' + $machineV5ModPath ; `
    [environment]::SetEnvironmentVariable('PSModulePath', $newMachineV5ModPath, [System.EnvironmentVariableTarget]::Machine)

ENTRYPOINT [ "C:\\Windows\\system32\\cmd.exe", "/C", "pwsh -f .\\DnsFix.ps1 -nol -noni -nop && .\\config.cmd --unattended --replace && .\\run.cmd" ]

Again, as with the Linux image, I preselected some things to install like the Az modules, Pester, PSScriptAnalyzer and the dotnet sdk. There is also a DnsFix.ps1 that is copied in the image to workaround an issue with Azure Container Instances and Windows containers. I’ve hit this issue myself hence the inclusion of this fix (have the container make use of google dns servers). Make sure you save the following as DnsFix.ps1 in the same folder as you saved the dockerfile.

# this is a workaround for dns issues with Windows Containers in ACI
$dnsClientSettings = Get-DnsClientServerAddress -AddressFamily IPv4 | Where-Object -FilterScript { $_.ElementName -NotLike "Loopback*" }
$dnsClientSettings | Set-DnsClientServerAddress -ServerAddresses @('8.8.8.8', '8.8.4.4')

Now to build the image:

docker build --force-rm -t azd-windows1809-pwsh6.2.0 -f Windows.Dockerfile .

Note that the Modules downloaded for Windows are shared between Windows PowerShell and PowerShell. Some additional configuration is done to make sure this Module path is available to both. The reason for this is that the Azure PowerShell task on Windows targets Windows PowerShell by default until this issue is fixed.

Running locally

The image requires a PAT token to connect with Azure DevOps. See the documentation on how to acquire one.

To test the resulting image locally run:

PATTOKEN="get a pat token from azure devops"
docker run --rm -it -e VSTS_AGENT_INPUT_URL=https://dev.azure.com/<MyOrch> -e VSTS_AGENT_INPUT_AUTH=pat -e VSTS_AGENT_INPUT_TOKEN=$PATTOKEN -e VSTS_AGENT_INPUT_POOL=<MyPoolName> -e VSTS_AGENT_INPUT_AGENT=agent-0 --user azd:azd azd-ubuntu1804-pwsh6.2.0

Publish container image to ACR

This blog assumes an Azure Container Registry is already created. See the guides on how to handle this task if not already created.

To publish the just created image to Azure Container Registry, make sure that the registry is enabled for admin access so you have a username and password to authenticate to it.

docker login <myregistry>.azurecr.io

Tag the image so docker knows where to send it:

# linux
docker tag azd-ubuntu1804-pwsh6.2.0 <myregistry>.azurecr.io/azd/azd-ubuntu1804-pwsh6.2.0

# windows
docker tag azd-windows1809-pwsh6.2.0 <myregistry>.azurecr.io/azd/azd-windows1809-pwsh6.2.0

Push the image:

# linux
docker push <myregistry>.azurecr.io/azd/azd-ubuntu1804-pwsh6.2.0

# windows
docker push <myregistry>.azurecr.io/azd/azd-windows1809-pwsh6.2.0

Deploy container instance

To deploy / update an Azure Container Instance, I’ve prepared and ARM template.

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "containerInstanceName": {
      "type": "string",
      "metadata": {
        "description": "Specifies the name of the Container Instance"
      }
    },
    "containerCount": {
      "type": "int",
      "minValue": 1,
      "defaultValue": 1,
      "metadata": {
        "description": "Specifies the amount of containers to deploy to the Container Instance. (for Windows the maximum is 1. You cannot update the count after deployment)"
      }
    },
    "patToken": {
      "type": "securestring",
      "metadata": {
        "description": "The personal access token used by the agent(s) to connect to Azure DevOps"
      }
    },
    "azdUrl": {
      "type": "string",
      "metadata": {
        "description": "The url of the Azure DevOps project (https://dev.azure.com/<projectName>)"
      }
    },
    "agentPool": {
      "type": "string",
      "metadata": {
        "description": "The pool in which the agent(s) should join"
      }
    },
    "agentNamePrefix": {
      "type": "string",
      "metadata": {
        "description": "The name prefix for the agent(s) (e.g. aci-agent-ubuntu- will result in aci-agent-ubuntu-1 for the first container and so on)"
      }
    },
    "registryLoginServer": {
      "type": "string",
      "metadata": {
        "description": "The Container Registry login url (e.g. <name>.azurecr.io) where the image will be pulled from"
      }
    },
    "registryUserName": {
      "type": "string",
      "metadata": {
        "description": "The Container Registry UserName"
      }
    },
    "registryPassword": {
      "type": "securestring",
      "metadata": {
        "description": "The Container Registry Password"
      }
    },
    "registryImageUri": {
      "type": "string",
      "metadata": {
        "description": "The Container Image Uri (e.g. <registryName>.azurecr.io/<libraryName>/<imageName>:<imageTag>)"
      }
    },
    "memoryInGb": {
      "type": "string",
      "defaultValue": "3.5",
      "metadata": {
        "description": "The amount of memory to assign to the container(s)"
      }
    },
    "cpuCount": {
      "type": "string",
      "defaultValue": "2",
      "metadata": {
        "description": "The amount of CPU(s) to assign to the container(s)"
      }
    },
    "osType": {
      "type": "string",
      "defaultValue": "Linux",
      "metadata": {
        "description": "The OSType of the container(s) correlating to the image being pulled"
      }
    },
    "restartPolicy": {
      "type": "string",
      "defaultValue": "Always",
      "metadata": {
        "description": "The restart policy for a container (Always, OnFailure, Never)"
      }
    },
    "assignManagedIdentity": {
      "type": "bool",
      "defaultValue": false,
      "metadata": {
        "description": "Specify if a Managed Identity should be assigned"
      }
    }
  },
  "variables": {},
  "resources": [
    {
      "type": "Microsoft.ContainerInstance/containerGroups",
      "apiVersion": "2018-10-01",
      "name": "[parameters('containerInstanceName')]",
      "location": "[resourceGroup().location]",
      "identity": "[if(parameters('assignManagedIdentity'), json('{\"type\": \"SystemAssigned\"}'), json('null'))]",
      "properties": {
        "copy": [
          {
            "name": "containers",
            "count": "[if(equals(parameters('osType'), 'Windows'), 1, parameters('containerCount'))]",
            "input": {
              "name": "[concat(parameters('agentNamePrefix'), padLeft(copyIndex('containers', 1), 3, '0'))]",
              "properties": {
                "image": "[parameters('registryImageUri')]",
                "environmentVariables": [
                  {
                    "name": "VSTS_AGENT_INPUT_URL",
                    "value": "[parameters('azdUrl')]"
                  },
                  {
                    "name": "VSTS_AGENT_INPUT_AUTH",
                    "value": "pat"
                  },
                  {
                    "name": "VSTS_AGENT_INPUT_TOKEN",
                    "secureValue": "[parameters('patToken')]"
                  },
                  {
                    "name": "VSTS_AGENT_INPUT_POOL",
                    "value": "[parameters('agentPool')]"
                  },
                  {
                    "name": "VSTS_AGENT_INPUT_AGENT",
                    "value": "[concat(parameters('agentNamePrefix'), padLeft(copyIndex('containers', 1), 3, '0'))]"
                  }
                ],
                "resources": {
                  "requests": {
                    "memoryInGB": "[parameters('memoryInGb')]",
                    "cpu": "[parameters('cpuCount')]"
                  }
                }
              }
            }
          }
        ],
        "imageRegistryCredentials": [
          {
            "server": "[parameters('registryLoginServer')]",
            "username": "[parameters('registryUserName')]",
            "password": "[parameters('registryPassword')]"
          }
        ],
        "restartPolicy": "[parameters('restartPolicy')]",
        "osType": "[parameters('osType')]"
      }
    }
  ],
  "outputs": {
    "principalId": {
      "condition": "[parameters('assignManagedIdentity')]",
      "type": "string",
      "value": "[reference(concat(resourceId('Microsoft.ContainerInstance/containerGroups/', parameters('containerInstanceName')), '/providers/Microsoft.ManagedIdentity/Identities/default'), '2015-08-31-preview').principalId]"
    }
  }
}

I’ve created some param files to go with the template:

Linux:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "containerInstanceName": {
      "value": "<instanceName>"
    },
    "containerCount": {
      "value": 1
    },
    "azdUrl": {
      "value": "https://dev.azure.com/<MyOrg>"
    },
    "agentPool": {
      "value": "<myagentpool>"
    },
    "agentNamePrefix": {
      "value": "aci-agent-ubuntu-"
    },
    "registryLoginServer": {
      "value": "<myregistry>.azurecr.io"
    },
    "registryImageUri": {
      "value": "<myregistry>.azurecr.io/azd/azd-ubuntu1804-pwsh6.2.0:latest"
    }
  }
}

Windows:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "containerInstanceName": {
      "value": "<instanceName>"
    },
    "containerCount": {
      "value": 1
    },
    "azdUrl": {
      "value": "https://dev.azure.com/<MyOrg>"
    },
    "agentPool": {
      "value": "<myagentpool>"
    },
    "agentNamePrefix": {
      "value": "aci-agent-windows-"
    },
    "registryLoginServer": {
      "value": "<myregistry>.azurecr.io"
    },
    "registryImageUri": {
      "value": "<myregistry>.azurecr.io/azd/azd-windows1809-pwsh6.2.0:latest"
    },
    "osType": {
      "value": "Windows"
    }
  }
}

Replace all values that have <> with your own.

Some arguments have to be provided securely. Deploy the template using PowerShell:

# get the pat token from the user and store as secureString
$patToken = Read-Host -AsSecureString -Prompt patToken

# get registry credentials
$registry = Get-AzContainerRegistry -Name <myregistry> -ResourceGroupName <resourcegroup>
$registryCred = $registry | Get-AzContainerRegistryCredential
$secureRegistryPassword = ConvertTo-SecureString -String $registryCred.Password -AsPlainText -Force

# deploy container instance
New-AzResourceGroupDeployment -ResourceGroupName <resourcegroup> -TemplateFile ./mytemplate.json -TemplateParameterFile ./azd-windows-containerinstance.parameters.json -registryUserName $registryCred.Username -registryPassword $secureRegistryPassword -patToken $patToken

ACI caveats

Some caveats I’ve found when working with ACI:

Note you can enable Managed Identity by setting the assignManagedIdentity parameter to true. The template will output the principalId of the resulting identity so you can assign it a Role on a certain scope (subscription / resource group). Note that Managed Identity for ACI is in preview at this time.
Preliminary tests with Managed Identity did not work. The metadata endpoint http://169.254.169.254 was not available from within the container
Managed Identity is not available at this time on Windows Container Instances.
Windows Container Groups can only contain 1 container. Old issue but still the case. The template enforces a count of 1 in case of Windows instead of throwing an error. Once this limitation is lifted, the template if condition can be removed.
Increasing the amount of containers post deployment by redeploying the template does not work. You need to delete / create a new Container Instance to do this.

Using the new Agents

To use the new Agents, select the correct Agent Pool in the build / release / pipeline (the configuration is part of the Agent Job). Add a demand for Agent.OS with either Windows_NT or Linux if both agent types exist in the same pool.

Azure ARM Linked Templates and Complete mode

Tue, 29 Jan 2019 12:00:00 +0000

Just a small blog post on Azure ARM Linked Template deployment and Complete mode since I couldn’t find a satisfying answer quickly enough via my favorite search engine.

TL;DR: Linked Template resources together with the master template resources are deployed / kept / updated. Resources out of the cumulative result of master + linked templates are deleted.

Today I was wondering what would happen if I had a “master” template being deployed in complete mode, what would happen if it had Linked template deployments.

I have this sample “master” deployment template:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {},
    "variables": {},
    "resources": [
        {
            "type": "Microsoft.Storage/storageAccounts",
            "apiVersion": "2018-07-01",
            "name": "bgelensst01",
            "location": "[resourceGroup().location]",
            "sku": {
                "name":"Standard_LRS"
            }
        },
        {
            "name": "createsecond",
            "type": "Microsoft.Resources/deployments",
            "apiVersion": "2015-01-01",
            "properties": {
                "mode": "Incremental",
                "templateLink": {
                    "uri": "https://boguslink/storage.json",
                    "contentVersion": "1.0.0.0"
                }
            }
        }
    ],
    "outputs": {}
}

Note that it is only supported to have Incremental mode specified for Linked deployements.

The linked storage.json looks like this:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {},
    "variables": {},
    "resources": [
        {
            "type": "Microsoft.Storage/storageAccounts",
            "apiVersion": "2018-07-01",
            "name": "bgelensst02",
            "location": "[resourceGroup().location]",
            "sku": {
                "name":"Standard_LRS"
            }
        }
    ],
    "outputs": {}
}

I created a resource group called complete and deployed the template to it using PowerShell:

New-AzResourceGroupDeployment -ResourceGroupName complete -TemplateFile .\azuredeploy.1.json -Mode Complete -Force

The result is 2 Storage Accounts. So no problem there even though you might think to have a conflicting situation with the Linked deployment being in incremental mode.

Now let’s add another resource to the resource group which is not specified in one of templates.

I’ve added a NSG. Let’s do another deployment of the master template in complete mode and check the result.

New-AzResourceGroupDeployment -ResourceGroupName complete -TemplateFile .\azuredeploy.1.json -Mode Complete -Force

The NSG is removed but the storage accounts remained. When checking the related events we find the delete operation of the NSG.

To summarize, Linked Template resources together with the master template resources are deployed / kept / updated. Resources out of the cumulative result of master + linked templates are deleted.

DSC Pull Server reloaded. Part 5: Containers!

Mon, 28 Jan 2019 12:00:00 +0000

With Windows Server 2016, container support was introduced. Now that the Pull Server supports SQL, it felt naturally to me to try and see if the Pull Server could be containerized. A containerized Pull Server allows us to run a Pull Server anywhere where Windows Server Containers are supported (also on Windows 10!). Let’s investigate a prototype!

On PSConfAsia and PSConfEU I demoed the Pull Server running in containers. Please see the video section for the session recordings.

Index:

Part 1: SQL Support
Part 2: Managing the Pull Server with DSCPullServerAdmin
Part 3: Pre-create the Pull Server database
Part 4: Migrate EDB or MDB to SQL
Part 5: Containers! (this post)

Environment

For this blog post I’m running everything on my laptop which is running Windows 10. I installed Hyper-V and the Containers feature as well as Docker for Desktop. Docker for Desktop is running in Windows Containers mode. To get started please follow the Microsoft documentation.

I’ve pre-pulled the Windows Server Core 2019 Windows Container image that I’ll use for building my Pull Server image.

docker pull mcr.microsoft.com/windows/servercore:ltsc2019

Docker project

I’ve created a folder containing a couple of files to setup my container image.

DockerFile

The DockerFile contains the build instructions for the Docker Image.

FROM mcr.microsoft.com/windows/servercore:ltsc2019
LABEL maintainer="ben@bgelens.nl"
COPY [ "Docker.ps1", "./Docker.ps1" ]
COPY [ "DockerMon.ps1", "./DockerMon.ps1" ]
RUN powershell.exe -Command .\Docker.ps1
EXPOSE 8080
CMD ["powershell.exe", "c:\\DockerMon.ps1"]

As you can see, we are basing on the ltsc2019 version of the servercore image. We are copying the other files from the directory. The Docker.ps1 script is executed and finally some metadata is added to the resulting image, telling you that it is expected to bind to the container local port 8080 and when running a container based on this image, the DockerMon.ps1 script is executed.

Docker.ps1

The Docker.ps1 script contains all the setup logic for the container image.

$null = Install-WindowsFeature DSC-Service
$null = Install-PackageProvider -Name PowerShellGet -ForceBootstrap -Force
$null = Install-Module -Name xPSDesiredStateConfiguration -RequiredVersion 8.4.0.0 -Force
$null = Invoke-DscResource -ModuleName xPSDesiredStateConfiguration -Name xDscWebService -Method Set -Property @{
    Ensure = 'Present'
    EndpointName = 'PSDSCPullServer'
    Port = 8080
    PhysicalPath = "$env:SystemDrive\inetpub\PSDSCPullServer"
    CertificateThumbPrint='AllowUnencryptedTraffic'
    ModulePath="$env:PROGRAMFILES\WindowsPowerShell\DscService\Modules"
    ConfigurationPath="$env:PROGRAMFILES\WindowsPowerShell\DscService\Configuration"
    State='Started'
    RegistrationKeyPath="$env:PROGRAMFILES\WindowsPowerShell\DscService"
    AcceptSelfSignedCertificates=$true
    UseSecurityBestPractices=$false
    SqlProvider=$true
    SqlConnectionString='#CONNECTIONSTRING#'
}
$null = Remove-Website -Name 'Default Web Site'
$null = Stop-Website -Name PSDSCPullServer

As you can see, the DSC-Service feature is installed, the xPSDesiredStateConfiguration module is installed and the xDscWebService DSC resource is executed to setup the Pull Server. A placeholder is used for the Sql Connection String. Finally the Pull Server is stopped as when we start a new container based on the image we are producing, we want the web.config changed to be applied from the start.

DockerMon.ps1

The DockerMon.ps1 script contains the logic to start the container using the correct specified settings (passed in via Environmental variables) and verify that the container is operating properly (or else terminating it).

$configFile = 'C:\inetpub\PSDSCPullServer\web.config'
$connString = $env:ConnectionString
if ($null -eq $connString) {
    # could also result in edb usage instead of sql so container can run in solo mode
    Write-Error -Message 'No ConnectionString found in Environment variables' -ErrorAction Stop
}
$webConfig = Get-Content -Path $configFile
$webConfig = $webConfig.replace('#CONNECTIONSTRING#', $connString)
if (Test-Path -Path c:\pullserver) {
    # if it doesn't keep defaults to at least be able to run
    $webConfig = $webConfig.replace('C:\Program Files\WindowsPowerShell\DscService', 'c:\pullserver')
}
$webConfig | Set-Content $configFile

Start-Website -Name PSDSCPullServer

function Monitor {
    $irmArgs = @{
        Headers = @{
            Accept = 'application/json'
            ProtocolVersion = '2.0'
        }
        UseBasicParsing = $true
        Uri = 'http://localhost:8080/PSDSCPullServer.svc'
        ErrorAction = 'Stop'
    }
    try {
        $null = Invoke-RestMethod @irmArgs
        "$([datetime]::UtcNow.ToString()) - Pull Server is responding"
        Start-Sleep -Seconds 10
        Monitor
    } catch {
        throw $_
    }
}

Monitor

As you can see, when no ConnectionString is found in the Environment variables, the container is terminated immediately. When it is found, the ConnectionString placeholder in the web.config file is replaced with the desired value.

When the folder c:\pullserver is found, all references in the web.config file that refer to C:\Program Files\WindowsPowerShell\DscService are replaced with c:\pullserver. For this prototype it is expected that a volume is mounted to the container (for persistent storage) containing the configurations, modules and registrationkeys.

The Pull Server website is started.

A self invoking monitor function is started that checks every 10 seconds if the internal Pull Server endpoint is responding (the Pull Server will throw 500 errors when the Database is not reachable for whatever reason). When something is wrong, the process is terminated resulting in a stopped container. In a more advanced usage scenario (e.g. Swarm or Kubernetes) this will result in the scheduler to start a new instance of the container knowing something has gone wrong.

Building the image

Now we have seen the files involved, we can start building the container image.

docker build . -t pullserver:latest

Build was successful. Let’s see if it’s available:

Running a Pull Server Container

To run the container we need a database pre created (or an existing one), and a directory structure containing configuration files, module archives, checksums and registrationkey files.

When setup correctly you can see we could scale out to multiple container relatively easily. The database already is easily accessible and sharable by multiple container instances and the directory structure could be made highly available as well which allows you to map it into multiple containers.

Prerequisites

I have created a new database as Azure SQL database already by using DSCPullServerAdmin Module. See part 3 of this series for the instructions on how to do this.

The directory which I will map into the container looks like this:

I just used a simple configuration to generate a mof. The configuration relies on a DSC Resource Module, so I included that one as well. I won’t go into detail for the configuration as it’s not really important for this blog post.

Starting and checking the container

We now start a container based from the image passing in what is needed.

docker run `
    -p 8080:8080 `
    -v C:\pullserver:C:\pullserver `
    -e ConnectionString="Provider=SQLOLEDB.1;Server=dscpullsrv.database.windows.net;Database=pullsrvdb;User ID=myUserName;Password=myPassword;" `
    -d pullserver

A couple of things to note here:

we bound port 8080 of the local machine to port 8080 of the container by specifying -p 8080:8080
we mapped the directory c:\pullserver to the container path c:\pullserver by specifying -v C:\pullserver:C:\pullserver
we passed in the environmental variable ConnectionString by specifying -e ConnectionString="Provider=SQLOLEDB.1;Server=dscpullsrv.database.windows.net;Database=pullsrvdb;User ID=myUserName;Password=myPassword;"
we specified that the container should be running detached from the terminal by specifying -d

Let’s see if the container is still running.

docker ps

So it is running, we could inspect the logs to see if we get some expected output there.

docker logs 96

So it seems the Pull Server is operating in the container! Let’s onboard a machine to it to see if registration will work out as well.

For me, the localhost mapping did not result in connectivity to the container. As I’m running an edge build of Docker for Desktop and a Windows Insider build it’s probably something specific to my setup. To work around the issue, I’ll connect the LCM using the container IP instead of localhost. You can find the IP address by using docker inspect or docker exec.

[dsclocalconfigurationmanager()]
configuration lcm {
    Settings {
        RefreshMode = 'Pull'
    }

    ConfigurationRepositoryWeb SQLPullWeb {
        ServerURL = 'http://172.17.18.205:8080/PSDSCPullServer.svc'
        RegistrationKey = 'cb30127b-4b66-4f83-b207-c4801fb05087'
        AllowUnsecureConnection = $true
    }
}

lcm
Set-DscLocalConfigurationManager .\lcm -Verbose

Onboarding worked as expected! Now let’s see if we can pull the configuration.

Awesome! Prototype successful!

If you want to cleanup your running Pull Server container run: docker rm -f (docker ps -aq)

New Terraform Azure Automation Resources

Sun, 27 Jan 2019 12:00:00 +0000

Some time ago I contributed 3 new Terraform resources to the AzureRM Resource Provider:

This was my first ever contribution to a Go project so I’m very proud this got merged!

These 3 resources where the missing pin (for me at least) to have an end-2-end scenario enabled where pre-compiled DSC mof files and DSC resource modules could be send to an Azure Automation Account. This enables Terraform VM deployments including Guest state configuration assignment handled through Azure Automation.

In this blog post I’ll demonstrate the use of these resources. I won’t go into detail on how to get started with Terraform and such as there is plenty of info available already.

I’m using Terraform 0.11.11 and I have added it to my path so it’s available from any folder I’m in. I’m using the azurerm provider version 1.21.

I’ve defined and applied a starter Terraform file and I’ll work from there. I’m using an SPN for authentication and have added the variable values to my environmental variables before I ran this.

variable "ARM_CLIENT_ID" {}
variable "ARM_CLIENT_SECRET" {}
variable "ARM_SUBSCRIPTION_ID" {}
variable "ARM_TENANT_ID" {}

provider "azurerm" {
  version         = "1.21"
  client_id       = "${var.ARM_CLIENT_ID}"
  client_secret   = "${var.ARM_CLIENT_SECRET}"
  subscription_id = "${var.ARM_SUBSCRIPTION_ID}"
  tenant_id       = "${var.ARM_TENANT_ID}"
}

resource "azurerm_resource_group" "aadscrg" {
  name     = "aadscrg"
  location = "westeurope"
}

resource "azurerm_automation_account" "aa" {
  name                = "aadsc01"
  location            = "${azurerm_resource_group.aadscrg.location}"
  resource_group_name = "${azurerm_resource_group.aadscrg.name}"

  sku {
    name = "Basic"
  }
}

For this blog post I’m keeping things simple as I’m just illustrating the use of the Terraform resources. We’ll be using the following DSC configuration:

configuration timezone {
    Import-DscResource -ModuleName ComputerManagementDsc -ModuleVersion 6.1.0.0

    TimeZone westeustandard {
        IsSingleInstance = 'Yes'
        TimeZone = 'W. Europe Standard Time'
    }
}

So we know we need ComputerManagementDsc DSC Resource module with version 6.1.0.0 to be available in Azure Automation as a Module asset so that LCMs can download this module when they don’t have it already. To do this we need to add the Terraform AzureRM resource azurerm_automation_module to our Terraform file.

The resource needs a download link where the zip or nupkg is available. You could of course publish a module to an Azure Storage Account and pass the link to the blob to this resource but in this case, the Module is available on the PowerShell Gallery. We only need to find the download link for it. So let’s find the download link for ComputerManagementDsc Resource Module of verion 6.1.0.0. As far as I know you cannot use Find-Module to fetch the download url so we need to query the Gallery ourselves.

$moduleInfo = Invoke-RestMethod -Uri "https://www.powershellgallery.com/api/v2/Packages?`$filter=Id eq 'ComputerManagementDsc' and Version eq '6.1.0.0'"
$moduleInfo.content.src

We get the following link: https://www.powershellgallery.com/api/v2/package/ComputerManagementDsc/6.1.0. The link is not a direct link to the package artifact itself but it is good enough for the resource. By using this link, we let the PowerShell Gallery be responsible for providing the resource with the correct link. This is good as the storage locations might change over time.

Let’s update the Terraform file with the resource (in the end of the post, I’ll have a complete example).

resource "azurerm_automation_module" "computermanagementdsc" {
  name                    = "ComputerManagementDsc"
  resource_group_name     = "${azurerm_resource_group.aadscrg.name}"
  automation_account_name = "${azurerm_automation_account.aa.name}"

  module_link = {
    uri = "https://www.powershellgallery.com/api/v2/package/ComputerManagementDsc/6.1.0"
  }
}

Now we need the configuration to be compiled and we are going to read it using the Terraform file function. First let’s compile it (Note that I’m compiling it in directory where the Terraform file is located, resulting in a sub-directory called timezone and containing a file called localhost.mof).

configuration timezone {
    Import-DscResource -ModuleName ComputerManagementDsc -ModuleVersion 6.1.0.0

    TimeZone westeustandard {
        IsSingleInstance = 'Yes'
        TimeZone = 'W. Europe Standard Time'
    }
}
timezone

I’m compiling my DSC Configuration using PowerShell 6.1.2 resulting in an UTF-8 encoded mof file. When running this on PowerShell 5.1 you’ll end up with a UTF-16 LE encoded file which needs to be converted to UTF-8 for the Terraform file function to be able to read the file.

Now we need to add 2 resources to the Terraform file:

azurerm_automation_dsc_configuration
azurerm_automation_dsc_nodeconfiguration

The reason we need both is that no mof file can be uploaded to Azure Automation without a corresponding configuration to be already present for it (even an empty one). This need is abstracted away from you when you use the Azure portal or other tooling but when dealing with the REST api directly you’ll quickly find it out).

Let’s add the resources:

resource "azurerm_automation_dsc_configuration" "timezone" {
  name                    = "timezone"
  resource_group_name     = "${azurerm_resource_group.aadscrg.name}"
  automation_account_name = "${azurerm_automation_account.aa.name}"
  location                = "${azurerm_automation_account.aa.location}"
  content_embedded        = "configuration timezone {}"
}

resource "azurerm_automation_dsc_nodeconfiguration" "timezone" {
  name                    = "timezone.localhost"
  resource_group_name     = "${azurerm_resource_group.aadscrg.name}"
  automation_account_name = "${azurerm_automation_account.aa.name}"
  depends_on              = ["azurerm_automation_dsc_configuration.timezone"]
  content_embedded        = "${file("${path.cwd}/timezone/localhost.mof")}"
}

Note the explicit depends_on at the azurerm_automation_dsc_nodeconfiguration. This is because terraform will not see the dependency itself.

The Terraform file is now complete, let’s see it in full, plan and apply.

variable "ARM_CLIENT_ID" {}
variable "ARM_CLIENT_SECRET" {}
variable "ARM_SUBSCRIPTION_ID" {}
variable "ARM_TENANT_ID" {}

provider "azurerm" {
  version         = "1.21"
  client_id       = "${var.ARM_CLIENT_ID}"
  client_secret   = "${var.ARM_CLIENT_SECRET}"
  subscription_id = "${var.ARM_SUBSCRIPTION_ID}"
  tenant_id       = "${var.ARM_TENANT_ID}"
}

resource "azurerm_resource_group" "aadscrg" {
  name     = "aadscrg"
  location = "westeurope"
}

resource "azurerm_automation_account" "aa" {
  name                = "aadsc01"
  location            = "${azurerm_resource_group.aadscrg.location}"
  resource_group_name = "${azurerm_resource_group.aadscrg.name}"

  sku {
    name = "Basic"
  }
}

resource "azurerm_automation_module" "computermanagementdsc" {
  name                    = "ComputerManagementDsc"
  resource_group_name     = "${azurerm_resource_group.aadscrg.name}"
  automation_account_name = "${azurerm_automation_account.aa.name}"

  module_link = {
    uri = "https://www.powershellgallery.com/api/v2/package/ComputerManagementDsc/6.1.0"
  }
}

resource "azurerm_automation_dsc_configuration" "timezone" {
  name                    = "timezone"
  resource_group_name     = "${azurerm_resource_group.aadscrg.name}"
  automation_account_name = "${azurerm_automation_account.aa.name}"
  location                = "${azurerm_automation_account.aa.location}"
  content_embedded        = "configuration timezone {}"
}

resource "azurerm_automation_dsc_nodeconfiguration" "timezone" {
  name                    = "timezone.localhost"
  resource_group_name     = "${azurerm_resource_group.aadscrg.name}"
  automation_account_name = "${azurerm_automation_account.aa.name}"
  depends_on              = ["azurerm_automation_dsc_configuration.timezone"]
  content_embedded        = "${file("${path.cwd}/timezone/localhost.mof")}"
}

First we run terraform init (note that the environment variables for the azurerm provider are already set):

Now we can run terraform plan:

We see the resources that will be created when we would run a terraform apply. Let’s do that:

Everything is now in place for a VM to onboard and have this configuration assigned!

DSC Pull Server reloaded. Part 4: Migrate EDB or MDB to SQL

Wed, 16 Jan 2019 12:00:00 +0000

In the last post, we have seen how to use pre-create the Pull Server Database as an Azure SQL Database. It’s always nice to be able to start fresh but a lot of you already deployed Pull Servers based on WMF5+ backed by an EDB or MDB database and you want to move forward right? Now it’s time to figure out how to “migrate” your existing Pull Servers to a new SQL backed Pull Server. The key here is to again use DSCPullServerAdmin to make your life easy!

If you haven’t already:

Install-Module DSCPullServerAdmin -Verbose -Force -Scope CurrentUser

Index:

Part 1: SQL Support
Part 2: Managing the Pull Server with DSCPullServerAdmin
Part 3: Pre-create the Pull Server database
Part 4: Migrate EDB or MDB to SQL (this post)
Part 5: Containers!

Migration process

In general the migration from EDB or MDB to SQL follows these steps:

Install one or more Pull Servers backed by a SQL Database (with multiple, have a load balancer in front off them).
Copy all configurations, checksums, DSC Resource modules to the new Pull Server(s).
Stop the EDB / MDB based Pull Server.
Copy data from the EDB / MDB database into the new SQL Database.
Update the DNS record pointing at the old Pull Server to point to the new Pull Server(s) / load balancer

It’s good to know that LCMs only use the Registration Key during initial onboarding to a Pull Server and from there on use their AgentId to communicate with the Pull Server. So once an LCM is onboarded, there need to be a record for it in the Database with the corresponding AgentId. If this is the case, the LCM is free to communicate with the Pull Server. Knowing this, you could go in the SQL Database and create the records manually. You could also re-register all your LCMs with the new Pull Server or… You can use DSCPullServerAdmin (I’m using version 0.4.2 for this blog post) to pull data out of the old EDB / MDB database and inject it into the new SQL Database in a single function call!

You could potentially move the other way around as well (SQL to EDB, or EDB to MDB, or… ). Since DSCPullServerAdmin version 0.3 EDB is fully supported (before only Get operations where available, but now all the other operations like new, set and remove are available as well) and since version 0.4 MDB support is added with full functionality as well!

For this blog post we’ll handle the scenario EDB to SQL as it will be (I expect) the most common scenario. I’ll also be targeting the same server to keep things simple.

At PSConfAsia 2018 I spoke about this subject in detail so if you like to watch a video instead of reading, please check my video section!

Existing Pull Server

The existing EDB Pull Server is configured using the following configuration:

configuration PullServerEDB {
    Import-DscResource -ModuleName PSDesiredStateConfiguration -ModuleVersion 1.1
    Import-DscResource -ModuleName xPSDesiredStateConfiguration -ModuleVersion 8.4.0.0

    WindowsFeature dscservice {
        Name   = 'Dsc-Service'
        Ensure = 'Present'
    }

    File PullServerFiles {
        DestinationPath = 'c:\pullserver'
        Ensure = 'Present'
        Type = 'Directory'
        Force = $true
    }

    xDscWebService PSDSCPullServer {
        Ensure                       = 'Present'
        EndpointName                 = 'PSDSCPullServer'
        Port                         = 8080
        PhysicalPath                 = "$env:SystemDrive\inetpub\PSDSCPullServer"
        CertificateThumbPrint        = 'AllowUnencryptedTraffic'
        ModulePath                   = "c:\pullserver\Modules"
        ConfigurationPath            = "c:\pullserver\Configuration"
        DatabasePath                 = "c:\pullserver" 
        State                        = 'Started'
        RegistrationKeyPath          = "c:\pullserver"
        UseSecurityBestPractices     = $false
        DependsOn                    = '[File]PullServerFiles', '[WindowsFeature]dscservice'
    }

    File RegistrationKeyFile {
        Ensure          = 'Present'
        Type            = 'File'
        DestinationPath = "c:\pullserver\RegistrationKeys.txt"
        Contents        = 'cb30127b-4b66-4f83-b207-c4801fb05087'
        DependsOn       = '[File]PullServerFiles'
    }
}

Let’s check the web.config for the pull server:

([xml](Get-Content -Path C:\inetpub\PSDSCPullServer\web.config)).configuration.appsettings.GetEnumerator()

Managed Node

I have one managed node which is currently onboarded and able to communicate with the EDB Pull Server.

The meta configuration looks like this:

[dsclocalconfigurationmanager()]
configuration lcm {
    Settings {
        RefreshMode = 'Pull'
    }

    ConfigurationRepositoryWeb SQLPullWeb {
        ServerURL = 'http://pullserver:8080/PSDSCPullServer.svc'
        RegistrationKey = 'cb30127b-4b66-4f83-b207-c4801fb05087'
        AllowUnsecureConnection = $true
        ConfigurationNames = 'MySuperServer'
    }

    ReportServerWeb SQLPullWeb {
        ServerURL = 'http://pullserver:8080/PSDSCPullServer.svc'
        RegistrationKey = 'cb30127b-4b66-4f83-b207-c4801fb05087'
        AllowUnsecureConnection = $true
    }
}

And to see that the LCM is able to talk to the Pull Server:

Get data from EDB

Back on the Pull Server, we can check the content of the database using DSCPullServerAdmin module.

New-DSCPullServerAdminConnection -ESEFilePath C:\pullserver\Devices.edb

Ouch, this is one of the reasons we need to stop the Pull Server to handle the migration. EDB is only accessible by one process at a time (this is also the reason the EDB based Pull Server is not scalable!).

Let’s stop the Pull Server and try again.

iisreset /stop
New-DSCPullServerAdminConnection -ESEFilePath C:\pullserver\Devices.edb
Get-DSCPullServerAdminRegistration

Great we have access!

Destroy EDB Pull Server

Normally you would not have to do this but in my case, since I’m deploying the SQL backed Pull Server on the same VM as the EDB, I need to destroy the current Pull Server.

configuration PullServerCleanup {
    Import-DscResource -ModuleName PSDesiredStateConfiguration -ModuleVersion 1.1
    Import-DscResource -ModuleName xPSDesiredStateConfiguration -ModuleVersion 8.4.0.0

    xDscWebService PSDSCPullServer {
        Ensure                       = 'Absent'
        EndpointName                 = 'PSDSCPullServer'
        UseSecurityBestPractices     = $false
        CertificateThumbPrint        = 'AllowUnencryptedTraffic'
    }
}

PullServerCleanup

Start-DscConfiguration -Wait -Verbose .\PullServerCleanup -Force
Remove-WebAppPool -Name PSWS

Create the new SQL backed Pull Server

On my server, SQL is installed locally already. I’m using the same configuration as in the first part of this blog series:

configuration PullServerSQL {
    Import-DscResource -ModuleName PSDesiredStateConfiguration -ModuleVersion 1.1
    Import-DscResource -ModuleName xPSDesiredStateConfiguration -ModuleVersion 8.4.0.0

    WindowsFeature dscservice {
        Name   = 'Dsc-Service'
        Ensure = 'Present'
    }

    File PullServerFiles {
        DestinationPath = 'c:\pullserver'
        Ensure = 'Present'
        Type = 'Directory'
        Force = $true
    }

    xDscWebService PSDSCPullServer {
        Ensure                       = 'Present'
        EndpointName                 = 'PSDSCPullServer'
        Port                         = 8080
        PhysicalPath                 = "$env:SystemDrive\inetpub\PSDSCPullServer"
        CertificateThumbPrint        = 'AllowUnencryptedTraffic'
        ModulePath                   = "c:\pullserver\Modules"
        ConfigurationPath            = "c:\pullserver\Configuration"
        State                        = 'Started'
        RegistrationKeyPath          = "c:\pullserver"
        UseSecurityBestPractices     = $false
        SqlProvider                  = $true
        SqlConnectionString          = 'Provider=SQLOLEDB.1;Server=localhost;Database=DemoDSC;User ID=SA;Password=Welkom01;Initial Catalog=master;'
        DependsOn                    = '[File]PullServerFiles', '[WindowsFeature]dscservice'
    }

    File RegistrationKeyFile {
        Ensure          = 'Present'
        Type            = 'File'
        DestinationPath = "c:\pullserver\RegistrationKeys.txt"
        Contents        = 'cb30127b-4b66-4f83-b207-c4801fb05087'
        DependsOn       = '[File]PullServerFiles'
    }
}

After setting it up, let’s quickly validate it is working.

Great! If you need more details about this setup, please check the first blog post.

See that Managed node is not able to communicate

On the managed node, let’s run Update-DscConfiguration -Wait -Verbose.

We see this now results in a 404 error, meaning the Pull Server has no knowledge of this node. Time to fix this.

Migration time

Let’s create a connection with both the EDB and the new SQL database.

$edb = New-DSCPullServerAdminConnection -ESEFilePath C:\pullserver\Devices.edb
$sql = New-DSCPullServerAdminConnection -SQLServer localhost -Database DemoDSC

# see impact
Copy-DSCPullServerAdminData -Connection1 $edb -Connection2 $sql -ObjectsToMigrate RegistrationData -WhatIf

# migrate data
Copy-DSCPullServerAdminData -Connection1 $edb -Connection2 $sql -ObjectsToMigrate RegistrationData

# get data from sql
Get-DSCPullServerAdminRegistration -Connection $sql

In the above example I’ve chosen to only migrate registration data. But you could easily migrate StatusReports and Devices as well by adding on to the ObjectsToMigrate parameter values.

Note that Copy-DSCPullServerAdminData has Connection1 and Connection2 parameters. It used to be only possible to migrate from EDB to SQL but since version 0.3 and 0.4 it is also possible to migratie from SQL to EDB, EDB to EDB, MDB to SQL, SQL to MDB, EDB to MDB, SQL to SQL and so on! Whatever you fancy.

I used the same directory for Configuration and Module artifacts so no need to migrate those. Keep in mind if you are moving stuff to a new server or to a farm that you need to move these items as well (potentially to some shared location in case of an HA setup).

Let’s check the LCM again.

Great! We now have out new Pull Server operational!

Finally let’s move over the status reports:

# see impact
Copy-DSCPullServerAdminData -Connection1 $edb -Connection2 $sql -ObjectsToMigrate StatusReports -WhatIf

# migrate data
Copy-DSCPullServerAdminData -Connection1 $edb -Connection2 $sql -ObjectsToMigrate StatusReports

# get data from sql
Get-DSCPullServerAdminStatusReport -Connection $sql

And see we could also use the Pull Server Rest API to get the data returned.

On the Managed node run:

$agentId = (Get-DscLocalConfigurationManager).AgentId
$irmArgs = @{
    Uri = "http://pullserver:8080/PSDSCPullServer.svc/Nodes(AgentId = '$agentId')/Reports"
    Headers = @{
        Accept = 'application/json'
        ProtocolVersion = '2.0'
    }
    UseBasicParsing = $true
}

$reports = (Invoke-RestMethod @irmArgs).value
$reports[0]

As you can see, this still works.

That is it for this post. To be continued!

DSC Pull Server reloaded. Part 3: Pre-create the Pull Server Database

Thu, 01 Nov 2018 12:00:00 +0000

In the last post, we have seen how to use some of the functions of DSCPullServerAdmin to make your life easier! In this post, we’ll look at another function from this module which allows you to pre-create the Pull Server Database. This is nice if you have a SQL Server where you don’t have the correct permissions to setup new databases (but are allowed to connect to a database of course) and somebody else needs to create it, or where you are trying to host the Database in a service like Azure SQL where the Pull Server itself is not capable of creating the Database due the inability to use the initial catalog in the connection string.

In this post we’ll use Azure SQL as our target Database platform.

Index:

Part 1: SQL Support
Part 2: Managing the Pull Server with DSCPullServerAdmin
Part 3: Pre-create the Pull Server database (this post)
Part 4: Migrate EDB or MDB to SQL
Part 5: Containers!

Creating Azure SQL DSC Pull Server Database

So let’s create an Azure SQL Server and Database. I’m using the AZ PowerShell Modules (version 0.4.0 at the time of writing).

# create a new ResourceGroup
$rg = New-AzResourceGroup -Name pullsrv -Location 'West Europe'

# prompt for the SQL login credentials
$sqlCred = Get-Credential -Message 'Provide SQL login Credentials'

# create a new Azure SQL Server
$sqlServer = $rg | New-AzSqlServer -ServerName pullsrv -SqlAdministratorCredentials $sqlCred -Location $rg.Location

# add your public ip address to the Azure SQL firewall to allow connections and allow all connections from Azure ips
$myPublicIp = (Invoke-RestMethod -Uri http://ipinfo.io/json -UseBasicParsing).ip
$null = $sqlServer | New-AzSqlServerFirewallRule -FirewallRuleName home -StartIpAddress $myPublicIp -EndIpAddress $myPublicIp 
$null = $sqlServer | New-AzSqlServerFirewallRule -AllowAllAzureIPs

# use DSCPullServerAdmin module to pre-create the database
New-DSCPullServerAdminSQLDatabase -SQLServer $sqlServer.FullyQualifiedDomainName -Credential $sqlCred -Name pullsrvdb -Confirm:$false

# print out the connection string
'Provider=SQLOLEDB.1;Server={0};Database={1};User ID={2};Password={3};' -f @(
    $sqlServer.FullyQualifiedDomainName,
    'pullsrvdb',
    $sqlCred.UserName,
    $sqlCred.GetNetworkCredential().Password
)

For me the above script generated the connection string:

Provider=SQLOLEDB.1;Server=pullsrv.database.windows.net;Database=pullsrvdb;User ID=myUserName;Password=myPassword;

Create a new Pull Server backed by Azure SQL Database

Now I have setup the Database in Azure we can install the Pull Server.

configuration PullServerSQL {
    param (
        [Parameter(Mandatory)]
        [ValidateNotNullOrEmpty()]
        [string] $ConnectionString
    )
    Import-DscResource -ModuleName PSDesiredStateConfiguration -ModuleVersion 1.1
    Import-DscResource -ModuleName xPSDesiredStateConfiguration -ModuleVersion 8.4.0.0

    WindowsFeature dscservice {
        Name   = 'Dsc-Service'
        Ensure = 'Present'
    }

    File PullServerFiles {
        DestinationPath = 'c:\pullserver'
        Ensure = 'Present'
        Type = 'Directory'
        Force = $true
    }

    xDscWebService PSDSCPullServer {
        Ensure                       = 'Present'
        EndpointName                 = 'PSDSCPullServer'
        Port                         = 8080
        PhysicalPath                 = "$env:SystemDrive\inetpub\PSDSCPullServer"
        CertificateThumbPrint        = 'AllowUnencryptedTraffic'
        ModulePath                   = "c:\pullserver\Modules"
        ConfigurationPath            = "c:\pullserver\Configuration"
        State                        = 'Started'
        RegistrationKeyPath          = "c:\pullserver"
        UseSecurityBestPractices     = $false
        SqlProvider                  = $true
        SqlConnectionString          = $ConnectionString
        DependsOn                    = '[File]PullServerFiles', '[WindowsFeature]dscservice'
    }

    File RegistrationKeyFile {
        Ensure          = 'Present'
        Type            = 'File'
        DestinationPath = "c:\pullserver\RegistrationKeys.txt"
        Contents        = 'cb30127b-4b66-4f83-b207-c4801fb05087'
        DependsOn       = '[File]PullServerFiles'
    }
}

$connectionString = 'Provider=SQLOLEDB.1;Server=;Database=pullserverdb;User ID=myUserName;Password=myPassword;'

PullServerSQL -ConnectionString $connectionString
Start-DscConfiguration -Path .\PullServerSQL -Wait -Verbose -Force

To test if the Pull Server is working properly let’s try to communicate with it.

(Invoke-WebRequest -Uri http://localhost:8080/PSDSCPullServer.svc -UseBasicParsing).StatusCode

This should result in a statuscode of 200.

So we setup a Pull Server with Azure SQL. Let’s onboard a node.

[dsclocalconfigurationmanager()]
configuration lcm {
    Settings {
        RefreshMode = 'Pull'
    }

    ConfigurationRepositoryWeb SQLPullWeb {
        ServerURL = 'http://pullserver:8080/PSDSCPullServer.svc'
        RegistrationKey = 'cb30127b-4b66-4f83-b207-c4801fb05087'
        AllowUnsecureConnection = $true
    }

    ReportServerWeb SQLPullWeb {
        ServerURL = 'http://pullserver:8080/PSDSCPullServer.svc'
        RegistrationKey = 'cb30127b-4b66-4f83-b207-c4801fb05087'
        AllowUnsecureConnection = $true
    }
}

lcm
Set-DscLocalConfigurationManager .\lcm -Verbose

Let’s take a look in the Azure Portal to see if an entry was made. Open the Query editor in the Database blade and login.

Once logged in, type the following query and run it.

select * from RegistrationData

Great! And of course we can make use of DSCPullServerAdmin to connect to the Database as well.

New-DSCPullServerAdminConnection -SQLServer pullsrv.database.windows.net -Database pullsrvdb -Credential pullsrv

Get-DSCPullServerAdminRegistration

Get-DSCPullServerAdminStatusReport -Top 1

That is it for this post. To be continued!

DSC Pull Server reloaded. Part 2: Managing the Pull Server with DSCPullServerAdmin

Wed, 03 Oct 2018 12:00:00 +0000

In the last post, we have seen how to setup a simple DSC Pull Server backed by a SQL Database. In this blog post, I’ll expand upon that Pull Server by onboarding a new node and exploring some of the options of the DSCPullServerAdmin PowerShell module to interact with the database directly.

DSCPullServerAdmin is a project I started a while ago to interface with the PullServer edb, making several administrative tasks better achievable. Since Microsoft announced the addition of SQL support in Windows Server 1803 and 2019, it was clear to me that this would be the way forward for the Pull Server so I decided to reboot the project focusing mainly on SQL support. Migrating from edb to SQL is also one of the things handled by this project, more on this in a later post!

Index:

Part 1: SQL Support
Part 2: Managing the Pull Server with DSCPullServerAdmin (this post)
Part 3: Pre-create the Pull Server database
Part 4: Migrate EDB or MDB to SQL
Part 5: Containers!

Onboarding a new node

The onboarding experience is exactly the same as it used to be. I used the below meta configuration to onboard a node to the Pull Server setup in part 1.

[dsclocalconfigurationmanager()]
configuration lcm {
    Settings {
        RefreshMode = 'Pull'
    }

    ConfigurationRepositoryWeb SQLPullWeb {
        ServerURL = 'http://pullserver:8080/PSDSCPullServer.svc'
        RegistrationKey = 'cb30127b-4b66-4f83-b207-c4801fb05087'
        AllowUnsecureConnection = $true
    }

    ReportServerWeb SQLPullWeb {
        ServerURL = 'http://pullserver:8080/PSDSCPullServer.svc'
        RegistrationKey = 'cb30127b-4b66-4f83-b207-c4801fb05087'
        AllowUnsecureConnection = $true
    }
}

lcm
Set-DscLocalConfigurationManager .\lcm -Verbose

As you can see, I did not specify the ConfigurationNames property in the Settings block. I left that out intentionally to demonstrate we can now control that easily on the server side (we could already do this before but changing it in the edb proved to be pretty difficult).

Providing the Configuration Name in the meta configuration is still valid and I think it’s perfect for a bootstrapping scenario. It is good to know that overwriting the Configuration Name later on on the server side takes precedence on what is defined locally.

Prolonged management is, in my opinion, better handled on the server side. Going this route saves you from going into the DSC Node (or connecting remotely to it), compile a new meta mof and re-register the node just to change the Configuration Name and pull down another config. In my experience, Pull Servers are often setup when network segregation is a blocker for pushing out to VMs but pulling in from the Pull Server is allowed. Being able to change things on the Server side therefore is very handy!

Staging a configuration

Now we have a LCM registered with the server, we need to compile a configuration for it so it has something to pull down.

Lets create a simple configuration and put it in the configuration store accompanied with a checksum file.

configuration MySuperServer {
    Import-DscResource -ModuleName PSDesiredStateConfiguration -ModuleVersion 1.1

    Node MySuperServer {
        File MySuperFile {
            Ensure = 'Present'
            DestinationPath = 'C:\Windows\Temp\MySuperFile.txt'
            Contents = 'DSC ROCKS!!!'
        }
    }
}
MySuperServer -OutputPath 'c:\pullserver\Configuration'
New-DscChecksum -Path 'c:\pullserver\Configuration\MySuperServer.mof' -Force

Now it is time to assign the Configuration to the node.

DSCPullServerAdmin

To assign the configuration Server side, we could open SQL management studio and update the entry in the RegistrationData table. We could also make use of my DSCPullServerAdmin module to handle this.

Install the DSCPullServerAdmin module from the PowerShell Gallery. You can install and use it anywhere as long as the machine can communicate with the SQL backend.

Install-Module -Name DSCPullServerAdmin

At the time of writing I’m using version 0.1.5

Now let’s create a new connection with the SQL database.

New-DSCPullServerAdminConnection -SQLServer sql.mshome.net -Database DemoDSC -Credential sa

It is possible to create multiple connections to multiple Pull Servers. It is even possible to setup a connection to an edb database (more on this in a later post). Only one connection at a time will be Active though, and this active connection will be used by default. We’ll see more about connection management in a later blog post.

Now we have setup a connection with the database, we can easily enumerate all registered nodes.

Get-DSCPullServerAdminRegistration

This function allows for filtering based on AgentId and NodeName to acquire a more specific node. Just know that NodeName is not a unique value, AgentId is.

So now we figured out how to get the node object out of the database, how do we assign it the Configuration Name we desire?

Get-DSCPullServerAdminRegistration -NodeName PULLSERVER | Set-DSCPullServerAdminRegistration -ConfigurationNames 'MySuperServer'

When getting the node object again, we can see the update worked.

Let’s pull down the configuration on the node.

Update-DscConfiguration -Wait -Verbose

Isn’t that nice! The node picked up the change in Configuration Name assignment and handled it from there!

What else can we do? Look at the reports?

Get-DSCPullServerAdminStatusReport -NodeName PULLSERVER

Sure do, we can do all kinds of things using this module. I encourage you to take a look at it and give me any feedback you might have!

A couple of things I worked with until now:

Removed old reports and node registrations (Database hygiene)
Manipulated Node Registrations
Moved reports from one database into another
Migrated edb to SQL
Pre-creation of the SQL Database (on local SQL instance and Azure SQL)
Build admin REST endpoint on top of this module (Polaris)
Build a Web portal on top of this module (Universal Dashboard)

A lot of these things I will address in this blog series.

That is it for this post. To be continued!

DSC Pull Server reloaded. Part 1: SQL Support

Wed, 26 Sep 2018 12:00:00 +0000

Windows Server 1803 (Semi-Annual Channel release) and the upcoming Windows Server 2019 introduces a big change for the DSC Pull Server in the form of SQL support. Having SQL as a (supported) backend unlocks a lot of potential for the “inbox” DSC Pull Server (making it more alive then ever). In this blog post, and the following blog posts in this series, I’m exploring some of the options that came to my mind and hopefully will get you inspired as well!

Index:

Part 1: SQL Support (this post)
Part 2: Managing the Pull Server with DSCPullServerAdmin
Part 3: Pre-create the Pull Server database
Part 4: Migrate EDB or MDB to SQL
Part 5: Containers!

SQL Support

The main pain points for the “inbox” DSC Pull server have always been about scalability, availability and the lack of administrative capabilities. This all changes with the SQL backed pull server. Things like, fetching reports for a single or multiple nodes (with or without knowledge of the AgentId up front), figure out what nodes are under management of the pull server and options to change the configuration name server side have all become very easy. Also, creating a farm of Pull Servers is not possible without any hacks.

Let’s look at the new possible DSC Pull Server (Farm) architecture:

As you can see in the picture above, we can now have a HA SQL Database (I tested on both SQL Server and Azure SQL), one or more Pull Servers targeting the same database and sharing configuration mofs, modules and registration keys using shared storage. Also drawn in the diagram is an Administrative interface with SQL. For now, just know it is possible to interact with SQL directly but keep reading the posts in the blog series as I will layer things on top of this, like a REST API, to enhance the accessibility and usability for Administrative purposes.

Setting it up

For this post, I’ll setup a single (and simple, and unsecure) Pull Server with a SQL Database using the following DSC configuration.

configuration PullServerSQL {
    Import-DscResource -ModuleName PSDesiredStateConfiguration -ModuleVersion 1.1
    Import-DscResource -ModuleName xPSDesiredStateConfiguration -ModuleVersion 8.4.0.0

    WindowsFeature dscservice {
        Name   = 'Dsc-Service'
        Ensure = 'Present'
    }

    File PullServerFiles {
        DestinationPath = 'c:\pullserver'
        Ensure = 'Present'
        Type = 'Directory'
        Force = $true
    }

    xDscWebService PSDSCPullServer {
        Ensure                       = 'Present'
        EndpointName                 = 'PSDSCPullServer'
        Port                         = 8080
        PhysicalPath                 = "$env:SystemDrive\inetpub\PSDSCPullServer"
        CertificateThumbPrint        = 'AllowUnencryptedTraffic'
        ModulePath                   = "c:\pullserver\Modules"
        ConfigurationPath            = "c:\pullserver\Configuration"
        State                        = 'Started'
        RegistrationKeyPath          = "c:\pullserver"
        UseSecurityBestPractices     = $false
        SqlProvider                  = $true
        SqlConnectionString          = 'Provider=SQLOLEDB.1;Server=sql.mshome.net;Database=DemoDSC;User ID=SA;Password=Welkom01;Initial Catalog=master;'
        DependsOn                    = '[File]PullServerFiles', '[WindowsFeature]dscservice'
    }

    File RegistrationKeyFile {
        Ensure          = 'Present'
        Type            = 'File'
        DestinationPath = "c:\pullserver\RegistrationKeys.txt"
        Contents        = 'cb30127b-4b66-4f83-b207-c4801fb05087'
        DependsOn       = '[File]PullServerFiles'
    }
}

Note that the configuration makes use of xPSDesiredStateConfiguration version 8.4.0.0 which has the SQL support coded in. SQL support started in an earlier version of the module but had some bugs, so make sure you have version 8.4.0.0 or later.

The SQL connection string used:

specifies a Provider of SQLOLEDB.1. You could also make use of the SQLNCLI11 provider but this requires you to install the SQL Server Native Client as an additional component of the Pull Server.
specifies the database name, the first time the Pull Server API is connected to, a database will be created automatically. Not specifying the name in the connection string will result in a database called DSC.
uses an Initial Catalog as master. This is required for the Database to get created. You could create the database up front using my PowerShell module DscPullServerAdmin (more on this in a later post) which would allow you to leave out the Initial Catalog.
uses a SQL login (with plain text password, can be encrypted manually). You could also use integrated authentication.

I’ve setup 2 VMs on my local Hyper-V:

sql.mshome.net (A Windows Server 1803 server with SQL Server 2017 installed with a default SQL instance name
pull.mshome.net (A Windows Server 1803 server)

As I’m using the Default Switch on my box, both VMs have the mshome.net domain suffix.

So let’s run this configuration.

PullServerSQL
Start-DscConfiguration -Path .\PullServerSQL -Wait -Verbose -Force

Done! The pull server is setup. Let’s quickly check the web.config settings.

([xml](Get-Content -Path C:\inetpub\PSDSCPullServer\web.config)).configuration.appsettings.GetEnumerator()

As you can see, the connection string is stored in plain text. So if you need to have this encrypted, you need to take additional steps.

The database is not created yet as the API has not been called in any way.

We can force the creation by just calling the api manually.

Invoke-WebRequest -Uri http://localhost:8080/PSDSCPullServer.svc -UseBasicParsing

A database is created containing 3 tables:

Devices (LCMv1, WMF4 clients are registered here)
RegistrationData (LCMv2+, WMF5+ clients are registered here)
StatusReport (LCMv2+, WMF5+ client reports are stored here)

Great! we now have a SQL backed pull server and nodes can start onboarding! Also scaling it out is as simple as running the same configuration on a second pull server and having a loadbalancer in front of them. Second and additional servers will connect to the same database and thus know about all the registered nodes. You just need to handle the content of c:\pullserver to be synced between all servers or have it coming from shared storage like storage spaces direct.

That is it for this post. To be continued!