Automatically renew the image used in an Azure DevOps private agent
In a previous post, I wrote about creating your own hosted Build and Release agents in Azure DevOps. That process could be improved by regenerating the VM image, for example, every month. By doing that, you stay up-to-date with both the latest versions of all the tools installed as well as with security patches. This post will describe how to do that using an Azure DevOps pipeline which will be triggered monthly.
One note before we get started. The process of creating the vhd takes somewhere between one and two hours for the Ubuntu 2004 version. The free license you get in Azure DevOps only allows you to run a pipeline for 60 minutes. You, therefore, need a paid license to runs this pipeline as that enables you to run up to six hours on a Microsoft Hosted Agent.
Generate the .vhd file
One of the first steps in the previous post was to use the ‘GenerateResourcesAndImage.ps1’ script to generate the vhd file. That script contains some logic to recreate a storage account etc which we don’t need and makes automating this process harder. I therefor created a trimmed-down version of it which I store in a file with the original name in a folder called ‘scripts’:
$ErrorActionPreference = 'Stop'
enum ImageType {
Windows2016 = 0
Windows2019 = 1
Windows2022 = 2
Ubuntu1604 = 3
Ubuntu1804 = 4
Ubuntu2004 = 5
}
Function Get-PackerTemplatePath {
param (
[Parameter(Mandatory = $True)]
[string] $RepositoryRoot,
[Parameter(Mandatory = $True)]
[ImageType] $ImageType
)
switch ($ImageType) {
([ImageType]::Windows2016) {
$relativeTemplatePath = Join-Path "win" "windows2016.json"
}
([ImageType]::Windows2019) {
$relativeTemplatePath = Join-Path "win" "windows2019.json"
}
([ImageType]::Windows2022) {
$relativeTemplatePath = Join-Path "win" "windows2022.json"
}
([ImageType]::Ubuntu1604) {
$relativeTemplatePath = Join-Path "linux" "ubuntu1604.json"
}
([ImageType]::Ubuntu1804) {
$relativeTemplatePath = Join-Path "linux" "ubuntu1804.json"
}
([ImageType]::Ubuntu2004) {
$relativeTemplatePath = Join-Path "linux" "ubuntu2004.json"
}
default { throw "Unknown type of image" }
}
$imageTemplatePath = [IO.Path]::Combine($RepositoryRoot, "images", $relativeTemplatePath)
if (-not (Test-Path $imageTemplatePath)) {
throw "Template for image '$ImageType' doesn't exist on path '$imageTemplatePath'"
}
return $imageTemplatePath;
}
Function Get-LatestCommit {
[CmdletBinding()]
param()
process {
Write-Host "Latest commit:"
git --no-pager log --pretty=format:"Date: %cd; Commit: %H - %s; Author: %an <%ae>" -1
}
}
Function GenerateResourcesAndImage {
param (
[Parameter(Mandatory = $True)]
[string] $SubscriptionId,
[Parameter(Mandatory = $True)]
[string] $ResourceGroupName,
[Parameter(Mandatory = $True)]
[string] $StorageAccountName,
[Parameter(Mandatory = $True)]
[ImageType] $ImageType,
[Parameter(Mandatory = $True)]
[string] $AzureLocation,
[Parameter(Mandatory = $False)]
[string] $ImageGenerationRepositoryRoot = $pwd,
[Parameter(Mandatory = $False)]
[string] $AzureClientId,
[Parameter(Mandatory = $False)]
[string] $AzureClientSecret,
[Parameter(Mandatory = $False)]
[string] $AzureTenantId,
[Parameter(Mandatory = $False)]
[Switch] $RestrictToAgentIpAddress
)
$builderScriptPath = Get-PackerTemplatePath -RepositoryRoot $ImageGenerationRepositoryRoot -ImageType $ImageType
$ServicePrincipalClientSecret = $env:UserName + [System.GUID]::NewGuid().ToString().ToUpper();
$InstallPassword = $env:UserName + [System.GUID]::NewGuid().ToString().ToUpper();
$AzSecureSecret = ConvertTo-SecureString $AzureClientSecret -AsPlainText -Force
$AzureAppCred = New-Object System.Management.Automation.PSCredential($AzureClientId, $AzSecureSecret)
Connect-AzAccount -ServicePrincipal -Credential $AzureAppCred -Tenant $AzureTenantId
Set-AzContext -SubscriptionId $SubscriptionId
# Parametrized Authentication via given service principal: The service principal with the data provided via the command line
# is used for all authentication purposes.
$spClientId = $AzureClientId
$ServicePrincipalClientSecret = $AzureClientSecret
$tenantId = $AzureTenantId
Get-LatestCommit -ErrorAction SilentlyContinue
$packerBinary = Get-Command "packer"
if (-not ($packerBinary)) {
throw "'packer' binary is not found on PATH"
}
if($RestrictToAgentIpAddress -eq $true) {
$AgentIp = (Invoke-RestMethod http://ipinfo.io/json).ip
echo "Restricting access to packer generated VM to agent IP Address: $AgentIp"
}
& $packerBinary build -on-error=ask `
-var "client_id=$($spClientId)" `
-var "client_secret=$($ServicePrincipalClientSecret)" `
-var "subscription_id=$($SubscriptionId)" `
-var "tenant_id=$($tenantId)" `
-var "location=$($AzureLocation)" `
-var "resource_group=$($ResourceGroupName)" `
-var "storage_account=$($StorageAccountName)" `
-var "install_password=$($InstallPassword)" `
-var "allowed_inbound_ip_addresses=$($AgentIp)" `
$builderScriptPath
}
To use that script in an Azure DevOps pipeline, we need the two steps as shown below (the entire pipeline template can be found at the end of this post).
- task: CmdLine@2
displayName: 'Clone Microsoft Virtual Environment repo'
inputs:
script: |
echo 'Cloning into sources folder at: $(Build.SourcesDirectory)'
git clone https://github.com/actions/virtual-environments.git --progress
ls -l
exit
- task: AzurePowerShell@5
displayName: Create new vhd
env:
AZURE_CLIENT_SECRET: $(AzureClientSecret)
inputs:
azureSubscription: '${{ variables.AzureServiceConnection }}'
ScriptType: 'InlineScript'
Inline: |
Import-Module .\AzureDevOpsAgents\scripts\GenerateResourcesAndImage.ps1
GenerateResourcesAndImage -SubscriptionId ${{ variables.SubscriptionId }} -ResourceGroupName $(ResourceGroupName) -StorageAccountName $(StorageAccountName) -ImageGenerationRepositoryRoot "$(Build.SourcesDirectory)/virtual-environments" -ImageType Ubuntu2004 -AzureLocation "West Europe" -AzureTenantId ${env:AZURE_TENANT_ID} -AzureClientId ${env:AZURE_CLIENT_ID} -AzureClientSecret ${env:AZURE_CLIENT_SECRET} -RestrictToAgentIpAddress
azurePowerShellVersion: 'LatestVersion'
pwsh: true
The first step clones the open-source repo created by Microsoft. The second step then runs the PowerShell file to generate the vhd file. When you create the pipeline in Azure DevOps, be sure to create variables for each of them used in the command. When this command is finished, there will be a .vhd file in a storage account that was created for you.
Shared Image Gallery
When working with these private Azure DevOps agent pools, you probably want to create one that you use for your production deployments and one that is used for non-production deployments. That allows your to run each of them in a seperate network that is connected with the network you need to deploy to (or use a subnet in that Virtual Network). This means that you will need a Shared Image Library since that will allow you to share an image across subscription and even Azure tenants. Here’s how to create the Stared Image Gallery and create the Image Definition. We will later add a new version to that image definition every time we create a new image.
# create storaga account
$RGNAME="rg-azdoagent-vhd"
$LOCATION=northeurope
$GALLERY="AzDoAgentImage"
# Create a resource group
az group create -l $LOCATION -n $RGNAME
# Create the storage account to hold the VHD files
az storage account create -n storazdoagentvhd -g $RGNAME --sku Standard_LRS
# Set permissions for the identity used on your Azure DevOps Service Connection
$storageId = $(az storage account show -n storazdoagentvhd --query id -o tsv)
az role assignment create --assignee <devops agent principal> --role "Storage Blob Data Contributor" --scope $storageId
# Create image gallery
az sig create --resource-group $RGNAME --gallery-name $GALLERY
sigid=$(az sig show `
--resource-group $RGNAME `
--gallery-name $GALLERY `
--query id -o tsv)
# Creating image Definition
az sig image-definition create `
--resource-group $RGNAME `
--gallery-name $GALLERY `
--gallery-image-definition $GALLERY `
--publisher $GALLERY `
--offer $GALLERY `
--sku $GALLERY `
--os-type Linux `
--os-state generalized
Creating the Image
Time to create the image. The packer command that creates the vhd does not return the name of the vhd as a variable. We do need that to create an image version. The first line in this script finds the vhd file by listing all available vhd’s and sort them by their creation time. Using the name property of the vhd, we can construct its URI. Now that we have that, we can create the new image version in the Shared Gallery by using another Azure CLI command. In the next step, you need the ID of that new image so we output that on the last line.
- task: AzureCLI@2
displayName: Create Image from VHD
inputs:
azureSubscription: '${{ variables.AzureServiceConnection }}'
scriptType: 'pscore'
scriptLocation: 'inlineScript'
inlineScript: |
$vhd = az storage blob list -c system --account-name $(StorageAccountName) | ConvertFrom-Json | Where-Object {$_.name -like "*.vhd"} | sort-object {[System.DateTime]::ParseExact($_.properties.creationTime,"MM/dd/yyyy HH:mm:ss",$null)} -Descending | Select-Object -first 1
$vhd
$url = "https://$(StorageAccountName).blob.core.windows.net/system/$($vhd.name)"
Write-Host "Url: " $url
$storageAccountId = az storage account show -n $(StorageAccountName) --query id --output tsv
$imageCreateResult = az sig image-version create --resource-group $(ResourceGroupName) `
--gallery-name $(GalleryName) `
--gallery-image-definition $(GalleryName) `
--gallery-image-version 1.0.$(Build.BuildId) `
--os-vhd-storage-account $storageAccountId `
--os-vhd-uri $url
$imageCreateResult
$imageCreateResultObject = $imageCreateResult | ConvertFrom-Json
$imageId = $imageCreateResultObject.id
$imageId
Write-Host "##vso[task.setvariable variable=ImageId;]$imageId"
Update the Image on the existing VM Scale Set
The last step is to point the Virtual Machine Scale Set to the newly created image version. That is done using below step.
- task: AzureCLI@2
displayName: Update Scale set to use new image
inputs:
azureSubscription: '${{ variables.AzureServiceConnection }}'
scriptType: 'pscore'
scriptLocation: 'inlineScript'
inlineScript: |
Write-Host "Image Id: " $(ImageId)
az vmss update --resource-group <resource-group-name> --name <scale-set-name> --set virtualMachineProfile.storageProfile.imageReference.id=$(ImageId)
The complete pipeline is shown below:
trigger: none
schedules:
- cron: "0 0 1 * *"
displayName: Monthly run
branches:
include:
- main
pool:
image: ubuntu-latest
variables:
AzureServiceConnection: <>
SubscriptionId: ""
stages:
- stage: 'RefreshImageOnAzureDevOpsAgents'
displayName: 'RefreshImageOnAzureDevOpsAgents'
variables:
ScriptFolderPath: '$(Build.SourcesDirectory)/AzureDevOpsAgents/scripts/'
jobs:
- job: RefreshImageOnAzureDevOpsAgents
timeoutInMinutes: 0
displayName: Refresh Image On Azure DevOps Agents
steps:
- task: CmdLine@2
displayName: 'Clone Microsoft Virtual Environment repo'
inputs:
script: |
echo 'Cloning into sources folder at: $(Build.SourcesDirectory)'
git clone https://github.com/actions/virtual-environments.git --progress
ls -l
exit
- task: AzurePowerShell@5
displayName: Create new vhd
env:
AZURE_CLIENT_SECRET: $(AzureClientSecret)
inputs:
azureSubscription: '${{ variables.AzureServiceConnection }}'
ScriptType: 'InlineScript'
Inline: |
Import-Module .\virtual-environments\helpers\GenerateResourcesAndImage.ps1
GenerateResourcesAndImage -SubscriptionId ${{ variables.SubscriptionId }} -ResourceGroupName $(ResourceGroupName) -StorageAccountName $(StorageAccountName) -ImageGenerationRepositoryRoot "$(Build.SourcesDirectory)/virtual-environments" -ImageType Ubuntu2004 -AzureLocation "West Europe" -AzureTenantId ${env:AZURE_TENANT_ID} -AzureClientId ${env:AZURE_CLIENT_ID} -AzureClientSecret ${env:AZURE_CLIENT_SECRET} -RestrictToAgentIpAddress
azurePowerShellVersion: 'LatestVersion'
pwsh: true
- task: AzureCLI@2
displayName: Create Image from VHD
inputs:
azureSubscription: '${{ variables.AzureServiceConnection }}'
scriptType: 'pscore'
scriptLocation: 'inlineScript'
inlineScript: |
$vhd = az storage blob list -c system --account-name $(StorageAccountName) | ConvertFrom-Json | Where-Object {$_.name -like "*.vhd"} | sort-object {[System.DateTime]::ParseExact($_.properties.creationTime,"MM/dd/yyyy HH:mm:ss",$null)} -Descending | Select-Object -first 1
$vhd
$url = "https://$(StorageAccountName).blob.core.windows.net/system/$($vhd.name)"
Write-Host "Url: " $url
$storageAccountId = az storage account show -n $(StorageAccountName) --query id --output tsv
$imageCreateResult = az sig image-version create --resource-group $(ResourceGroupName) `
--gallery-name $(GalleryName) `
--gallery-image-definition $(GalleryName) `
--gallery-image-version 1.0.$(Build.BuildId) `
--os-vhd-storage-account $storageAccountId `
--os-vhd-uri $url
$imageCreateResult
$imageCreateResultObject = $imageCreateResult | ConvertFrom-Json
$imageId = $imageCreateResultObject.id
$imageId
Write-Host "##vso[task.setvariable variable=ImageId;]$imageId"
- task: AzureCLI@2
displayName: Update Scale set to use new image
inputs:
azureSubscription: '${{ variables.AzureServiceConnection }}'
scriptType: 'pscore'
scriptLocation: 'inlineScript'
inlineScript: |
Write-Host "Image Id: " $(ImageId)
az vmss update --resource-group <resource-group-name> --name <scale-set-name> --set virtualMachineProfile.storageProfile.imageReference.id=$(ImageId)