Passing variables between stages in Azure DevOps pipelines
The other day I was working on an Infrastructure as Code project that involved deploying an Azure Container registry. That ACR is typically a resource that you deploy just ones in your production environment and not one per environment. You do that because you want container images to be used as immutable artifacts that are progressively deployed across all your environments.
The problem you then get is that you do need to, for example, assign permissions on that ACR for the AKS user in your dev and test environment (your kubelet identity needs the ‘ACR Pull’ permissions). When you deploy your dev or test resources, you usually cannot do that. That connection you use there should not have permission to assign that role to the production environment.
Multi-stage pipeline
To come around that, we can pass the AKS identity to the production stage and do the role assignment there. The Bicep sources and pipeline can be found here. Let’s go over some details.
First, the main Bicep file will create a resource group, the identity used for the nodes in AKS, and the registry:
targetScope = 'subscription'
@allowed([
'dev'
'test'
'prod'
])
param environmentShort string
param location string = 'westeurope'
param locationShort string = 'we'
resource rgshared 'Microsoft.Resources/resourceGroups@2021-04-01' = {
name: 'rg-shared-${environmentShort}-${locationShort}-001'
location: location
}
var kubeletIdentityName = 'id-demo-kubelet-${environmentShort}-${locationShort}-001'
module kubeletIdentity 'userAssignedIdentities.bicep' = {
scope: rgshared
name: 'kubeletIdentity'
params: {
identityName: kubeletIdentityName
location: location
}
}
var acrName = 'acrdemo${environmentShort}${locationShort}001'
module acr 'registries.bicep' = if (environmentShort == 'prod') {
scope: rgshared
name: 'acr'
params: {
location: location
name: acrName
}
}
output kubeletIdentityPrincipalId string = kubeletIdentity.outputs.principalId
output acrId string = environmentShort == 'prod' ? acr.outputs.id : ''
Notice how the kubelets principalId and the ID of the ACR are returned to the Azure DevOps pipeline. The ID of the ACR will only be there on the production deployment.
The pipeline then contains a task that converts that output into a pipeline variable:
- task: PowerShell@2
displayName: 'Set outputs'
name: SetOutputs
inputs:
targetType: inline
script: |
Write-Host $env:MAINDEPLOYMENTS_OUTPUTS
$armOutputObj = $env:MAINDEPLOYMENTS_OUTPUTS | convertfrom-json
Write-Host $armOutputObj
$kubeletPrincipalId = $armOutputObj.kubeletIdentityPrincipalId.value
Write-Host "kubeletPrincipalId:"
Write-Host $kubeletPrincipalId
Write-Host "##vso[task.setvariable variable=kubeletPrincipalId;isOutput=true]$kubeletPrincipalId"
In the next stage, we can grab that variable and use that as input by declaring it as a variable on the stage:
- stage: DeployProd
variables:
- name: devKubeletPrincipalId
value: $[ stageDependencies.DeployDev.DeployResources.outputs['DeployResources.SetOutputs.kubeletPrincipalId'] ]
The last task in the pipeline can then use that in the role assignments:
- task: AzureCLI@2
displayName: Role assignments on ACR
inputs:
azureSubscription: ${{ variables.ProdAzureResourceManagerConnection }}
scriptType: pscore
scriptLocation: inlineScript
addSpnToEnvironment: true
inlineScript: |
$armOutputObj = $env:MAINDEPLOYMENTS_OUTPUTS | convertfrom-json
write-host "prod ID"
write-host $armOutputObj.kubeletIdentityPrincipalId.value
write-host "Dev ID"
write-host $(devKubeletPrincipalId)
write-host "ACR ID"
write-host $armOutputObj.acrId.value
az role assignment create --role "AcrPull" --assignee-object-id $(devKubeletPrincipalId) --scope $armOutputObj.acrId.value --assignee-principal-type ServicePrincipal
az role assignment create --role "AcrPull" --assignee-object-id $armOutputObj.kubeletIdentityPrincipalId.value --scope $armOutputObj.acrId.value --assignee-principal-type ServicePrincipal
Here’s the complete pipeline. Don’t forget to create the environments used in the pipeline, and the identity used to run the production stage needs to have permission to do a role assignment.
trigger: none
pool:
vmImage: ubuntu-latest
variables:
ModulesFolderPath: $(Build.SourcesDirectory)/multi-stage-pass-variables/Bicep
ParameterFilesFolderPath: $(Build.SourcesDirectory)/multi-stage-pass-variables/Pipelines
NonProdSubscriptionId: 'f0e483fc-9d2f-4a4b-8aee-887a398ff27e'
NonProdAzureResourceManagerConnection: 'Azure MVP Sponsorship'
ProdSubscriptionId: 'f0e483fc-9d2f-4a4b-8aee-887a398ff27e'
ProdAzureResourceManagerConnection: 'Azure MVP Sponsorship'
stages:
- stage: 'PublishTemplates'
displayName: 'PublishTemplates'
variables:
MainBicepFilePath: '${{ variables.ModulesFolderPath }}/mainDeployment.bicep'
MainTemplateFilePath: '${{ variables.ModulesFolderPath }}/mainDeployment.json'
MainTemplateTestParameterFilePath: '${{ variables.ParameterFilesFolderPath }}/parameters.dev.json'
jobs:
- job: PublishTemplates
displayName: Publish Templates
steps:
- checkout: self
path: src
- task: PowerShell@2
displayName: Bicep Build
inputs:
targetType: 'inline'
script: |
az bicep build --file ${{ variables.MainBicepFilePath }}
pwsh: true
- task: AzureResourceManagerTemplateDeployment@3
displayName: Validate ARM Template
inputs:
azureResourceManagerConnection: ${{ variables.NonProdAzureResourceManagerConnection }}
deploymentScope: 'Subscription'
subscriptionId: '${{ variables.NonProdSubscriptionId }}'
location: 'West Europe'
templateLocation: 'Linked artifact'
csmFile: '${{ variables.MainTemplateFilePath }}'
csmParametersFile: '${{ variables.MainTemplateTestParameterFilePath }}'
deploymentMode: 'Validation'
- task: CopyFiles@2
displayName: 'Copy templates to $(build.artifactstagingdirectory)/templates'
inputs:
SourceFolder: ${{ variables.ModulesFolderPath }}
Contents: '*.json'
TargetFolder: '$(build.artifactstagingdirectory)/templates'
- task: CopyFiles@2
displayName: 'Copy parameters to $(build.artifactstagingdirectory)/templates'
inputs:
SourceFolder: ${{ variables.ParameterFilesFolderPath }}
Contents: '*.json'
TargetFolder: '$(build.artifactstagingdirectory)/templates'
- task: PublishBuildArtifacts@1
displayName: 'Publish Artifact: drop'
- stage: DeployDev
displayName: Deploy template to dev environment
jobs:
- deployment: DeployResources
environment: 'IaCDevEnvironment'
strategy:
runOnce:
deploy:
steps:
- template: resourcesDeployment.yml
parameters:
SubscriptionId: ${{ variables.NonProdSubscriptionId }}
Location: West Europe
AzureResourceManagerConnection: ${{ variables.NonProdAzureResourceManagerConnection }}
Environment: 'dev'
- stage: DeployProd
variables:
- name: devKubeletPrincipalId
value: $[ stageDependencies.DeployDev.DeployResources.outputs['DeployResources.SetOutputs.kubeletPrincipalId'] ]
displayName: Deploy template to prod environment
jobs:
- deployment:
environment: 'IaCProdEnvironment'
strategy:
runOnce:
deploy:
steps:
- template: resourcesDeployment.yml
parameters:
SubscriptionId: ${{ variables.ProdSubscriptionId }}
Location: West Europe
AzureResourceManagerConnection: ${{ variables.ProdAzureResourceManagerConnection }}
Environment: 'prod'
- task: AzureCLI@2
displayName: Role assignments on ACR
inputs:
azureSubscription: ${{ variables.ProdAzureResourceManagerConnection }}
scriptType: pscore
scriptLocation: inlineScript
addSpnToEnvironment: true
inlineScript: |
$armOutputObj = $env:MAINDEPLOYMENTS_OUTPUTS | convertfrom-json
write-host "prod ID"
write-host $armOutputObj.kubeletIdentityPrincipalId.value
write-host "Dev ID"
write-host $(devKubeletPrincipalId)
write-host "ACR ID"
write-host $armOutputObj.acrId.value
az role assignment create --role "AcrPull" --assignee-object-id $(devKubeletPrincipalId) --scope $armOutputObj.acrId.value --assignee-principal-type ServicePrincipal
az role assignment create --role "AcrPull" --assignee-object-id $armOutputObj.kubeletIdentityPrincipalId.value --scope $armOutputObj.acrId.value --assignee-principal-type ServicePrincipal