Reliability in Azure VM Image Builder

Azure VM Image Builder is a managed service that builds and customizes virtual machine (VM) images, then distributes the resulting images to targets such as Azure Compute Gallery or virtual hard disks (VHDs) in Azure Blob Storage. The service doesn't require you to maintain long-lived build infrastructure.

When you use Azure, reliability is a shared responsibility. Microsoft provides a range of capabilities to support resiliency and recovery. You're responsible for understanding how those capabilities work within all of the services you use, and selecting the capabilities you need to meet your business objectives and uptime goals.

This article describes how to make Azure VM Image Builder resilient to various potential outages and problems, including transient faults, availability zone failures, region-wide failures, and service maintenance. It also describes backup and restore options, custom multiregion solutions, and key information about the Azure VM Image Builder service-level agreement (SLA).

Production deployment recommendations

Consider the following recommendations when you design a production image-building pipeline:

Reliability architecture overview

This section describes some of the important aspects of how the service works that are most relevant from a reliability perspective. The section introduces the logical architecture, which includes some of the resources and features that you deploy and use. It also discusses the physical architecture, which provides details on how the service works under the covers.

Logical architecture

When you use VM Image Builder, you deploy an image template to a single Azure region, which becomes the template's home region.

After you create the template, you run a build, which executes a series of steps to create the VM image. Those steps might execute scripts or include packages that you want to embed in the image.

The service creates or uses a staging resource group and deploys build infrastructure in your subscription. After the build, the service removes temporary resources, including the build VM and associated networking resources. The staging resource group and its storage resources remain for as long as the image template resource exists.

VM Image Builder distributes the finished image to one or more targets that you specify. Common targets include:

• Azure Compute Gallery (formerly known as Shared Image Gallery): Compute Gallery provides multiregion replication of VM images. For example, you can build an image in one region and distribute the resulting Azure Compute Gallery image version to several target regions.

• Azure Blob Storage: The service can store the VHD in the storage account that VM Image Builder deploys or in a storage account and blob container that you specify.

  If you use the service-managed storage account as a VHD distribution destination, the service stores the VHD in the staging resource group. When you delete the image template, the service also deletes the VHD. To retain the VHD independently of the image template, distribute it to a custom storage account or copy it to another location before you delete the template.

Azure Compute Gallery and Blob Storage are separate services from VM Image Builder. You can distribute images to multiple targets, including targets in different Azure regions. This distribution model helps you separate the region where you build an image from the regions where you consume that image.

Physical architecture

VM Image Builder uses Isolated Image Builds to run the core image customization and validation process. The service deploys a dedicated Azure Container Instances resource and a build VM in the staging resource group in your subscription. The Container Instances resource runs HashiCorp Packer to customize and validate the image, and VM Image Builder coordinates the build.

To run image builds, register the Microsoft.ContainerInstance resource provider and ensure that your subscription has sufficient Container Instances quota. For more information about the service architecture, supported regions, and build process, see Azure VM Image Builder overview.

Separate Azure services, such as Azure Compute Gallery and Blob Storage, store finished images. Configure each distribution target to meet your resilience requirements.

Resilience to transient faults

Transient faults are short, intermittent failures in components. They occur frequently in a distributed environment like the cloud, and they're a normal part of operations. Transient faults correct themselves after a short period of time. It's important that your applications can handle transient faults, usually by retrying affected requests.

All cloud-hosted applications should follow the Azure transient fault handling guidance when they communicate with any cloud-hosted APIs, databases, and other components. For more information, see Recommendations for handling transient faults.

You typically interact with Azure VM Image Builder by using the Azure portal, Azure CLI, Azure PowerShell, or infrastructure as code such as Bicep. Your build pipeline is responsible for handling transient failures. If Azure Resource Manager returns a transient error response, retry the request with appropriate backoff.

If a build fails, inspect its lastRunStatus, customization and validation logs, and distribution run outputs before you rerun it. These details help you determine whether a transient fault condition, customization step, or image-distribution operation caused the failure. For more information, see Troubleshoot VM Image Builder build failures.

Build operations can also depend on external systems, such as storage accounts, package repositories, source image locations, and distribution targets. Design your customization steps and pipelines to support retries after a temporary network, platform, or dependency issue.

Resilience to availability zone failures

Build operations and finished images behave differently during zone failures.

• Builds: VM Image Builder doesn't support availability zones and doesn't provide a zone-resilience guarantee for image templates or build operations. If a zone failure affects resources that a build uses, the build might fail to start or might be interrupted. VM Image Builder doesn't automatically move the build to another zone.

  If you need to run image builds during zone outages, deploy your image templates and supporting resources in multiple Azure regions. For more information, see Custom multiregion solutions for resiliency.

• Images: Finished images remain available if their storage is zone-resilient.

  • Azure Compute Gallery: Azure Compute Gallery image versions can use zone-redundant storage (ZRS) in target regions that support availability zones. For each target region, select ZRS as the storage account replication type. The default is locally redundant storage (LRS), which isn't resilient to zone failures. For more information, see Azure Compute Gallery high availability.

  • VHDs in Azure Blob Storage: If you don't use a custom storage account, VM Image Builder creates a ZRS storage account by default in regions that support availability zones. If you use a custom storage account, you're responsible for configuring its redundancy to meet your requirements. Use ZRS to store copies of the VHD across multiple availability zones. For more information about how ZRS works, see Reliability in Azure Blob Storage.

Resilience to region-wide failures

Build operations and finished images behave differently during region-wide failures.

• Builds: Azure VM Image Builder is a single-region service and doesn't provide built-in multiregion failover. If the region that hosts your image template becomes unavailable, the image template and build operations in that region are also unavailable. The service doesn't automatically fail over to another region.

  If you need to run image builds during region outages, deploy your image templates and supporting resources in multiple Azure regions. For more information, see Custom multiregion solutions for resiliency.

• Images: Finished images remain available if their storage is resilient to region-wide failures.

  • Azure Compute Gallery: You can configure Azure Compute Gallery to replicate image versions to multiple regions. Replication is asynchronous, and you can use an image version in a target region only after replication to that region finishes.

    During an outage in one region, completed replicas remain available for VM deployments in healthy target regions. However, if the region that contains the gallery resource is unavailable, you can't create or update gallery resources. If you need to continue publishing image versions during a region outage, maintain separate galleries in separate build regions. For more information, see Azure Compute Gallery replication and best practices.

  • VHDs in Azure Blob Storage: The default service-managed VHD destination doesn't provide geo-redundancy. In regions that support availability zones, it uses ZRS, which doesn't protect against a region-wide failure.

    In a region with a paired region, you can distribute the VHD to a custom storage account that uses geo-redundant storage (GRS) or geo-zone-redundant storage (GZRS). Read-access GRS (RA-GRS) and read-access GZRS (RA-GZRS) provide read-only access through a secondary endpoint, but they don't perform failover. To restore normal read-write access in the paired region, initiate a customer-managed failover or copy the VHD from the secondary endpoint to a separate storage account. Geo-replication is asynchronous, so an unplanned failover might lose data that wasn't replicated. Check the storage account's Last Sync Time property before failover when possible.

    For more information about Blob Storage failover behavior and alternative approaches for replicating data between nonpaired regions, see Reliability in Azure Blob Storage.

Custom multiregion solutions for resiliency

If you need to keep building images during a region outage, you're responsible for deploying and operating independent build environments in multiple regions.

Choose regions that support VM Image Builder. In each build region, predeploy an image template and its region-specific dependencies, including managed identities and role assignments, networking configuration, customization artifacts, a source image or image replica, and distribution targets.

If you need to publish images while a gallery's region is unavailable, use an independent Compute Gallery resource in each build region. Replicating image versions from a single gallery doesn't make the gallery a fully multiregion resource.

Define these resources by using infrastructure as code, and treat those definitions as the source of truth. Keep the regional environments synchronized when the configuration changes, and regularly test that each environment can build and distribute an image.

Backup and restore

Azure VM Image Builder doesn't replace your backup and recovery process. In most cases, the artifact that matters for recovery is the finished image that you publish to Azure Compute Gallery or VHD storage.

To improve recovery from corruption or an extended unplanned outage, follow these recommendations:

• Retain published images in Azure Compute Gallery or other distribution targets according to your retention requirements.

• If you create a VHD and use the default VHD destination, copy the VHD to your own durable storage before you delete the image template.

• Use source control to store and version image template definitions.

• Use source control or another type of durable storage to store and version customization scripts and packages.

• Regularly validate that each predeployed recovery environment can build and distribute an image.

Resilience to service maintenance

Microsoft manages maintenance for Azure VM Image Builder. VM Image Builder doesn't serve application traffic at runtime, so maintenance doesn't affect VMs that you already created from published images. If a build is delayed or interrupted during maintenance or another service event, inspect its status and logs, and rerun it.

Use Azure Service Health to monitor planned maintenance and service incidents that might affect your build region.

Service-level agreement

Azure VM Image Builder doesn't provide an availability service-level agreement (SLA).

Any SLA that applies to a distribution target or storage service applies separately and doesn't provide an availability commitment for VM Image Builder template or build operations.

Related content

• Azure VM Image Builder overview
• Create a VM Image Builder template
• Isolated Image Builds for Azure VM Image Builder
• Troubleshoot Azure VM Image Builder
• Azure Compute Gallery overview