Intermittent ACA-to-ACR TCP timeouts across 10 workloads in Japan East (2026-08-04)

Hi @Bjorn Jee ,

Thank you for sharing the detailed diagnostics. We reviewed the Container Apps, Azure Firewall, and ACR logs, and the findings help narrow down the issue significantly.

The dial tcp ...:443: i/o timeout error indicates that the connection to ACR is timing out before the image can be pulled successfully. The Allow entries in Azure Firewall confirm that some traffic was permitted, but they don't necessarily confirm that every connection completed successfully.

To help us isolate and resolve the issue, please try the following checks:

1. Check whether the failures are reaching Azure Firewall

Compare the timestamp of the i/o timeout errors in Container Apps with the Azure Firewall logs.

If there is no corresponding firewall entry when the timeout occurs, the issue is likely before the traffic reaches the firewall.

If there is a corresponding Allow entry, we should investigate the connection between the firewall and ACR.

2. Check Azure Firewall SNAT utilization

Please review SNAT port utilization during the failure period and use Max aggregation rather than Average.

If utilization is high on a particular firewall instance, intermittent connection failures can occur when many Container Apps replicas are pulling images simultaneously.

3. Verify ACR endpoints and DNS resolution

Please confirm that the following ACR endpoints are reachable from the Container Apps network path:

• <registry>.azurecr.io
• <registry>.<region>.data.azurecr.io

If you are using Private Link, verify that both the registry endpoint and data endpoint resolve to the expected private IP addresses.

4. Review the Azure Firewall rules

If the traffic currently goes through an application rule, consider testing ACR traffic through a network rule using the AzureContainerRegistry service tag.

Also verify that the required Azure Storage endpoints are allowed, because ACR image layers may be retrieved through storage endpoints.

5. Test with an ACR Private Endpoint

As a longer-term and more reliable option, consider configuring a Private Endpoint for ACR.

This removes the public ACR endpoint, Azure Firewall application proxy path, and associated SNAT dependency from the image-pull path.

If you use this option, please make sure both the ACR login/REST endpoint and regional data endpoint are correctly configured in DNS.

6. For Azure Container Apps Jobs

If the affected workloads are Container Apps Jobs, review the retry and timeout configuration.

Make sure replicaRetryLimit and replicaTimeout provide enough time for a temporary infrastructure/network interruption to recover. Increasing retries alone may not help if the overall replica timeout is reached first.

Could you please share the results of the following:

• Correlation of the Container Apps timeout events with the Azure Firewall logs.
• Azure Firewall SNAT utilization, preferably using Max aggregation per instance.
• Any Service Health or Resource Health events for Japan East during the affected time window.

Based on the current evidence, there is no indication that the application itself is causing the failure. If the issue is confirmed to be on the network path, we can then focus on a more reliable architecture, such as using an ACR Private Endpoint, or optimizing the existing Firewall path.

We also recommend enabling the appropriate monitoring/alerts for image-pull failures so that the next occurrence can be captured with the required network-level details.

I hope this helps. Please let us know the results of the above checks, and we can assist further.

Hi Likhitha, thank you. We completed the suggested checks:

• The ACA pull timeouts occurred during 00:02:33–01:06:49 UTC and 05:19:15–06:29:50 UTC.
• Azure Firewall logs in those windows show only Allow decisions for the relevant ACR FQDNs, and DNS-proxy responses were NOERROR.
• Maximum SNAT utilisation peaked at approximately 2%, with firewall health at 100%.
• The manifests existed before the incident, and ACR continued serving successful pulls during both windows.
• No relevant Azure control-plane changes were found.
• The affected jobs failed during image pull, before the application containers started. Each replica tried twice, and six separately triggered executions over approximately ten minutes all failed.
• The same ACA → Azure Firewall → public ACR architecture and rules work successfully in our other Azure accounts/environments.

ACA logs do not expose the managed-node source IP or a flow/correlation ID, so we cannot determine where the individual TCP connections timed out from customer-visible telemetry.

Please help investigate the ACA/ACR or regional network path in Japan East for these windows and advise on supported mitigation for ACA pre-start image-pull failures. We can provide exact resource identifiers and detailed logs privately if needed.

Hi @Bjorn Jee ,

Based on the evidence you shared, there is no clear indication of an application issue, DNS failure, or SNAT exhaustion. The intermittent i/o timeout appears to be occurring during the connection to the ACR endpoint.

Here are the recommended steps to isolate and mitigate the issue:

1. Correlate the timeout with Azure Firewall logs Compare each i/o timeout timestamp with AZFWApplicationRule. An Allow entry confirms that the TLS request reached the firewall, but it does not confirm that the firewall successfully connected to ACR. This will help identify where the connection is failing.
2. Enable VNet flow logging Enable flow logs/Traffic Analytics for the ACA infrastructure subnet. This can help confirm whether traffic leaves the ACA subnet and whether return traffic is received.
3. Check the ACR endpoints Your current error is related to the ACR login/manifest endpoint (<registry>.azurecr.io). Also verify that the required ACR data endpoint and *.blob.core.windows.net traffic are allowed, as these may be required after manifest retrieval succeeds.
4. Try an Azure Firewall network rule instead of an application rule Allow ACR using the AzureContainerRegistry service tag. This removes the application-layer inspection path and provides better L3/L4 visibility for troubleshooting.
5. Consider an ACR Private Endpoint Since your ACR is Premium, using Private Link is a strong mitigation. It removes the public ACR endpoint, firewall application-rule path, and public SNAT dependency from the image-pull path. Ensure both the registry and data endpoints resolve correctly through Private DNS.
6. For Azure Container Apps Jobs, review replicaRetryLimit, replicaTimeout, and parallelism. Increasing retries alone may not help because replicaTimeout takes precedence when it expires first.
7. Add monitoring for future occurrences Create alerts for Failed to pull image, ErrImagePull, ImagePullBackOff, and specifically dial tcp ... i/o timeout in ContainerAppSystemLogs. Also monitor ACR/ACA Resource Health and Azure Firewall SNAT utilization.

Based on your current results, I would recommend VNet flow logging + Firewall network-rule testing first, followed by ACR Private Endpoint as the longer-term mitigation. This should help determine whether the issue is specific to the Firewall path or the public ACR endpoint without changing your application or introducing an image-mirroring workflow.

Thank you. We will enable VNet flow logging on the ACA infrastructure subnet and correlate it with the existing firewall and ACA timestamps if the issue recurs. We will also evaluate a controlled AzureContainerRegistry network-rule test. As the incident is intermittent and not currently reproducible, please let us know if Microsoft can correlate the two reported windows with service-side telemetry; we can provide exact identifiers and detailed logs only through an approved private Microsoft channel.

Hi @Bjorn Jee ,
Thank you for the detailed investigation and for sharing the additional findings.

Based on the evidence so far, we have ruled out several common causes, including image availability, permissions, DNS resolution, SNAT exhaustion, firewall health, and registry-wide issues.

The remaining question is where the connection is being lost during the image pull: between Azure Container Apps and the firewall, or between the firewall and Azure Container Registry.

Could you please run the following two checks using your existing logs?

1. Identify the source IP from the firewall logs

AZFWApplicationRule
| where TimeGenerated between (datetime(2026-08-04 00:02:33) .. datetime(2026-08-04 01:06:49))
    or TimeGenerated between (datetime(2026-08-04 05:19:15) .. datetime(2026-08-04 06:29:50))
| where Fqdn has "azurecr.io"
| project TimeGenerated, SourceIp, SourcePort, Fqdn, Action, ActionReason, Rule
| order by TimeGenerated asc

Please compare the number of returned records with the 110 Container Apps failure records you already identified.

• If the numbers are similar, the requests reached the firewall, so we can focus on the firewall-to-ACR path.
• If there are significantly fewer records, we can focus on the Container Apps-to-firewall path.

2. Check for firewall-side DNS failures

Please query AZFWInternalFqdnResolutionFailure for August 4, 2026, 00:00–07:00 UTC.

• No records would rule out intermittent DNS resolution failures inside Azure Firewall.
• Any records would make firewall-side DNS resolution a strong area for further investigation. Please share the results of the two checks, and we can take the investigation forward from there.