Troubleshoot bad gateway (502) errors in Azure Application Gateway

Summary

Learn how to troubleshoot bad gateway (502) errors in Azure Application Gateway so you can quickly restore reliable access to your web apps.

Symptoms

After you configure an application gateway, you might see the error "Server Error: 502 - Web server received an invalid response while acting as a gateway or proxy server". This error can happen for the following reasons:

• Network security group (NSG), user-defined route (UDR), or custom Domain Name System (DNS) issue
• Default health probe can't reach backend VMs
• Backend request time-out is exceeded
• Application gateway's backend pool isn't configured or is empty
• Unhealthy instances in BackendAddressPool
• Upstream SSL certificate doesn't match

Network security group, user-defined route, or custom DNS issue

Cause

If an NSG, UDR, or custom DNS blocks access to the backend, application gateway instances can't reach the backend pool. This issue causes probe failures, resulting in 502 errors.

You might have an NSG or UDR in either the application gateway subnet or the subnet where the application virtual machines (VMs) are deployed.

Similarly, the presence of a custom DNS in the virtual network (VNet) can also cause problems. The user-configured DNS server for the VNet might not correctly resolve a fully qualified domain name (FQDN) used for backend pool members.

Solution

Validate your NSG, UDR, and DNS configurations. To do so, follow these steps:

1. Check the NSGs associated with the application gateway subnet. Ensure that communication to the backend isn't blocked. For more information, see Network security groups.

2. Check the UDR associated with the application gateway subnet. Ensure that the UDR isn't directing traffic away from the backend subnet. For example, check for routing to network virtual appliances or default routes being advertised to the application gateway subnet by using Azure ExpressRoute or Azure VPN. Run the following commands in Azure PowerShell.

   $vnet = Get-AzVirtualNetwork -Name vnetName -ResourceGroupName rgName
   Get-AzVirtualNetworkSubnetConfig -Name appGwSubnet -VirtualNetwork $vnet
   

3. Check for effective NSGs and routes with the backend VM. Run the following commands in Azure PowerShell.

   Get-AzEffectiveNetworkSecurityGroup -NetworkInterfaceName nic1 -ResourceGroupName testrg
   Get-AzEffectiveRouteTable -NetworkInterfaceName nic1 -ResourceGroupName testrg
   

4. Check for the presence of custom DNS in the VNet. Check DNS by looking at the details of the VNet properties in the output.

   Get-AzVirtualNetwork -Name vnetName -ResourceGroupName rgName 
   DhcpOptions            : {
                              "DnsServers": [
                                "x.x.x.x"
                              ]
                            }
   

5. If present, ensure that the DNS server can resolve the backend pool member's FQDN correctly.

Default health probe can't reach backend VMs

Cause

A 502 error can also indicate that the default health probe can't reach backend VMs.

When you provision an application gateway instance, it automatically configures a default health probe to each BackendAddressPool by using properties of the BackendHttpSetting.

You don't need to provide any input to set this probe. Specifically, when you configure a load-balancing rule, you associate a BackendHttpSetting with a BackendAddressPool. Each of these associations has a default probe configured, and the application gateway starts a periodic health check connection to each instance in the BackendAddressPool at the port specified in the BackendHttpSetting element.

The following table lists the values associated with the default health probe.

Solution

Use the following guidance to troubleshoot the default health probe.

• The host value of the request is set to 127.0.0.1. Ensure that a default site is configured and is listening at 127.0.0.1.
• The BackendHttpSetting protocol determines the protocol of the request.
• The URI Path is set to /*.
• If BackendHttpSetting specifies a port other than 80, configure the default site to listen at that port.
• The call to protocol://127.0.0.1:port should return an HTTP result code of 200. This code should be returned within the 30-second timeout period.
• Ensure the configured port is open and there are no firewall rules or Azure NSGs blocking incoming or outgoing traffic on the port configured.
• If you use Azure classic VMs or Cloud Service with an FQDN or a public IP, ensure that you open the corresponding endpoint.
• If you configure the VM using Azure Resource Manager (ARM) and it's outside the VNet where the application gateway is deployed, configure a NSG to allow access on the desired port.

For more information, see Application Gateway infrastructure configuration.

Invalid or improper configuration of custom health probes

Cause

Custom health probes give you more flexibility than the default probing behavior. When you use custom probes, you can set the probe interval, the URL, the path to test, and how many failed responses to accept before marking the backend pool instance as unhealthy.

The following table describes the additional properties you can set.

Solution

Validate that you configured the custom health probe correctly, as shown in the preceding table. In addition to the preceding troubleshooting guidance, follow this guidance as well.

• Ensure that you specify the probe correctly as per the guide.
• If you configure the application gateway for a single site, specify the default host name as 127.0.0.1, unless you configure this otherwise in the custom probe.
• Ensure that a call to http://<host>:<port><path> returns an HTTP result code of 200.
• Ensure that Interval, Timeout, and UnhealthyThreshold values are within the acceptable ranges.
• If you use an HTTPS probe in the v2 SKU, Application Gateway also sends the probe host name as the SNI value. Make sure that the host name matches the backend certificate's subject alternative name (SAN), or its common name (CN) if the certificate has no SAN, as described in RFC 6125. Application Gateway doesn't send SNI when the host name is an IP address, including the default 127.0.0.1. For steps to compare the probe host name against the certificate, see Resolution F in Troubleshoot HTTP 502 errors in Azure Application Gateway. If you can't change the backend certificate, adjust the backend HTTPS validation settings to use a specific SNI value or to skip subject name validation.

Backend request time-out is exceeded

Cause

When the application gateway receives a user request, it applies the configured rules to the request and routes it to a backend pool instance. It waits for a configurable interval of time for a response from the backend instance. By default, this interval is 20 seconds. In Application Gateway v1, if the application gateway doesn't receive a response from the backend application within this interval, the user request gets a 502 error. In Application Gateway v2, if the application gateway doesn't receive a response from the backend application within this interval, the request is tried against a second backend pool member. If the second request also fails, the user request gets a 504 error instead. If users receive 504 errors instead of 502 errors, see Troubleshoot HTTP 504 errors in Azure Application Gateway.

If your backend application routinely takes longer than the configured interval to respond (for example, a long-running query or report), increase the request time-out on the backend HTTP setting used by that backend pool.

Solution

The request time-out is a property of the backend HTTP setting, so different backend pools can have different request time-out values. Increase it by using the Azure portal, Azure CLI, or Azure PowerShell. The accepted range is 1 to 86400 seconds. For more information, see Request timeout.

Azure portal

Follow these steps:

1. In the Azure portal, go to your application gateway.
2. Under Settings, select Backend settings.
3. Select the backend setting used by the affected backend pool.
4. Increase the Request time-out (seconds) value to allow more time for the backend to respond.
5. Select Save.

Azure CLI

Run the following command in Azure CLI:

az network application-gateway http-settings update \
  --gateway-name <application-gateway-name> \
  --resource-group <resource-group-name> \
  --name <backend-http-settings-name> \
  --timeout <time-out-in-seconds>

Azure PowerShell

Run the following command in Azure PowerShell:

New-AzApplicationGatewayBackendHttpSettings -Name 'Setting01' -Port 80 -Protocol Http -CookieBasedAffinity Enabled -RequestTimeout 60

Application gateway's backend pool isn't configured or is empty

Cause

If the application gateway has no VMs or virtual machine scale set configured in the backend address pool, it can't route any customer request and sends a bad gateway error.

Solution

Ensure that the backend address pool isn't empty. You can check this condition through Azure PowerShell, Azure CLI, or the portal. See the following example to check the backend address pool through Azure PowerShell.

Get-AzApplicationGateway -Name "SampleGateway" -ResourceGroupName "ExampleResourceGroup"

The output from the preceding cmdlet should contain a nonempty backend address pool. The following example shows two returned pools that are configured with an FQDN or IP addresses for the backend VMs. The provisioning state of the BackendAddressPool must be Succeeded.

[{
    "BackendAddresses": [{
        "ipAddress": "10.0.0.10",
        "ipAddress": "10.0.0.11"
    }],
    "BackendIpConfigurations": [],
    "ProvisioningState": "Succeeded",
    "Name": "Pool01",
    "Etag": "W/\"00000000-0000-0000-0000-000000000000\"",
    "Id": "/subscriptions/<subscription id>/resourceGroups/<resource group name>/providers/Microsoft.Network/applicationGateways/<application gateway name>/backendAddressPools/pool01"
}, {
    "BackendAddresses": [{
        "Fqdn": "xyx.cloudapp.net",
        "Fqdn": "abc.cloudapp.net"
    }],
    "BackendIpConfigurations": [],
    "ProvisioningState": "Succeeded",
    "Name": "Pool02",
    "Etag": "W/\"00000000-0000-0000-0000-000000000000\"",
    "Id": "/subscriptions/<subscription id>/resourceGroups/<resource group name>/providers/Microsoft.Network/applicationGateways/<application gateway name>/backendAddressPools/pool02"
}]

Unhealthy instances in BackendAddressPool

Cause

If all the instances of BackendAddressPool are unhealthy, the application gateway doesn't have any backend to route user requests to. This condition can also occur when backend instances are healthy but don't have the required application deployed.

Solution

Ensure that the instances are healthy and the application is properly configured. Check if the backend instances can respond to a ping from another VM in the same virtual network. If you configure a public endpoint, ensure a browser request to the web application is serviceable.

Upstream SSL certificate doesn't match

Cause

The Transport Layer Security (TLS) certificate installed on backend servers doesn't match the hostname received in the HTTP host request header.

In scenarios where end-to-end TLS is enabled, achieve the configuration by editing the appropriate Backend HTTP settings. Change the configuration of the Backend protocol setting to HTTPS where necessary. Ensure that the DNS NAME of the TLS certificate installed on backend servers matches the hostname coming to the backend in the HTTP host header request.

When you complete this step, the second part of the communication that happens with Application Gateway and the backend servers is encrypted with TLS.

By default, Application Gateway sends the same HTTP host header to the backend as it receives from the client. Ensure that the TLS certificate installed on the backend server is issued with a DNS NAME that matches the host name received by that backend server in the HTTP host header. This hostname should be the same as the one received from the client.

Solution

For steps to align the probe host name and SNI with the certificate's SAN or CN, see Resolution F in Troubleshoot HTTP 502 errors in Azure Application Gateway. For how the Pick host name from backend target and Override with specific domain name backend settings map to the certificate name that Application Gateway expects, see Common Name (CN) doesn't match.