Azure Firewall Application Rules Processing

Intro

Recently, we had a discussion about Azure firewall rules in the AONE discord chat. When Azure firewall processes HTTP/S traffic via application rules, what happens to the traffic? Does it get SNAT'd? Does it get Proxied? Does it get Snoxied?

The Azure SDN handles network traffic in non-traditional magical ways. I was curious to understand what is actually happening, so in this post we will dive deep into the mystrious world of Azure firewall application rule processing for HTTP/S traffic.

Terraformo 🪄

I used my wizarding skills to build this lab. If you want to follow along, you can find the incantation here.

Architecture

The following high level architecture diagram is referenced in this post.

+−Reset

✕

The following points describe the above diagram:

• The network is deployed in a Hub and Spoke topology.
• The Hub vNet contains an Azure firewall which is the default gateway for all spoke subnets.
• Azure firewall has a single Public IP address assigned to facilitate communication to/from the internet.
• Spoke vNets are peered with the hub via vNeT peering.
• Spoke VMs are assinged RFC1918 Private IP addresses only.
• Egress/Ingress traffic to/from the internet traverses the Azure firewall.
• Spoke-to-Spoke traffic traverses the Azure firewall.
• Private DNS Zones allow DNS resolution internal to Azure.
• Key Vault stores the TLS inspection certificate.

IP Addressing

Prefixes

IP Addresses

Routing

The routing for this lab is depicted in the following diagram.

+−Reset

✕

The following points describe the above diagram:

• Traffic to/from the internet traverses the Azure firewall.
• Outbound spoke traffic to the internet is Source NAT'd (SNAT) to 4.197.152.210
• Inbound SSH traffic from $MY_PUBLIC_IP to 4.197.152.210 is Destination NAT'd (DNAT) to the appropriate host.
• Inter-spoke vNet traffic transits via the hub vNet.
• Azure firewall is the default gateway for all spoke subnets.
• Intra-spoke vNet traffic stays within the spoke vNet.

DNS

To allow TLS inspection to work seemlessly without throwing scary warnings to clients we need to setup a few things, the first of which is DNS.

DNS Architecture

The following diagram shows the DNS architecture for this lab.

+−Reset

✕

The components of the above design are explained in the following sections.

Domains

There are two domains used throughout this lab.

Private DNS Zones

Private DNS zones are deployed to allow Azure resources to resolve domain names within Azure.

• Both the stuffandthings.internal and stratuslabs.net Private Zones are deployed.
• The private DNS zones are linked to the Hub vNet.
• The Hub vNet is configured to use the Azure DNS resolver, this allows the records from the private DNS zones to be resolved by Azure firewall.

Public DNS Zone

The stratuslabs.net domain is registered publicly with Cloudflare.

• We need a publicly registered domain to capture traffic from both sides of the TLS connection (More on this later).
• No public A records are defined for the hosts in this lab.

Host Records

The following host records are added to the private DNS zones.

Domain Names

Virtual machines have the following domain names.

Proxy DNS

When utilising Azure firewall rules with FQDNs, it is recommended that the spoke resources use the same DNS server(s) as the Azure firewall to resolve hostnames.

The following configurations are enabed to meet this requirement.

• Azure Firewall has Proxy DNS enabled. This allows the virtual machines to use the Azure Firewall as their DNS server.
• Spoke vNets are configured to use the Azure firewall as the DNS server. The vNet DNS config is inherited by the spoke virtual machines when they boot up.
• When DNS requests are received, they are forwarded to the Azure DNS resolver 168.63.129.16.
• The Azure DNS resolver looks for records in Private DNS Zones attached to the hub vNet.

Certificates

To perform TLS inspection, x509 certificates need to be setup on both the client and servers with a mutual trusted Root Certificate Authority (CA). This lab uses both Self-Signed and Public CA certificates.

Chain of Trust

To establish a trust between a client and server, the server needs to present a certificate signed by a Root CA that the client also trusts.

In this lab we will setup a chain of trust as follows:

Root CA Certificate
|
+--> Intermediate CA Certificate
     |
     +--> Leaf (Server) Certificate

Some important points about the Certificate Chain:

• Root CA - Signs Intermediate certificate and delegates the signer attribute. This allows the Intermediate CA to generate TLS certificates and man in the middle TLS connections.
• Intermediate CA - When a request transits the firewall, a Leaf certificate for the destination is generated on the fly and signed by the Intermediate CA certificate.
• The Root CA's public certificate must be installed on the client. Any server certificate signed by the intermediate CA will be trusted by the client as they share a common Root CA.

TLS Inspection Certificate Requirements

To perform TLS inspection, the Intermediate certificate must have the following configuration parameters:

• Must be password-less and in the PKCS#12 format (with a certificate and a private key).
• Must be a single certificate, and not include the full chain of certificates.
• Must have an RSA private key >= 4096 bytes.
• The CA flag, must be set to True.
• Must have the KeyUsage extension marked as Critical with the KeyCertSign flag.
• Must have the BasicConstraints extension marked as Critical.
• Must be non-exportable, with the Path Length set to >=1.

Certificate Generation

Self-Signed Certificates

You can use the openssl CLI tool to create the necessecary certificates.

The following openssl.cnf file is used to define the correct configration parameters for the Root and Intermediate CA certificates.

[ req ]
default_bits        = 4096
distinguished_name  = req_distinguished_name
string_mask         = utf8only
default_md          = sha512

[ req_distinguished_name ]
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

[ rootCA_ext ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ interCA_ext ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:1
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ server_ext ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:false
keyUsage = critical, digitalSignature
extendedKeyUsage = serverAuth

Create the Root CA certificate.

# Create a Root CA private key and certificate.
openssl req -x509 -new -nodes \
    -newkey rsa:4096 -keyout rootCA.key \
    -sha256 -days 1024 -out rootCA.crt \
    -subj "/C=AU/ST=Queensland/O=StuffandThings/CN=StuffandThings Root CA" \
    -config openssl.cnf -extensions rootCA_ext"

Create the Intermediate CA certificate.

# Create intermediate CA private key and signing request.
openssl req -new -nodes -newkey rsa:4096 \
    -keyout interCA.key -sha256 -out interCA.csr \
    -subj "/C=AU/ST=Queensland/O=StuffandThings/CN=StuffandThings Intermediate CA"

# Sign on the intermediate CA request with the root CA producing an intermediate certificate.
openssl x509 -req -in interCA.csr \
    -CA rootCA.crt -CAkey rootCA.key \
    -CAcreateserial -out interCA.crt \
    -days 1024 -sha256 \
    -extfile openssl.cnf -extensions interCA_ext

# Export the intermediate CA into PFX format, excluding a password.
# The `interCA.pfx` file will be imported into Azure Key Vault.
openssl pkcs12 -export -out interCA.pfx \
    -inkey interCA.key -in interCA.crt \
    -password "pass:"

Create the certificates for websevers 1 and 2.

# Create webserver certificates.
for i in {1..2}
do
  # Generate a webserver private key.
  openssl genrsa -out webserver$i.key 2048

  # Create a certificate signing request.
  openssl req -new -key webserver$i.key -out webserver$i.csr \
    -subj "/C=AU/ST=Queensland/L=Brisbane/O=StuffandThings/CN=webserver$i.stuffandthings.internal" \
    -addext "subjectAltName=DNS:webserver$i.stuffandthings.internal"

  # Sign the webserver cert request with the intermediate CA producing a webserver certificate.
  openssl x509 -req -in webserver$i.csr -CA interCA.crt -CAkey interCA.key -CAcreateserial \
    -out webserver$i-alone.crt -days 365 -sha256 -extfile <(printf "subjectAltName=DNS:webserver$i.stuffandthings.internal")

  # Create a full-chain server certificate including the intermediate and root CA's.
  cat webserver$i-alone.crt interCA.crt rootCA.crt > webserver$i.crt
done

Public Certificate (Cloudflare)

I use Clouldflare for this lab as I host the stratuslabs.net domain there, and there's a Let's Encrypt plugin that makes requesting certificates a breeze. We need to generate an API Token to allow webserver3 to request a server certificate.

From your Cloudflare console, navigate to:

As a minimum, configure the following parameters.

Permissions

Zone Resources

Let's Encrypt Certificate

I won't cover the steps to request a certificate in detail, however, I can recommend this excellent blog post for additional details. The TL/DR is below.

On webserver3, switch to the root user and create a .secrets directory and cloudflare.ini file.

# Chanage to root user
sudo su -

# Create directory and file.
mkdir /root/.secrets && touch /root/.secrets/cloudflare.ini"

# Update permissions to only permit root user access.
chmod 0700 /root/.secrets && sudo chmod 0400 /root/.secrets/cloudflare.ini

Add your Cloudflare API token to the cloudflare.ini file.

dns_cloudflare_api_token="${CLOUDFLARE_API_TOKEN}"

Generate a Let's Encrypt certificate for webserver3.

sudo certbot certonly --agree-tos --no-eff-email -m ${EMAIL}@${DOMAIN} --dns-cloudflare --dns-cloudflare-credentials /root/.secrets/cloudflare.ini -d 'webserver3.stratuslabs.net'

The certificate and key will be created in the following locations.

Key Vault

Azure Key Vault can securely store Keys (Software and HSM), Certificates and Secrets. In this lab, we use key vault to store the previously created Intermediate CA certificate.

The following diagram outlines the Azure Key Vault deployment.

+−Reset

✕

The above diagram can be described as follows:

• The Intermediate Certificate: interCA.pfx is saved in Azure Key Vault in the PKCS#12 format.
• The certificate has 2 parts: The public certificate, stored in Certificates, and the private key, stored in Secrets.
• A Managed Identity is assigned to the firewall policy and used to access the certificate.

Managed Identitity

To access the certificate, Azure firewall needs access to the Key Vault. To allow this to happen, a User Assigned Managed Identity is configured with the following permissions:

• Key Vault Certificate User
• Key Vault Secrets User

The identity is then attached to the Azure firewall policy.

Firewall Policy

Most of the Azure firewall configuration, is done within a Firewall Policy object.

TLS Inspection

TLS inspection is enabled at the policy level. When enabled, you select the Key Vault where the Intermediate CA certificate is stored.

DNAT Rules

I have configured destination NAT to the virtual machines which allows me to SSH to them from my local network. No other DNAT is configured.

Network Rules

Network rules only permit ICMP traffic from RFC1918 private addresses. This is to aid in connectivity troubleshooting within Azure.

Application Rules

For the application rules, HTTP/S is permitted to the hosts/domains we want to test.

The Any/Any rule exists mostly to allow the webservers to install the required packages via APT. For the purpose of this lab and to simplify the rules, I permitted all other HTTP/S (with TLS inspection disabled).

Rule Processing Precedence

It's important to understand the order in which Azure firewalls process rules.

1. Rules and collections are numbered 100-65500, with lower numbers having higer priority.
2. Rule collection groups are processed in priority order.
3. Rule collections are processed in priority order.
4. DNAT rules.
5. Network rules.
6. Application rules.

Application Rule Considerations

Important information for Azure firewall application rule procesing:

• Application rules are evaluated in the outbound direction only.
• If required, Web Application Firewall (WAF) can be used to perform application filtering inbound.
• TLS inspection is an application rule construct, therefore is performed in the outbound direction only. If inbound inspection is required, then an Application Gateway or Front Door is required.
• Unlike Application Gateway, you cannot tell Azure Firewall to trust a certificate that is not issued by a well-known CA. This effectively means you can only perform TLS inspection on traffic destined for sites with certificates issued by a well-known CA.
• For HTTP, rules are matched on the Host header.
• For HTTPS, rules are matched on the Sever Name Indication (SNI) only.
• In the cases of HTTP and TLS inspected HTTPS:
  • Azure firewall ignores the packet's destination IP address and uses the DNS resolved IP address from the Host header/SNI.
  • Azure firewall inserts an X-Forwarded-For header with the source clients IP Address.

Webserver Configuration

I am using Nginx as the webserver for this lab. I have enabled a very basic configuration to allow testing for both HTTP and HTTPS traffic.

Certificate and Private Key Location

Nginx will look for a Certificate and Private key in the following locations:

Nginx Configuration

The configuration file is located at /etc/nginx/sites-available/default and is used for all 3 webservers.

server {
    listen 80 default_server;
    server_name _;
    root /var/www/html;
    index index.html;
    location / {
        try_files $uri $uri/ =404;
    }
}

server {
    listen 443 ssl;
    ssl_certificate ${PUBLIC_CERTIFICATE_LOCATION}; # CHANGE_ME
    ssl_certificate_key ${PRIVATE_KEY_LOCATION}; # CHANGE_ME
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    root /var/www/html;
    index index.html;
    location / {
        try_files $uri $uri/ =404;
    }
}

Test Cases

The follow considerations are pertinent to test cases:

• The tests goal is to determine how Azure firewall application rules process HTTP/S traffic.
• Traffic is captured via tcpdump.
• Where possible, traffic is captured on both sides of the firewall (Client and Server).
• Client traffic is captured for every test case.
• On the server side capture, traffic filters capture traffic from/to both the client and hub vNets.
• The client IP address is 192.168.0.4 for all test cases.

The following table summarises the test case parameters.

Each test case has a record of the commands run to execute the test which include:

• tcpdump commands to capture traffic on the client and server (where applicable).
• curl command used to initiate traffic to the webservers.

The data caputured for the test cases can be found here.

Client -> webserver1.stuffandthings.internal

HTTP

HTTPS

Client -> webserver2.stuffandthings.internal

HTTP

HTTPS

Client -> webserver3.stratuslabs.net

HTTP

HTTPS

Client -> google.com

HTTP

HTTPS

Client -> neverssl.com

HTTP

HTTPS

Client -> idontexist.intheether

A host entry was added to the client for the idontexist.intheether domain. This simulates what occurs when Azure Firewall cannot resolve a domain name.

HTTP

HTTPS

Test Results

Finally, let's determine what happens with NAT/Proxy when application rules are processed. When examining the packet captures to try and determine what is occuring, and in what order, I looked mainly at the following:

• Source and Destination IP addresses and ports
• TCP sequence numbers
• TCP/TLS handshakes
• TLS version numbers
• Timestamps

Plain-Text HTTP Traffic

The following diagram show plain-text HTTP traffic flow.

+−Reset

✕

1. Client initiates TCP connection to webserver.
2. Firewall intercepts the TCP connection.
3. Firewall responds to the TCP connection from the client spoofing the webserver IP address as the source address.
4. Client receives the TCP response from the firewall.
5. Client initiates HTTP request to the webserver.
6. Firewall intercepts the HTTP request, resolving the destination IP from the Host header.
7. Firewall opens a TCP connection to the webserver.
8. Webserver received TCP connection from the firewall.
9. Webserver responds to TCP connection from the firewall.
10. Firewall receives the TCP response from the webserver.
11. Firewall initiates HTTP request to the webserver, copying the client payload and inserting the X-Forwarded-For header with the client IP address.
12. Webserver receives HTTP request from the firewall.
13. Webserver sends HTTP response to the firewall.
14. Firewall receives HTTP response from the webserver.
15. Firewall sends HTTP response the client, copying the webserver payload and spoofing the webserver IP address as the source address.
16. Client receives HTTP response from the firewall.

Packet Capture

The below diagram shows a side-by-side packet capture of the traffic. Webserver capture is on the left and Client capture is on the right.

+−Reset

✕

Conclusion

Azure firewall acts as a transparent full proxy in this scenario.

TLS Inspected HTTPS

The following diagram shows a TLS inspected HTTPS traffic flow.

+−Reset

✕

1. Client initiates TCP connection to webserver.
2. Firewall intercepts the TCP connection.
3. Firewall responds to the TCP connection from the client spoofing the webserver IP address as the source address.
4. Client receives the TCP response from the firewall.
5. Client initiates TLS handshake with webserver.
6. Firewall inspects TLS handshake to determine the destination from the SNI field. Intercepting the TLS connection for inspected domains.
7. Firewall responds to TLS handshake from the client spoofing the webserver IP address as the source address.
8. Client receives TLS handsake response from the webserver.
9. Client initiates HTTPS request to the webserver.
10. Firewall intercepts the HTTPS request, resolving the destination IP using the SNI field from the TLS handshake.
11. Firewall opens a TCP connection to the webserver.
12. Webserver received TCP connection from the firewall.
13. Webserver responds to TCP connection from the firewall.
14. Firewall receives the TCP response from the webserver.
15. Firewall initiates TLS handshake with webserver.
16. Webserver receives TLS handshake from the firewall.
17. Webserver responds to TLS handshake.
18. Firewall receives TLS handshake response from webserver.
19. Firewall initiates HTTPS request to the webserver, copying the client payload and inserting the X-Forwarded-For header with the client IP address.
20. Webserver receives HTTPS request from the firewall.
21. Webserver sends HTTPS response to the firewall.
22. Firewall receives HTTPS response from the webserver.
23. Firewall sends HTTPS response the client, copying the webserver payload and spoofing the webserver IP address as the source address.
24. Client receives HTTPS response from the firewall.

Packet Capture

The below diagram shows a side-by-side packet capture of the traffic. Webserver capture is on the left and Client capture is on the right.

+−Reset

✕

Conclusion

Azure firewall acts as a transparent full proxy in this scenario.

Non-TLS Inspected HTTPS

The following diagram shows a Non-TLS inspected HTTPS traffic flow.

+−Reset

✕

1. Client initiates TCP connection to webserver.
2. Firewall intercepts the TCP connection.
3. Firewall responds to the TCP connection from the client spoofing the webserver IP address as the source address.
4. Client receives the TCP response from the firewall.
5. Client initiates TLS handshake with webserver.
6. Firewall buffers TLS handshake to determine the destination from the SNI field in the client hello message.
7. Firewall opens a TCP connection to the webserver.
8. Webserver received TCP connection from the firewall.
9. Webserver responds to TCP connection from the firewall.
10. Firewall receives the TCP response from the webserver.
11. Firewall forwards TLS connection to webserver which receives the TLS handshake from the client.
12. Webserver responds to TLS handshake from client.
13. Client receives TLS handshake response from webserver.
14. Client initiates HTTPS request to the webserver.
15. Webserver receives HTTPS request from the client.
16. Webserver sends HTTPS response to the client.
17. Client receives HTTPS response from the webserver.

Packet Capture

The below diagram shows a side-by-side packet capture of the traffic. Webserver capture is on the left and Client capture is on the right.

+−Reset

✕

Conclusion

Non-TLS inspected HTTPS traffic is Source NAT'd (SNAT). Azure firewall cannot terminate the HTTPS connection, without triggering errors on the client. Therefore, SNAT at the network layer is the only option. The TLS and HTTP sessions are established directly between the Client and Server.

Unknown Certificate Issuer

When Azure firewall does not trust the webserver Root CA certificate the following occurs.

+−Reset

✕

1. Client initiates TCP connection to webserver.
2. Firewall intercepts the TCP connection.
3. Firewall responds to the TCP connection from the client spoofing the webserver IP address as the source address.
4. Client receives the TCP response from the firewall.
5. Client initiates TLS handshake with webserver.
6. Firewall inspects TLS handshake to determine the destination from the SNI field. Intercepting the TLS connection for inspected domains.
7. Firewall responds to TLS handshake from the client spoofing the webserver IP address as the source address.
8. Client receives TLS handsake response from the webserver.
9. Client initiates HTTPS request to the webserver.
10. Firewall intercepts the HTTPS request, resolving the destination IP using the SNI field from the TLS handshake.
11. Firewall opens a TCP connection to the webserver.
12. Webserver received TCP connection from the firewall.
13. Webserver responds to TCP connection from the firewall.
14. Firewall receives the TCP response from the webserver.
15. Firewall initiates TLS handshake with webserver.
16. Webserver receives TLS handshake from the firewall.
17. Webserver responds to TLS handshake.
18. Firewall receives TLS handshake response from webserver and looks up the server certificate in the list of well-known certificate authorities. Which it does not find.
19. Firewall sends HTTPS response the client, spoofing the webserver IP address as the source address, with a 500 error: certificate signed by unknown authority.
20. Client receives HTTPS response from the firewall.

Packet Capture

The below diagram shows a side-by-side packet capture of the traffic. Webserver capture is on the left and Client capture is on the right.

+−Reset

✕

Conclusion

Azure firewall is unable to validate the webserver certificate issuer and returns an HTTP/500 error to the client.

Unknown Domain Name

When Azure firewall cannot resolve the Domain Name the follow occurs.

+−Reset

✕

1. Client initiates TCP connection to webserver.
2. Firewall intercepts the TCP connection.
3. Firewall responds to the TCP connection from the client spoofing the webserver IP address as the source address.
4. Client receives the TCP response from the firewall.
5. Client initiates HTTP request to the webserver.
6. Firewall intercepts the HTTP request, resolving the destination IP from the Host header.
7. The firewall sends a HTTP response the client, spoofing the webserver IP address as the source address, with a 500 error: Failed to resolve address
8. Client receives HTTP response from the firewall.

Packet Capture

The below diagram shows a packet capture of the Client traffic. There is no Webserver in this scenario.

+−Reset

✕

Conclusion

Azure firewall is unable to resolve the IP address of the destination and returns an HTTP/500 error to the client.

Summary

Most of the Azure documentation states that Application rules are always SNAT'd. However, it appears that Azure firewall is doing more than SNAT, in some scenarios it's also acting like a Transparent Full Proxy. I did find one reference in some Azure documentation which aligns to this theory: "Application rules are always SNATed using a transparent proxy."^[*]

It looks like Azure firewall buffers packets to inspect the HTTP Header/SNI to determine if it needs to process the packet via the application rules engine. If so, it responds to the source TCP session, spoofing the destination address, then initiates a new connection to the destination. This behaviour is most evident when a Domain cannot be resolved by Azure firewall, the client gets a HTTP response from the Azure firewall, when the connection would normally timeout.

Firewall Rules Processing

The following diagram shows the flow of traffic through the Azure firewall rules.

+−Reset

✕

The test results are summarised in the table below.

Outro

In this post, we took a deep dive into Azure Firewall Application Rule processing. It was a pretty long post and if you got this far, thanks for hanging in there.

✌️ Peace out nerds. Stay weird! ✌️

Links

https://learn.microsoft.com/en-us/azure/firewall/premium-certificates

https://learn.microsoft.com/en-us/azure/firewall/premium-deploy-certificates-enterprise-ca

https://medium.com/@TakuSewo/tls-inspection-through-application-gateway-azure-firewall-and-web-server-how-to-and-tips-8f95e7dc6595

https://hovermind.com/azure-firewall/tls-inspection.html

https://tcude.net/enabling-https-for-your-internal-nginx-instance/

https://learn.microsoft.com/en-us/azure/firewall/rule-processing

https://learn.microsoft.com/en-us/azure/web-application-firewall/overview

https://techcommunity.microsoft.com/blog/azurenetworksecurityblog/azure-firewall-nat-behaviors/3825834

https://blog.cloudtrooper.net/2025/02/10/private-link-reality-bites-azure-firewall-app-rules/

https://learn.microsoft.com/en-us/azure/firewall/fqdn-filtering-network-rules

https://techcommunity.microsoft.com/blog/azurenetworksecurityblog/azure-firewall-new-features-and-region-availability/3911830

https://learn.microsoft.com/en-us/azure/firewall/snat-private-range

https://hovermind.com/azure-firewall/concepts.html

https://learn.microsoft.com/en-us/azure/firewall-manager/private-link-inspection-secure-virtual-hub

https://docs.azure.cn/en-us/firewall/snat-private-range

Tags