Azure Firewall Application Rules Processing

Intro

When Azure firewall processes application rules, what happens? Does it SNAT? Does it Proxy? Does it do both? In this post we will find out.

Terraformo 🪄

I used my wizarding skills to build this lab. If you want to follow along, you can find the incantation here.

Architecture

The following high level architecture diagram is referenced in this post.

The following points describe the above diagram:

• The network is deployed in a Hub and Spoke topology.
• The Hub vNet contains an Azure firewall which is the default gateway for all spoke subnets.
• Azure firewall has a single Public IP address assigned to facilitate communication to/from the internet.
• Spoke vNets are peered with the hub via vNeT peering.
• Spoke VMs are assinged RFC1918 Private IP addresses only.
• Egress/Ingress traffic to/from the internet traverses the Azure firewall.
• Spoke-to-Spoke traffic traverses the Azure firewall.
• Private DNS Zones allow DNS resolution internal to Azure.
• Key Vault stores the TLS inspection certificate.

IP Addressing

Prefixes

IP Addresses

Routing

The routing for this lab is depicted in the following diagram.

The following points describe the above diagram:

• Traffic to/from the internet traverses the Azure firewall.
• Outbound spoke traffic to the internet is Source NAT'd (SNAT) to 4.197.152.210
• Inbound SSH traffic from $MY_PUBLIC_IP to 4.197.152.210 is Destination NAT'd (DNAT) to the appropriate host.
• Inter-spoke vNet traffic transits via the hub vNet.
• Azure firewall is the default gateway for all spoke subnets.
• Intra-spoke vNet traffic stays within the spoke vNet.

DNS

To allow TLS inspection to work seemlessly without throwing scary warnings to clients we need to setup a few things, the first of which is DNS.

DNS Architecture

The following diagram shows the DNS architecture for this lab.

The components of the above design are explained in the following sections.

Domains

There are two domains used throughout this lab.

Private DNS Zones

Private DNS zones are deployed to allow Azure resources to resolve domain names within Azure.

• Both the stuffandthings.internal and stratuslabs.net Private Zones are deployed.
• The private DNS zones are linked to the Hub vNet.
• The Hub vNet is configured to use the Azure DNS resolver, this allows the records from the private DNS zones to be resolved by Azure firewall.

Public DNS Zone

The stratuslabs.net domain is registered publicly with Cloudflare.

• We need a publicly registered domain to capture traffic from both sides of the TLS connection (More on this later).
• No public A records are defined for the hosts in this lab.

Host Records

The following host records are added to the private DNS zones.

Domain Names

Virtual machines have the following domain names.

Proxy DNS

When utilising Azure firewall rules with FQDNs, it is recommended that the spoke resources use the same DNS server(s) as the Azure firewall to resolve hostnames.

The following configurations are enabed to meet this requirement.

• Azure Firewall has Proxy DNS enabled. This allows the virtual machines to use the Azure Firewall as their DNS server.
• Spoke vNets are configured to use the Azure firewall as the DNS server. The vNet DNS config is inherited by the spoke virtual machines when they boot up.
• When DNS requests are received, they are forwarded to the Azure DNS resolver 168.63.129.16.
• The Azure DNS resolver looks for records in Private DNS Zones attached to the hub vNet.

Certificates

To perform TLS inspection, x509 certificates need to be setup on both the client and servers with a mutual trusted Root Certificate Authority (CA). This lab uses both Self-Signed and Public CA certificates.

Chain of Trust

To establish a trust between a client and server, the server needs to present a certificate signed by a Root CA that the client also trusts.

In this lab we will setup a chain of trust as follows:

Root CA Certificate
|
+--> Intermediate CA Certificate
     |
     +--> Leaf (Server) Certificate

Some important points about the Certificate Chain:

• Root CA - Signs Intermediate certificate and delegates the signer attribute. This allows the Intermediate CA to generate TLS certificates and man in the middle TLS connections.
• Intermediate CA - When a request transits the firewall, a Leaf certificate for the destination is generated on the fly and signed by the Intermediate CA certificate.
• The Root CA's public certificate must be installed on the client. Any server certificate signed by the intermediate CA will be trusted by the client as they share a common Root CA.

TLS Inspection Certificate Requirements

To perform TLS inspection, the Intermediate certificate must have the following configuration parameters:

• Must be password-less and in the PKCS#12 format (with a certificate and a private key).
• Must be a single certificate, and not include the full chain of certificates.
• Must have an RSA private key >= 4096 bytes.
• The CA flag, must be set to True.
• Must have the KeyUsage extension marked as Critical with the KeyCertSign flag.
• Must have the BasicConstraints extension marked as Critical.
• Must be non-exportable, with the Path Length set to >=1.

Certificate Generation

Self-Signed Certificates

You can use the openssl CLI tool to create the necessecary certificates.

The following openssl.cnf file is used to define the correct configration parameters for the Root and Intermediate CA certificates.

[ req ]
default_bits        = 4096
distinguished_name  = req_distinguished_name
string_mask         = utf8only
default_md          = sha512

[ req_distinguished_name ]
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

[ rootCA_ext ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ interCA_ext ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:1
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ server_ext ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:false
keyUsage = critical, digitalSignature
extendedKeyUsage = serverAuth

Create the Root CA certificate.

# Create a Root CA private key and certificate.
openssl req -x509 -new -nodes \
    -newkey rsa:4096 -keyout rootCA.key \
    -sha256 -days 1024 -out rootCA.crt \
    -subj "/C=AU/ST=Queensland/O=StuffandThings/CN=StuffandThings Root CA" \
    -config openssl.cnf -extensions rootCA_ext"

Create the Intermediate CA certificate.

# Create intermediate CA private key and signing request.
openssl req -new -nodes -newkey rsa:4096 \
    -keyout interCA.key -sha256 -out interCA.csr \
    -subj "/C=AU/ST=Queensland/O=StuffandThings/CN=StuffandThings Intermediate CA"

# Sign on the intermediate CA request with the root CA producing an intermediate certificate.
openssl x509 -req -in interCA.csr \
    -CA rootCA.crt -CAkey rootCA.key \
    -CAcreateserial -out interCA.crt \
    -days 1024 -sha256 \
    -extfile openssl.cnf -extensions interCA_ext

# Export the intermediate CA into PFX format, excluding a password.
# The `interCA.pfx` file will be imported into Azure Key Vault.
openssl pkcs12 -export -out interCA.pfx \
    -inkey interCA.key -in interCA.crt \
    -password "pass:"

Create the certificates for websevers 1 and 2.

# Create webserver certificates.
for i in {1..2}
do
  # Generate a webserver private key.
  openssl genrsa -out webserver$i.key 2048

  # Create a certificate signing request.
  openssl req -new -key webserver$i.key -out webserver$i.csr \
    -subj "/C=AU/ST=Queensland/L=Brisbane/O=StuffandThings/CN=webserver$i.stuffandthings.internal" \
    -addext "subjectAltName=DNS:webserver$i.stuffandthings.internal"

  # Sign the webserver cert request with the intermediate CA producing a webserver certificate.
  openssl x509 -req -in webserver$i.csr -CA interCA.crt -CAkey interCA.key -CAcreateserial \
    -out webserver$i-alone.crt -days 365 -sha256 -extfile <(printf "subjectAltName=DNS:webserver$i.stuffandthings.internal")

  # Create a full-chain server certificate including the intermediate and root CA's.
  cat webserver$i-alone.crt interCA.crt rootCA.crt > webserver$i.crt
done

Public Certificate (Cloudflare)

I use Clouldflare for this lab as I host the stratuslabs.net domain there, and there's a Let's Encrypt plugin that makes requesting certificates a breeze. We need to generate an API Token to allow webserver3 to request a server certificate.

From your Cloudflare console, navigate to:

As a minimum, configure the following parameters.

Permissions

Zone Resources

Let's Encrypt Certificate

I won't cover the steps to request a certificate in detail, however, I can recommend this excellent blog post for additional details. The TL/DR is below.

On webserver3, switch to the root user and create a .secrets directory and cloudflare.ini file.

# Chanage to root user
sudo su -

# Create directory and file.
mkdir /root/.secrets && touch /root/.secrets/cloudflare.ini"

# Update permissions to super-duper-secret.
chmod 0700 /root/.secrets && sudo chmod 0400 /root/.secrets/cloudflare.ini

Add your Cloudflare API token to the cloudflare.ini file.

dns_cloudflare_api_token="${CLOUDFLARE_API_TOKEN}"

Generate a Let's Encrypt wildcard certificate for webserver3.

sudo certbot certonly --agree-tos --no-eff-email -m ${EMAIL}@${DOMAIN} --dns-cloudflare --dns-cloudflare-credentials /root/.secrets/cloudflare.ini -d 'webserver3.stratuslabs.net'

The certificate and key will be created in the following locations.

Key Vault

Azure Key Vault can securely store Keys (Software and HSM), Certificates and Secrets. In this lab, we use key vault to store the previously created Intermediate CA certificate.

The following diagram outlines the Azure Key Vault deployment.

The above diagram can be described as follows:

• The Intermediate Certificate: interCA.pfx is saved in Azure Key Vault in the PKCS#12 format.
• The certificate has 2 parts: The public certificate, stored in Certificates, and the private key, stored in Secrets.
• A Managed Identity is assigned to the firewall policy and used to access the certificate.

Managed Identitity

To access the certificate, Azure firewall needs access to the Key Vault. To allow this to happen, a User Assigned Managed Identity is configured with the following permissions:

• Key Vault Certificate User
• Key Vault Secrets User

The identity is then attached to the Azure firewall policy.

Firewall Policy

Most of the Azure firewall configuration, is done within a Firewall Policy object.

TLS Inspection

TLS inspection is enabled at the policy level. When enabled, you select the Key Vault where the Intermediate CA certificate is stored.

DNAT Rules

I have configured destination NAT to the virtual machines which allows me to SSH to them from my local network. No other DNAT is configured.

Network Rules

Network rules only permit ICMP traffic from RFC1918 private addresses. This is to aid in connectivity troubleshooting within Azure.

Application Rules

For the application rules, HTTP/S is permitted to the hosts/domains we want to test.

The Any/Any rule exists mostly to allow the webservers to install the required packages via APT. For the purpose of this lab and to simplify the rules, I permitted all other HTTP/S (with TLS inspection disabled).

Rule Processing Precedence

It's important to understand the order in which Azure firewalls process rules.

1. Rules and collections are numbered 100-65500, with lower numbers having higer priority.
2. Rule collection groups are processed in priority order.
3. Rule collections are processed in priority order.
4. DNAT rules.
5. Network rules.
6. Application rules.

Webserver Configuration

I am using Nginx as the webserver for this lab. I have enabled a very basic configuration to allow testing for both HTTP and HTTPS traffic.

Certificate and Private Key Location

Nginx will look for a Certificate and Private key in the following locations:

Nginx Configuration

The configuration file is located at /etc/nginx/sites-available/default and is used for all 3 webservers.

server {
    listen 80 default_server;
    server_name _;
    root /var/www/html;
    index index.html;
    location / {
        try_files $uri $uri/ =404;
    }
}

server {
    listen 443 ssl;
    ssl_certificate ${PUBLIC_CERTIFICATE_LOCATION}; # CHANGE_ME
    ssl_certificate_key ${PRIVATE_KEY_LOCATION}; # CHANGE_ME
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    root /var/www/html;
    index index.html;
    location / {
        try_files $uri $uri/ =404;
    }
}

Application Rule Considerations

Important information for Azure firewall application rule procesing:

• Application rules are evaluated in the outbound direction only.
• If required, Web Application Firewall (WAF) can be used to perform application filtering inbound.
• TLS inspection is an application rule construct, therefore is performed in the outbound direction only. If inbound inspection is required, then an Application Gateway or Front Door is required.
• Unlike Application Gateway, you cannot tell Azure Firewall to trust a certificate that is not issued by a well-known CA. This effectively means you can only perform TLS inspection on traffic destined for sites with certificates issued by a well-known CA.
• For HTTP, rules are matched on the Host header.
• For HTTPS, rules are matched on the Sever Name Indication (SNI) only.
• In the cases of HTTP and TLS inspected HTTPS:
  • Azure firewall ignores the packet's destination IP address and uses the DNS resolved IP address from the Host header/SNI.
  • Azure firewall inserts an X-Forwarded-For header with the source clients IP Address.

Test Cases

The follow considerations are pertinent to test cases:

• The tests goal is to determine how Azure firewall application rules process HTTP/S traffic.
• Traffic is captured via tcpdump.
• Where possible, traffic is captured on both sides of the firewall (Client and Server).
• Client traffic is captured for every test case.
• On the server side capture, traffic filters capture traffic from/to both the client and hub vNets.
• The client IP address is 192.168.0.4 for all test cases.

The following table summarises the test case parameters.

Each test case has a record of the commands run to execute the test which include:

• tcpdump commands to capture traffic on the client and server (where applicable).
• curl command used to initiate traffic to the webservers.

The data caputured for the test cases can be found UPDATE_ME.

Client -> webserver1.stuffandthings.internal

HTTP

HTTPS

Client -> webserver2.stuffandthings.internal

HTTP

HTTPS

Error

Azure firewall does not recognize the Root CA that issued the server certificate, therefore the client receives an error: certificate signed by unknown authority. The error comes from Azure firewall.

Client -> webserver3.stratuslabs.net

HTTP

HTTPS

Client -> google.com

HTTP

HTTPS

Client -> neverssl.com

HTTP

HTTPS

Client -> idontexist.intheether

A host entry was added to the client for the idontexist.intheether domain. This simulates what occurs when Azure Firewall cannot resolve a domain name.

HTTP

HTTPS

Test Results

Finally, let's determine what happens with NAT/Proxy when application rules are processed.

Plain-Text HTTP Traffic

TLS Inspected HTTPS

Non-TLS Inspected HTTPS

Non-TLS inspected HTTPS traffic is Source NAT'd (SNAT). Azure firewall cannot terminate the HTTPS connection, without triggering errors on the client. Therefore, SNAT at the network layer is the only option. The TCP and HTTP session is established between the Client and Server.

Unknown Domain Name

When Azure firewall cannot resolve the Domain Name the follow occurs.

1. TCP request to Destination IP.
2. AZFW receives TCP connection.
3. AZFW responds to TCP connection.
5. Client sends a HTTP request.
6. AZFW looks up domain from Host Header/SNI and resolves destination hostname.
7. AZFW, unable to resolve hostname sends HTTP/500 to client.
8. Client receives HTTP response.

Summary

Theory

Azure firewall is doing more than SNAT, it's also acting like a Transparent Full Proxy. To quote some Azure documentation: "Application rules are always SNATed using a transparent proxy."^[*]

It looks like AZFW buffers packets to inspect the HTTP Header/SNI to determine if it needs to process the packet via the application rules engine. If so, it responds to the source TCP session, spoofing the destination address, then initiates a new connection to the destination. This behaviour is evident when a Domain cannot be resolved by Azure firewall, the client gets a response from the Azure firewall, when the connection would normally timeout.

The test results are summarised in the table below:

Outro

In this post, we took a deep dive into Azure Firewall Application Rule processing. It was a pretty long post and if you got this far, thanks for hanging in there.

✌️ Peace out nerds. Stay weird! ✌️

Links

https://learn.microsoft.com/en-us/azure/firewall/premium-certificates

https://learn.microsoft.com/en-us/azure/firewall/premium-deploy-certificates-enterprise-ca

https://medium.com/@TakuSewo/tls-inspection-through-application-gateway-azure-firewall-and-web-server-how-to-and-tips-8f95e7dc6595

https://hovermind.com/azure-firewall/tls-inspection.html

https://tcude.net/enabling-https-for-your-internal-nginx-instance/

https://learn.microsoft.com/en-us/azure/firewall/rule-processing

https://learn.microsoft.com/en-us/azure/web-application-firewall/overview

https://techcommunity.microsoft.com/blog/azurenetworksecurityblog/azure-firewall-nat-behaviors/3825834

https://blog.cloudtrooper.net/2025/02/10/private-link-reality-bites-azure-firewall-app-rules/

https://learn.microsoft.com/en-us/azure/firewall/fqdn-filtering-network-rules

https://techcommunity.microsoft.com/blog/azurenetworksecurityblog/azure-firewall-new-features-and-region-availability/3911830

https://learn.microsoft.com/en-us/azure/firewall/snat-private-range

https://hovermind.com/azure-firewall/concepts.html

https://learn.microsoft.com/en-us/azure/firewall-manager/private-link-inspection-secure-virtual-hub

https://docs.azure.cn/en-us/firewall/snat-private-range

Tags