Can't get to Azure VM from on-prem network, using Palo Alto VM firewall

Joe Grover 526

I have a test Azure virtual network set up with two virtual machines--a vendor's VM firewall and a Windows Server VM.

The virtual network has two subnets:

Hub subnet (has firewall's outside interface in it)
VM subnet (has firewall's inside interface in it, as well as the Windows Server VM that has its software firewall disabled)

I created a route table to direct traffic in VM subnet to the inside interface of the vendor VM firewall as the default gateway. After doing so the VM can get online fine. What I can't do is get traffic from the outside into the VM. I have the firewall policies set up the same as I have them on my on-prem firewall (albeit using the Azure private IP in the rules), and even tried allowing all traffic from our on-prem network's public IP to the Azure public IP of the firewall. I've had no luck.

At first, I thought this was just some weird bi-directional NAT thing to figure out on the firewall, but at the same time I wound up configuring a site-to-site VPN to the VM firewall from our on-prem network. I was able to get the tunnel up, and I can ping from the Windows Server VM through to my on-prem network...but I can't access the VM from on-prem over the VPN, just like I can't access it over the internet.

This makes me think that both issues have the same root cause, but I'm at a loss to figure it out. I opened up a support case with the vendor and spent the last few hours with them running captures on the on-prem firewall, the VM firewall, and the VM itself.

The traffic seems to be getting to the VM firewall's inside interface, but never makes it to the Windows VM itself. This is odd to me since the firewall can definitely see the VM (or the VM wouldn't be able to get online, or ping through the VPN tunnel to the on-prem network). All policies appear to be good--nothing is getting denied, it just is getting lost before it gets to the VM.

Has anyone seen this before or have any ideas of what to look at? Both myself and the vendor believe there's something in Azure that needs to be looked at/configured properly, but this is my first foray into Azure virtual networks so I'm not sure what to look at. Running the packet captures had us chasing our tails a bit because all of the Azure packets show as coming from 12:34:56:78:9a:bc, so we didn't know if those were the Windows VM or the firewall VM.

My primary concern is the VPN traffic--if I can't get that working, this is all a wash anyway. I need to find out what's happening to the traffic between the inside interface of my firewall and the virtual NIC of my server VM.

KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-09-23T06:07:16.74+00:00
@Joe Grover ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you are configuring a 3rd party firewall as NVA in Azure and you are not able to initiate traffic towards the VMs in the subnet via the NVA-Vm.

I take it that the VPN is also established using the NVA and not Azure's VPN Gateway - correct me if I am wrong.

Since you are using a 3rd party NVA, I would suggest you follow the recommended best practices or documentation from their end regarding the configuration to use as NVA in Azure.

With that said,

Can you please try the same with a VM that is not in the same subnet as the NVA-Vm's NIC

Fundamentally, you should not attach a UDR to a subnet which contains the nextHop

In your case, I see UDR is attached to the subnet of the NVA-Vm's NIC

Let us know how this goes.

Cheers,

Kapil
Joe Grover 526 Reputation points

2024-09-23T12:12:08.1066667+00:00
Yes, 3rd-party firewall as NVA.

VPN tunnel is established from on-prem to NVA.

I have followed best-practice from the vendor, and worked with them on Friday troubleshooting the issue. As far as they can see everything is configured properly, particularly because I can see the on-prem network from the server VM, just not the other way around. Something is preventing the packet from getting from the firewall VM to the server VM.

With regards to your note about UDR, what exactly do you mean? I'd set up the route table because it was the only way I could find to get the traffic to flow through the firewall NVA. Prior to that the subnet hands out X.X.X.1 as the gateway, but since that IP is reserved by Azure I couldn't set the inside interface of my firewall NVA to that IP. My research indicated that if I wanted to do so I had to create a route table, create a default route pointing to the IP of my firewall internal interface, and associate the route table with that subnet.. If there's another/better way to set the default gateway in a virtual subnet to one of my choosing, I'll do that instead.
KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-09-23T12:35:02.3866667+00:00
Thanks @Joe Grover, for the insights.

Your research is correct.

To route traffic to a NVA from a VM, create a route table, create a default route pointing to the IP of the NVA interface, and associate the route table with that subnet.

I am saying,

The VM and the NVA should not be in the same subnet

More precisely, NVA Subnet should not have any Route Table attached to it.

From your verbatim,

Hub subnet (has firewall's outside interface in it)

VM subnet (has firewall's inside interface in it, as well as the Windows Server VM that has its software firewall disabled)

Here, you mentioned NVA's interface as well as the VM are in the same subnet.

Please deploy the VM in a different subnet and attach the UDR here.

Do not attach any UDR to the subnet of the NVA

And let us know how it goes.

Cheers,

Kapil
Joe Grover 526 Reputation points

2024-09-23T13:48:49.87+00:00

I guess I'm confused by this setup.

A default gateway needs to be in the same subnet as the device using it, so I'm not sure how to follow your instructions.

For example, by default we'll say my VM had an IP of 192.168.1.10, and Azure assigned 192.168.1.1 as the default gateway. I needed my VM to go through my firewall VM, and I'd configured one of its interfaces in this same subnet as 192.168.1.250, then applied the route table to the subnet to send 0.0.0.0/0 to 192.168.1.250. This appeared to work for everything but my inbound VPN traffic, though from what you're saying is not what should have been done.

So, if I create a different subnet (such as 192.168.2.x), move the server VM there and give it 192.168.2.10, it will be given 192.168.2.1 as its default gateway. At this point, what do I need to do in order to get the traffic to go to my VM firewall inside interface (192.168.1.250)?
KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-09-23T14:10:33.56+00:00
@Joe Grover ,

Creating a UDR such as 0.0.0.0 -----> NextHop as a NVA is not same as a default gateway.

Wrt,

"So, if I create a different subnet (such as 192.168.2.x), move the server VM there and give it 192.168.2.10, it will be given 192.168.2.1 as its default gateway."

Correct

" At this point, what do I need to do in order to get the traffic to go to my VM firewall inside interface (192.168.1.250)?"

In the subnet 192.168.2.x, attach the Route Table that has the route 0.0.0.0 -----> NextHop NVA IP (192.168.1.250).

To be more precise, use OnPrem IP Address range.

i.e., <OnPrem IP Range> -----> NextHop NVA IP(192.168.1.250).

And make sure there is no Route Table at the NVA's subnet.

Can you please try this configuration and let us know.

Can you also please share the documentation from vendor side where they configured this in Azure?

Cheers,

Kapil
Joe Grover 526 Reputation points

2024-09-23T17:11:50.3633333+00:00
Thank you. I'm working on it now.

I have the outside interface of the firewall in a new vnet (WAN-vnet).

I have the inside interface of the firewall in the HUB-vnet (192.168.100.250), and configured it with a route directing traffic for 192.168.1.0/24 to the subnet's Azure default gateway of 192.168.100.1.

I have the Windows Server VM in the VM-vnet (192.168.1.4).

I edited the route table on the VM vnet, changing the next hop from 192.168.1.250 to 192.168.100.250 and associated it with the VM-vnet again.

I currently only have connectivity to the VM via the serial console, but the VM cannot get to the internet. Looking at the logs on the firewall I'm not seeing any hits from the VM to the inside interface of the firewall. I can ping the firewall interface from the server VM (and vice versa), at which time I do see activity on a packet capture, but I don't see anything if I try to ping out (say, to Google DNS server).

It's as if the route table isn't taking effect. For clarity, I created this route in the route table:
Name: To-Firewall
Address prefix: 0.0.0.0/0
Next hop Type: Virtual Appliance
Next hop IP address: 192.168.100.250

When going into the route to get the above information I did see when creating a route to a virtual appliance Azure has a note to ensure IP Forwarding is enabled. It was not enabled on this interface (the inside interface that 192.168.100.250 is configured on) so I tried turning it on, but it didn't seem to make a difference.
Joe Grover 526 Reputation points

2024-09-23T18:26:12.0066667+00:00
Thanks for the further information. So far I've done the following, though without success:

Created a new subnet and moved the firewall VM's outside interface into it.

Moved the firewall's inside interface from the VM subnet to the Hub subnet (which has no route tables associated with it) and gave it an IP in that subnet (192.168.100.250).

Changed the default route in the VM's subnet routing table to point to the virtual appliance at IP 192.168.100.250.

Using the serial console, reconfigured the server VM for DHCP and disabled/re-enabled the interface.

Confirmed the server can ping the firewall at 192.168.100.250, and the firewall can ping the server VM at 192.168.1.4. This is progress, as I was unable to ping the server VM from the firewall VM before.

However, I'm unable to get the server online through the firewall in this configuration. Using packet capture on the firewall I can see the traffic hitting it when I ping it, but if I try to ping something else (like 8.8.8.8) or try to get online to a website I don't see any traffic from the VM hitting the firewall. It's almost like it's ignoring the route in the route table (or at the very least, traffic is going to the azure gateway at 192.168.1.1 and going nowhere after that).

I did try enabling IP forwarding on the firewall's inside interface in the Azure IP interface config, but it didn't seem to make any difference.
Joe Grover 526 Reputation points

2024-09-23T19:53:39.6833333+00:00

(didn't mean to double post, it's been a while and the comment wasn't showing up, and looked like it was a draft on my end still).
KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-09-24T10:05:00.62+00:00
@Joe Grover ,

Thanks for sharing the details.

To check the effective nextHop of a VM, you can use Azure Network Watcher.

See : Test network communication using Network Watcher next hop

Can you please confirm if the nextHop here shows NVA's IP?

For OnPrem range and Internet IP Addresses?

You can also get a generic view of effective routes of a VM here

Cheers,

Kapil
Joe Grover 526 Reputation points

2024-09-24T11:51:43.27+00:00

Thank you.

The Next Hop does show the information from the route in my Route Table (next hop type VirtualAppliance, IP address 192.168.100.250), but the traffic never hits that .250 interface as far as I can see in packet captures on the firewall.
KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-09-24T12:21:19.55+00:00
@Joe Grover ,

This means packets are being routed correctly to the NVA.

Note that OS configuration (on Windows VM) will not be able to override the platform's routing.

It appears the packet capture tool at NVA is not capturing the packets being sent from the VM.

or the NVA's OS is dropping the packets and the tool is not able to see this.

Please check with the vendor on how this is to be configured.

Were you able to find any documentation from vendor side on how this is to be configured in Azure?

Cheers,

Kapil
Joe Grover 526 Reputation points

2024-09-24T12:25:42.3266667+00:00

I was not able to find any documentation from the vendor other than the initial deployment of the appliance via the Azure marketplace. I did not see anything regarding post-deployment configuration, which was why I reached out to open a support case with them initially. Essentially I've done this a bunch of times with these firewalls, but never in Azure, so I wasn't aware of the various Azure things that would get in my way, heh (route tables, etc).

I did update them with the changes you've had me make and asked to schedule time to look at the firewall VM again to ensure there isn't something else that needs to be changed to accommodate these changes. I'll let you know what I hear back, and thanks so much for what you've given me so far.
Joe Grover 526 Reputation points

2024-09-24T12:36:01.16+00:00

On other question right now: With the configuration I had before you'd indicated that I shouldn't have a UDR in the subnet with the inside/trust interface of the firewall VM in it. Is that a general rule, or just based on how I had things configured?

I ask because your comment about how the OS configuration won't override the system routing made me think. When I made the aforementioned changes (putting the inside firewall interface into a different subnet than the Windows VM), I added a route in the firewall VM telling it to go to 192.168.100.1 if it needed to go to 192.168.1.0/24.

What I did not do is create a route table for that subnet telling the subnet in general to direct traffic for 192.168.1.0/24 to 192.168.100.1. Should I? Or will the route in the firewall be enough? I used the Network Watcher to get the next hop and this was the result:
KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-09-25T06:42:59.37+00:00
@Joe Grover , You are most welcome.

Wrt, "With the configuration I had before you'd indicated that I shouldn't have a UDR in the subnet with the inside/trust interface of the firewall VM in it. Is that a general rule, or just based on how I had things configured?"

This is a general rule.

If the NVA had a UDR with nextHop pointing to itself, this will lead to a routing loop.

If you have two NVAs, say NVA1 and NVA2 and want traffic to pass to NVA2 after being inspected by NVA1 - in this case you can go ahead with a Route Table on NVA1's subnet provided that the nextHop is NVA2 and that is in a different subnet.

Environment :

VM = 192.168.1.4 and

NVA = 192.168.100.250

Wrt, "I added a route in the firewall VM telling it to go to 192.168.100.1 if it needed to go to 192.168.1.0/24."

This is not necessary and generally will be auto configured.

As the NVA and VM are in the same VNET (or peered VNET), the routes are auto learned.

Maybe you can remove this additional manual route and give the setup another try?

Wrt, "What I did not do is create a route table for that subnet telling the subnet in general to direct traffic for 192.168.1.0/24 to 192.168.100.1. Should I?"

No

"Or will the route in the firewall be enough?"

The default system route in Azure will be enough, as you see in Network Watcher.

You can find more details on : How Azure selects a route

Azure selects a route based on the destination IP address, using the longest prefix match algorithm.

Only and only if multiple routes contain the same address prefix, Azure selects the route type, based on the following priority:

User-defined route

BGP route

System route

Thanks,

Kapil
Joe Grover 526 Reputation points

2024-09-25T13:16:40.8933333+00:00
I'd added the route back to 192.168.1.0/24 on the NVA because the inside interface of the router is not getting its address via DHCP. I have it statically assigned to that interface on the NVA.

I did try removing the manual route I'd created on the NVA and setting the 192.168.100.250 interface to DCHP, but then I lost the ability to ping it from the server VM. There's an option in the DHCP configuration to "Automatically create default route pointing to default gateway provided by server" (which is not supposed to be checked per the vendor) but enabling that did not restore my connectivity to the server VM. Only when I re-added the route in the NVA directing any 192.168.1.0/24 traffic to 192.168.100.1 was I able to resume ping connectivity to the server VM.

If the NVA had a UDR with nextHop pointing to itself, this will lead to a routing loop.

My question about that was less about having a UDR in the subnet pointing to itself and more about having a UDR in the subnet of any kind. I was wondering if instead of having the route to 192.168.1.0/24 defined on the NVA I should have it defined in a UDR in the NVA's inside interface subnet (particularly since without the route on the NVA I can't see the server VM subnet at all).

Here's a diagram I shared with my vendor showing what I had before (that was mostly working) and how I'm trying to do it now (the right way).
KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-09-25T14:49:54.4266667+00:00
@Joe Grover ,

The new configuration seems to be right way, atleast from UDR Perspectives.

Wrt, "I was wondering if instead of having the route to 192.168.1.0/24 defined on the NVA I should have it defined in a UDR in the NVA's inside interface subnet"

This is not the general practise

The NIC will auto learn the VNET (and Peered VNET's) address range.

If you define it in the UDR of the NVA Subnet,

What would be the nextHop?

GatewayIP should not be used as it is not a NVA. (I have never seen anyone do this configuration)

Cheers,

Kapil
Joe Grover 526 Reputation points

2024-09-25T15:52:44.2166667+00:00

I'm not sure what I'm missing, then. The vendor has confirmed that the firewall will log any packet it sees whether it accepts it or drops it, and it never sees packets from 192.168.1.4 except pings directed at the 192.168.100.250 interface.

The Network Watcher shows what I've defined in the route table, so I can only assume there's something else in the vnet that isn't configured properly.
KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-09-30T10:37:56.93+00:00
Hi @Joe Grover ,

There are no routing configuration settings in the VNET, except route table.

Next Hop would always be a NVA - we should not change any OS Level Settings related to default gateway.

As next steps,

You can raise a support ticket, where a support engineer can assist you further.

You should work with both the NVA Support Team and Microsoft Engineer

If you have a support plan you may file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.

P.S : We will be able to provide you one-time support if and only if your subscription is not managed by a CSP

Cheers,

Kapil

Share via

Can't get to Azure VM from on-prem network, using Palo Alto VM firewall

Your answer