GRE and Traceroute

[JohnTylerPearce]

I have a Cisco 4900m switch, which has a GRE tunnel interface configured on it.

We have two networks 172.16.0.0/21 and 172.16.8.0/16. Both these networks use the same Internet pipe.

The internet pipe includes to network ranged 71.20.111.0/26 and 71.20.111.64/26. The DG for those two

networks are 71.20.111.1 and 71.20.111.65. The default route on the Cisco 4900m switch is

the 71.20.111.65 address.

VLAN768 = 172.16.8.0/21 VLAN199 = 172.16.0.0/21

interface Vlan768

ip address 172.16.8.9 255.255.248.0

ip policy route-map ZSCALER_WEB

Anything in the 172.16.8.0/21 network uses this interface as it's default gateway.

The route-map is configured as follows

route-map ZSCALER_WEB permit 10
match ip address ZSCALER_WEB
set ip next-hop 172.17.160.146

The ZSCALER_WEB ACL is as follows

ip access-list extended ZSCALER_WEB

permit tcp any any eq www

permit tcp any any eq 443

So, anything matching the ZSCALER_WEB ACL is PBRd to a next hop of 172.17.160.146.

Tunnel0 Configuration

There is a directly connected interface, which is Tunnel0 to the 172.17.0.0/30 network.

So I would assume, once it PBRs the specified traffic it arps it out to 172.17.160.146, since it

is the next hop.

interface Tunnel0

description Zscaler GRE tunnel--primary

ip address 172.17.160.145 255.255.255.252

ip mtu 1476

ip tcp adjust-mss 1436

keepalive 10 3

tunnel source Loopback0

tunnel destination 216.52.207.65

!

interface Loopback0

ip address 71.20.111.117 255.255.255.192

On a user on VLAN199 (172.16.0.0/21) the next hop is as follows

1. 71.20.111.65 (This is the correct hop according to our routing)

2. 10.66.18.161 (This addres is inside the SP's network)

On a user on VLAN768 (172.16.8.0/21) the next hop is as follows

1. 172.16.8.9 (This is the correct hip according to our routing)

2. 10.66.18.161 (confusing because the only way to get to this network is through 71.20.111.65)

What I dont understand is, why VLAN768 doesn't have 71.20.111.65 as it's next hop after 172.16.8.9.

The only way to get to 10.66.18.161 is through 71.20.111.65?

Sorry for the long post guys.....

[Konstantin Dunaev]

I don't think it's tunnel, because the routing has changed on the hop after the 4900 switch.

[JohnTylerPearce]

Well, I can't locate where the next hop is to be perfectly honest with you.

I know the next hop, for things that have been PBRd(vlan768) go to 172.17.160.146, which is

a directly connected interface according to the 4900. Also, if it's not HTTP/HTTPS according

to the route-map it follows your normal routing rules, which will match the default route

that goes to 172.16.8.1, which is on the wireless ASA, which has a default route to

71.20.111.65. It's very confusing to say the least.

[JohnTylerPearce]

Color me stupid... The next hop for HTTP/HTTPS traffic is the other isde of the

GRE tunnel0. But, that still doesn't make since why non HTTP/HTTPS traffic would not hit 71.20.111.65 since it follows

the same rules as the other network outside of HTTP/HTTPS.

[Konstantin Dunaev]

just to be sure that we don'T really have a problem with PBR and a Tunnel - just remove the PBR configuration and test the trace without it.

[JohnTylerPearce]

Well, once I removed the PBR configuration, and when I did a traceroute to 8.8.8.8 from a 172.16.8.0/21 network, it's first hop turned out to bbe 71.20.111.65. So, I decided to put i back, and it almost looks like it's load balancing.

If I do a traceroute to any new flow it goes to 172.16.8.9 and then the follow traceroutes to the same IP gets 71.20.111.65.

[Konstantin Dunaev]

it's really strange.

Try to find out how the normal routing looks like on all transit devices, just do "sh ip route 8.8.8.8"  find the next-hop IP and then "sh ip route .

[JohnTylerPearce]

I decided to give up on it for right now. This wireless network will be going away anyway....

Thanks for all the help Konstantin, I learned a lot via your posts.