Weird Trace Route over MPLS

[jpl861]

Hi Experts,

I encountered this weird trace output.

The entire network is running EIGRP in a single AS. We are also running EIGRP on the ASA firewalls to advertise the networks behind it.

P1, P2, and P3 are running LDP because we have some routers running MPLS-VPN (not shown in the diagram). Only WAN1 and WAN2 runs LDP. The VLANs are not running LDP so VLANs 300,301, and 224 at the left handside are untagged on P1 and VLANs 10, 166, and 20 at the right hand part are also untagged at P3.

The weird thing is, when I conduct a trace route from FW1 to VLAN166, I cannot see WAN1(P2) address. First hop is the core switches in VLAN301, 2nd hop is P1, 3rd hop is blank, 4th hop is WAN2(P3).

If I do a trace from Fw1 towards VLAN20 which is connected to core1 at the right hand side, I can see all hops.

The only difference I see is that, VLAN166 is being advertised to P3 from a firewall and VLAN20 is being advertised to P3 from a 6500 switch.

If I do a trace from FW2 to VLAN224, I can see all hops.

Do you guys have any clue?

Thanks,

John

[jpl861]

I tried to do some packet captures at the P2 side using local capture.

I was able to verify that ICMP traffic from FW1 is reaching P2 for both destinations, VLAN20 and VLAN166.

Labels are correct and everything.

However, P2 only replied with Time-exceeded message when the destination is VLAN20. It did not respond with Time-Exceeded message when the destination is VLAN166. The next Time-Exceeded message I saw for VLAN166 came from WAN2(P3).

This is really odd. I don't know what I'm missing but only difference is that VLAN166 is the network behind firewall and 20 is behind a 6500 switch.

[jpl861]

I tried to create another small /30 network segment to test. I have assigned this behind the ASA and advertise via EIGRP. Same result, the trace from FW1 doesn't have WAN1(P2) on the trace route. I deleted this /30 network segment and assigned it behind Core1 instead. The trace works fine.

I don't know what's with this ASA. P3 router is a 3845 router and the rest of the routers are 6504s. But I do have another network wherein P3 is also a 6504 running the same IOS as all the others so I can say for now that IOS/platform isn't the one to blame.

[jpl861]

Problem solved, since the edge router routes the traffic using IP and hitting firewall,  the firewall wasn't able to respond back with the ICMP unlike the MSFC  that can route the data.

Configured all P routers using mpls ip ttl-expiration pop 1 so that the initial hop returns the traffic back to the source.

Just click the photo to zoom in.