Solved: MPLS trace issue

littlegrave · ‎08-03-2007

Hello,

I have a problem for which I can't give any explanation to myself. Here's what I have:

10.0.0.0 || CE_WEST -- PE_WEST -- P-SP -- PE_EAST -- CE_EAST || 20.0.0.0

I deployed an MPLS VPN between the two sites.

I read in the MPLS and VPN Architectures vol II the following:

----

When TTL propagation is disabled in an MPLS VPN backbone, a user who executes a

trace on a CE router will usually not see the egress PE router because that router

performs only a label lookup and is not affected with the IP TTL

----

I disabled TTL propagation and did a trace from 10.0.0.1 to 20.0.0.1 (CE router interfaces) and I saw PE_WEST, PE_EAST and CE_EAST in the trace, which is not what I expected to see. In the LFIB of PE_EAST, a packet to 20.0.0.0 should not trigger a L3 lookup since this network is not directly connected and is not an aggregate and thus this router should not examine the TTL field of the packet at all. I executed a few debugs on the routers and noticed that PE_EAST does not do any L3 lookup. However, I see it in the trace and can't find an explanation for this.

Maybe I did not understand something or it's just something that has changed over time in recent IOS version(let's not forget the book is from 2003).

Any ideas why this happens will be very much appreciated.

Thank you

Harold Ritter · ‎08-05-2007

Pavel,

The behavior described in your original post used to be true but has since been changed as of 12.3(13), 12.0(31)S, 12.2(18)SXE and 12.3(14)T.

Hope this helps,

Harold Ritter
Sr Technical Leader
CCIE 4168 (R&S, SP)
harold@cisco.com
México móvil: +52 1 55 8312 4915
Cisco México
Paseo de la Reforma 222
Piso 19
Cuauhtémoc, Juárez
Ciudad de México, 06600
México

View solution in original post

mohammedmahmoud · ‎08-03-2007

Hi,

Your understanding is correct, using "no mpls ip propagate-ttl" causes a traceroute command to ignore all hops traversed by the MPLS packet in the network, are you sure that you've added this command on both PEs (to work in both ways).

http://www.cisco.com/en/US/tech/tk436/tk428/technologies_tech_note09186a008020a42a.shtml#no_mpls

Are you trying to do the trace from the PE itself, if yes, then for this command to take effect you need to use the local keyword.

http://www.cisco.com/en/US/products/ps6350/products_command_reference_chapter09186a0080430c47.html#wp1049262

HTH,

Mohammed Mahmoud.

littlegrave · ‎08-03-2007

Hi,

I have enabled this command on all PE routers and even the P routers (although this does not concern me in this case). I am trying to trace from a CE to a CE (their loopback interfaces).

I examined the LFIB and the CEF table, tried some debugs and saw that L3 lookup is not done on the Egress PE router (and that's how it is supposed to be), but somehow this PE router appears in the path.

littlegrave · ‎08-03-2007

Here's the LFIB of the PE-East router:

PE-East#sh mpls for

Local Outgoing Prefix Bytes tag Outgoing Next Hop

tag tag or VC or Tunnel Id switched interface

16 Pop tag 23.0.0.0/8 0 Fa0/0 34.0.0.3

17 18 22.22.22.22/32 0 Fa0/0 34.0.0.3

18 Untagged 20.0.0.0/8[V] 126 Fa0/1 45.0.0.5

19 Aggregate 45.0.0.0/8[V] 0

20 Untagged 80.0.0.0/8[V] 0 Fa0/1 45.0.0.5

21 Aggregate 2001:0:0:45::/64 0

22 Aggregate 2001:0:0:20::/64 0

and here's the trace from the 10.0.0.0 network to the 20.0.0.0 network executed on CE-West:

CE-West#trace 20.0.0.1

Type escape sequence to abort.

Tracing the route to 20.0.0.1

1 12.0.0.2 308 msec 236 msec 308 msec

2 45.0.0.4 [AS 1] [MPLS: Label 18 Exp 0] 720 msec 432 msec 668 msec

3 45.0.0.5 [AS 1] 692 msec 540 msec 948 msec

CE-West#

Harold Ritter · ‎08-04-2007

Pavel,

It is actually normal behavior to see the egress PE in the traceroute output as the egress PE pops the label, checks the IP TTL and forwards the packet to the CE or sends an ICMP TTL expired message back to the source, depending on the result of the TTL check.

Hope this helps,

Harold Ritter
Sr Technical Leader
CCIE 4168 (R&S, SP)
harold@cisco.com
México móvil: +52 1 55 8312 4915
Cisco México
Paseo de la Reforma 222
Piso 19
Cuauhtémoc, Juárez
Ciudad de México, 06600
México

mohammedmahmoud · ‎08-05-2007

Hi Harold,

AFAIK, that the ingress router should appear in the trace as the first trace packet is sent from the CE with TTL of 1 and thus the first ingress router decrements the TTL, drops the packet, and sends an ICMP TTL-exceeded message to the source, and then the second packet on until the packet reaches the final egress router, the issue here is that the packet reached the egress PE with a large TTL value (since the ingress PE converted the TTL to 255, as we have disabled TTL propagation), and thus the egress PE shouldn't appear in the TTL as it won't send a TTL expired message, Harold please share us your opinion.

BR,

Mohammed Mahmoud.

littlegrave · ‎08-05-2007

Hi Harold,

Thank you for clearing this up! However, I still have a couple of questions for this case? Is the passage that I posted in my first post outdated? Is this something that has changed over time in newer IOS releases? Is the IP TTL field the only field from the IP packet that is examined (considering the packet is not for a directly connected network)?

I would be very thankful if you help me out with these questions.

Mohammed,

I think what Harold said was that not only the label TTL field is checked, but also the IP TTL. This is just how I image it would work, but I may be wrong.

Greetings,

Pavel

mohammedmahmoud · ‎08-05-2007

hi Pavel,

You are totally correct about the IP TTL field, after looking out to the big picture, i think that both PEs should appear, the ingress should appear as per my last post, and the egress should appear as per Harold's last post, and thus your output is consistent.

Harold,

If you may please confirm these results.

BR,

Mohammed Mahmoud.

Harold Ritter · ‎08-05-2007

Mohammed,

That is correct.

Regards,

Harold Ritter
Sr Technical Leader
CCIE 4168 (R&S, SP)
harold@cisco.com
México móvil: +52 1 55 8312 4915
Cisco México
Paseo de la Reforma 222
Piso 19
Cuauhtémoc, Juárez
Ciudad de México, 06600
México

Harold Ritter · ‎08-05-2007

Pavel,

The behavior described in your original post used to be true but has since been changed as of 12.3(13), 12.0(31)S, 12.2(18)SXE and 12.3(14)T.

Hope this helps,

Harold Ritter
Sr Technical Leader
CCIE 4168 (R&S, SP)
harold@cisco.com
México móvil: +52 1 55 8312 4915
Cisco México
Paseo de la Reforma 222
Piso 19
Cuauhtémoc, Juárez
Ciudad de México, 06600
México

littlegrave · ‎08-05-2007

Harold,

Thank you so much for explaining everything that was unclear to me!

This shows me why I should avoid reading older books :)

Harold Ritter · ‎08-05-2007

Unfortunately/fortunately, everything moves so fast that on one side it is hard to keep up but on the other it keeps all of us employed ;-)

Regards,

Harold Ritter
Sr Technical Leader
CCIE 4168 (R&S, SP)
harold@cisco.com
México móvil: +52 1 55 8312 4915
Cisco México
Paseo de la Reforma 222
Piso 19
Cuauhtémoc, Juárez
Ciudad de México, 06600
México

mohammedmahmoud · ‎08-05-2007

Hi Harold,

Let me guess, does this change has anything to do with PHP.

BR,

Mohammed Mahmoud.

Harold Ritter · ‎08-05-2007

Mohammed,

This change is directly related to the way the egress PE processes the TTL. It is therefore not related to the PHP processing.

Regards,

Harold Ritter
Sr Technical Leader
CCIE 4168 (R&S, SP)
harold@cisco.com
México móvil: +52 1 55 8312 4915
Cisco México
Paseo de la Reforma 222
Piso 19
Cuauhtémoc, Juárez
Ciudad de México, 06600
México

mohammedmahmoud · ‎08-05-2007

Harold,

Correct me if i am wrong, according to the PHP, tracing a LAN on a CE router from another CE router should differ from tracing the WAN ip of the same CE router from the other CE router.

HTH,

Mohammed Mahmoud.