Traceroute Command Output for routes with equal metrics

Unanswered Question
Sep 1st, 2010
User Badges:

=>Routing Protocol in Question EIGRP.

=>Two equal metric routes for destination A(through R1 and R2-SVIs on two upstream 6500s)

e.g. Destination A=10.100.100.1

R1(SVI 1)=172.16.1.1

R2(SVI 2)=172.16.10.1

<Show IP Route> excerpt example for the destination IP 10.100.100.1:

D EX   10.100.100.0/29 [170/3400448] via 172.16.1.1  Vlan11
                                  [170/3400448] via 172.16.10.1 Vlan111



=>Traceroute Output example:

tracing to 10.100.100.1

1. 172.16.1.1

    172.16.10.1

    172.16.1.1

2.(entries on #2 and #3 are Irrelevant or just in place to show destination is beyond directly connected 6500 MLSs)

    192.168.50.1

3. 192.168.75.1

4.10.100.100.1(Destination)

Complete

=>5 MILLION DOLLAR OR 5 PENY QUESTION depending on subject:

Looking at #1 of Traceroute Output, is the output that alternates between 1.1=>10.1=>1.1 normal granted the two routes are "equal metric routes for the same routing procotol in use" or is that "round robin behavior" indicative of a routing problem?

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
jorge.calvo Thu, 09/02/2010 - 00:02
User Badges:
  • Bronze, 100 points or more

Hello,


What can be seen in hop #1 is not a routing problem. It is the way a tracerouter shows the two equal paths. It will list you both paths first (172.16.1.1 & 172.16.10.1) and the third line shows the chosen path (172.16.1.1).


Please note that EIGRP uses round-robin for load balancing so, if you repeat the traceroute several times you will how the output wil alternate between:


1. 172.16.1.1

    172.16.10.1

    172.16.1.1


and


1. 172.16.10.1

    172.16.1.1

    172.16.10.1


Hope this helps.

Richard Burts Thu, 09/02/2010 - 18:32
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Jorge


I agree with you that what we are seeing is not a routing problem. I disagree with your statement that :"EIGRP uses round-robin for load balancing". EIGRP does not do the load balancing. EIGRP discovers routes and places them into the routing table. It is another process that does the load balancing (round-robin or otherwise).


What we are really seeing here is the effect of process switched traffic. Any traffic that is generated by the router is inherently process switched. And process switched traffic will always round robin between available paths to the destination.


HTH


Rick

ctranminh Mon, 10/24/2011 - 01:48
User Badges:

I am sorry to up this old post. I have same problem as dimzaaaa. Now I understand the the way traceroute works in the case which has more than one paths to the destination. But the problem is that I cannot see the change when doing traceroute command serveral times. What's wrong with me?


Message was edited by: Cong Tran Minh

Richard Burts Mon, 10/24/2011 - 10:44
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

I do not understand what you are asking. You say that you do not see changes when you do traceroute several times. But you do not show us anything, so how can we help you?


Perhaps if you would post the output of your several tiems of doing traceroute then we might be able to be more helpful.


HTH


Rick

ctranminh Mon, 10/24/2011 - 19:03
User Badges:

I am sorry, because of the first time I posted in this forum.

I mean that the output of traceroute command like this:


1.192.168.13.3

   192.168.12.2

   192.168.13.3

2.192.168.34.4

   192.168.23.4 *

Completed!


1.192.168.13.3

   192.168.12.2

   192.168.13.3

2.192.168.34.4

   192.168.23.4 *

Completed!


No changes like Mr Jorge.Calvo said. I think there is a problem with loadbalancing model. I put everythings in default

The "show ip route" command also have two paths to reach the destination. Do I need more commnads for round-robin mode to work?


Thanks

Vaibhava Varma Mon, 10/24/2011 - 19:50
User Badges:
  • Silver, 250 points or more

Hi All


The Routing protocol supports load-balancing but the actual load-balancing mechanism is controlled by the routers's switching process as precisely mentioned by Richards earlier.


Default is the Fast-Switching in current IOS with CEF enabled and that leads to per-flow also popularly known as per-destination load-balaning which means between a Source-Destination Pair always the same Link will be used. So if we do multiple traces between same source-destination we would always see same path to be followed and no round-robin.


Other option for load-balancing is per-packet load-balancing which is CPU intensive and is done via process-switching and disabling CEF and ip route-cache under interface config. The effect of this is that router resorts back to legacy process switching whereby the whole frame is parsed for each packet being switches and thus packets are sent in round-robin over the available equal cost links...


Hope this helps to answer the queries.


Regards

Varma

Richard Burts Mon, 10/24/2011 - 20:37
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Varma


You have presented a good explanation of the switching process for transit traffic (traffic which is generated outside of the router and is received on one router interface and is forwarded out another router interface). But this question is about traffic which is generated by the router itself (the traceroute from the router to some external address). And as I explained in a previous post in this thread, traffic generated by the router is always processed switched.


In this new part of the discussion we can clearly see the effects of process switching (and that there are 2 equal cost routes to the destination) in this output:

1.192.168.13.3

   192.168.12.2

   192.168.13.3


The poster did not include the traceroute command so we do not know what was the destination address, and we do not know whether there were any parameters in the traceroute command which might affect how traffic was forwarded. But clearly 192.168.13.3 and 192.168.12.2 are both next hop addresses on the path to the destination.


I do not have a good explanation about why both traceroute outputs have the same pattern of next hop addresses. If 192.168.13.3 was the first next hop in the first traceroute then I would expect 192.168.12.2 to be the first next hop in the next traceroute. I wonder if there was some other traffic to that destination address in between the 2 traceroute outputs. If the new poster wants to investigate this issue further I would suggest this as an experiment:

- terminal monitor

- debug ip packet

- traceroute

- traceroute

- undeb all

- terminal no monitor

then post all of the output generated.


If there is lots of traffic flowing through the router there might be a lot of debug output which complicates understanding what is going on. If that is the case then I would suggest modifying the debug to use an access list to select traffic to be reported. It might look something like this:

access-list <1nn> permit ip any host 

debug ip packet <1nn>

where <1nn> is an available number for an extended IP access list (the range is 100 to 199) and where is the destination ip address.


HTH


Rick

ctranminh Tue, 10/25/2011 - 00:16
User Badges:

Thank you Richard!

I learn alot from your post.

I has been use the command "no ip cef" and see how round-robin work by using show ip route 4.4.4.4. The wilcard (*) is up and down each time issue the command "ping 4.4.4.4 source loopback 0 repeat 1"


This is output of some command after Issued "no ip cef"

ping 4.4.4.4 source loopback 0 repeat 10


*Mar  1 03:44:53.631: IP: tableid=0, s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/0), routed via RIB

*Mar  1 03:44:53.635: IP: s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/0), len 100, sending

*Mar  1 03:44:53.655: IP: tableid=0, s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/1), routed via RIB

*Mar  1 03:44:53.655: IP: s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/1), len 100, sending

*Mar  1 03:44:53.723: IP: tableid=0, s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/0), routed via RIB

*Mar  1 03:44:53.723: IP: s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/0), len 100, sending


traceroute 4.4.4.4 source loopback 0 (first time)


1 192.168.12.2 12 msec

    192.168.13.3 68 msec

    192.168.12.2 28 msec

  2 192.168.34.4 12 msec

    192.168.24.4 32 msec

Complete!

*Mar  1 03:53:48.595: IP: tableid=0, s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/0), routed via RIB

*Mar  1 03:53:48.595: IP: s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/0), len 28, sending

*Mar  1 03:53:48.631: IP: tableid=0, s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/1), routed via RIB

*Mar  1 03:53:48.635: IP: s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/1), len 28, sending

*Mar  1 03:53:48.715: IP: tableid=0, s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/0), routed via RIB

*Mar  1 03:53:48.719: IP: s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/0), len 28, sending

*Mar  1 03:53:48.731: IP: tableid=0, s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/1), routed via RIB


traceroute 4.4.4.4 source loopback 0 (second time)


1 192.168.12.2 12 msec

    192.168.13.3 68 msec

    192.168.12.2 28 msec

  2 192.168.34.4 12 msec

    192.168.24.4 32 msec

Complete!


*Mar  1 03:53:56.291: IP: tableid=0, s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/0), routed via RIB

*Mar  1 03:53:56.291: IP: s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/0), len 28, sending

*Mar  1 03:53:56.355: IP: tableid=0, s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/1), routed via RIB

*Mar  1 03:53:56.359: IP: s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/1), len 28, sending

*Mar  1 03:53:56.395: IP: tableid=0, s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/0), routed via RIB

*Mar  1 03:53:56.399: IP: s=1.1.1.1 (local), d=4.4.4.4 (FastEthernet0/0), len 28, sending


I may see how round-robin mode work through debug info of ping and traceroute. But the output of traceroute is not what I expected:

1 192.168.12.2 12 msec

    192.168.13.3 68 msec

    192.168.12.2 28 msec

  2 192.168.34.4 12 msec

    192.168.24.4 32 msec

and

1 192.168.13.3 40 msec

    192.168.12.2 60 msec

    192.168.13.3 16 msec

  2 192.168.24.4 68 msec

    192.168.34.4 48 msec


One more thing, I use ping and traceroute with source parameter, do this like as traffic generated by router itself or transit traffic?

Richard Burts Tue, 10/25/2011 - 09:38
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

I am glad that using debug has helped to show what is going on.


It does not really matter whether you specify source address or just let the router use its default. When you execute traceroute on the router then the packets are generated by the router and are not transit traffic.


The only way to get transit traffic is to have some other device generate the packets and send them to the router to forward toward the destination.


HTH


Rick

ctranminh Tue, 10/25/2011 - 21:58
User Badges:

lab_traceroute.jpeg


















Sorry I have more questions. In this lab I add more 2 routers. The lab now already looks like transit traffic.

First: On R2  "debug ip packet 101" (access-list to permit ip any host 6.6.6.6)

                    "no ip cef"

        On R1, "traceroute 6.6.6.6 source loopback 0"    two times              

                    1 192.168.12.2 36 msec 48 msec 16 msec

                    2 192.168.23.3 12 msec 40 msec 12 msec

                    3 192.168.35.5 44 msec 44 msec 12 msec

                    4 192.168.56.6 108 msec *  116 msec

               Complete!


                    1 192.168.12.2 36 msec 48 msec 16 msec

                    2 192.168.23.3 12 msec 40 msec 12 msec

                    3 192.168.35.5 44 msec 44 msec 12 msec

                    4 192.168.56.6 108 msec *  116 msec

               Complete!


         Jump to R2: no debug output

               But on R2 when I put "debug ip packet". After traceroute again, debug output:  

*Mar  1 02:07:13.439: IP: tableid=0, s=192.168.12.2 (local), d=1.1.1.1 (FastEthernet1/0), routed via RIB

*Mar  1 02:07:13.439: IP: s=192.168.12.2 (local), d=1.1.1.1 (FastEthernet1/0), len 56, sending

*Mar  1 02:07:13.503: IP: tableid=0, s=192.168.12.2 (local), d=1.1.1.1 (FastEthernet1/0), routed via RIB

*Mar  1 02:07:13.503: IP: s=192.168.12.2 (local), d=1.1.1.1 (FastEthernet1/0), len 56, sending

*Mar  1 02:07:13.519: IP: tableid=0, s=192.168.12.2 (local), d=1.1.1.1 (FastEthernet1/0), routed via RIB

*Mar  1 02:07:13.519: IP: s=192.168.12.2 (local), d=1.1.1.1 (FastEthernet1/0), len 56, sending

I don't understand these debug info


          Additional, when I "ping 6.6.6.6 source loopback 0" no debug info output at all.

Plz tell me why there is no debug output when debug ip packet with access-list?


Second: when  I put more command in R2 "no ip route-cache " in subcommand of  interface fa0/0 and fa0/1 not interface fa1/0, the result go as I  expected.   

          On R1, "ping 6.6.6.6 source loopback 0"

          The output on R2:                 

*Mar  1 02:38:37.883: IP: tableid=0, s=1.1.1.1 (FastEthernet1/0), d=6.6.6.6 (FastEthernet0/0), routed via RIB

*Mar  1 02:38:37.883: IP: s=1.1.1.1 (FastEthernet1/0), d=6.6.6.6 (FastEthernet0/0), g=192.168.23.3, len 100, forward

*Mar  1 02:38:37.983: IP: tableid=0, s=1.1.1.1 (FastEthernet1/0), d=6.6.6.6 (FastEthernet0/1), routed via RIB

*Mar  1 02:38:37.983: IP: s=1.1.1.1 (FastEthernet1/0), d=6.6.6.6 (FastEthernet0/1), g=192.168.24.4, len 100, forward

*Mar  1 02:38:38.035: IP: tableid=0, s=1.1.1.1 (FastEthernet1/0), d=6.6.6.6 (FastEthernet0/0), routed via RIB

*Mar  1 02:38:38.035: IP: s=1.1.1.1 (FastEthernet1/0), d=6.6.6.6 (FastEthernet0/0), g=192.168.23.3, len 100, forward

          On R1 "traceroute 6.6.6.6 source loopback 0" two times


  1 192.168.12.2 20 msec 68 msec 8 msec

  2 192.168.24.4 72 msec

    192.168.23.3 28 msec

    192.168.24.4 32 msec

  3 192.168.35.5 116 msec

    192.168.45.5 12 msec

    192.168.35.5 72 msec

  4 192.168.56.6 80 msec *  136 msec


  1 192.168.12.2 32 msec 48 msec 32 msec

  2 192.168.23.3 24 msec

    192.168.24.4 28 msec

    192.168.23.3 44 msec

  3 192.168.45.5 52 msec

    192.168.35.5 16 msec

    192.168.45.5 12 msec

  4 192.168.56.6 88 msec *  96 msec


           Round-robin works well

Why I need more "no ip route-cache" command in specified interface subcommand?

Richard Burts Wed, 10/26/2011 - 09:53
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

You may need to use no ip route-cache to get debug output. The reason for this is that debug is a process that runs in the router CPU and it can only report of traffic that is routed by the router CPU. If route cache/fast switching/CEF/hardware switching is forwarding packets without using the CPU then debug will not see or report those packets.


HTH


Rick

Actions

This Discussion