tracert show same hop twice

[fsebera]

When I perform a traceroute from a remote location that traverses through my border router, when the traceroute reaches my border router, I receive that hop twice followed by the final hop. (see hops 4 and 5). Traceroute and all data packets are flowing but fear this double hop is an issue waiting to happen.

EX:

traceroute 10.10.50.5

1     10.10.10.1

2     10.10.20.2

3     10.10.30.3

4     10.10.40.4

5     10.10.40.4

6     10.10.50.5

Border router config

interface Tunnel10
ip address 10.10.40.4 255.255.255.252
ip access-group IN-ACL in
ip access-group OUT-ACL out
no ip redirects
tunnel source Serial0/1/0
tunnel destination 172.16.5.1
!
interface GigabitEthernet0/0
no ip address
!   
interface GigabitEthernet0/1
no ip address
!
interface GigabitEthernet0/0.20
encapsulation dot1Q 31
bridge-group 20
!
interface GigabitEthernet0/1.20
encapsulation dot1Q 31
bridge-group 20
!
interface Serial0/1/0
ip address 10.10.99.1 255.255.255.252
ip access-group IN-LINK in
no ip redirects
load-interval 30
no service-module t1 remote-loopback full
no cdp enable
crypto map ENCRYPT-DATA
!
interface BVI20
ip address 10.10.50.1 255.255.255.240
!
bridge 20 priority 65535
bridge 20 protocol ieee
bridge 20 route ip

--end--

Show ip route on the border router shows the border router is layer two adjacent to the last hop 10.10.50.5 via BVI20.

The ip routing table shows 10.10.50.0/28 is directly connected via BVI20

BORDER#sh ip ro

Gateway of last resort is 10.10.50.12 to network 0.0.0.0

C       10.10.40.0/30 is directly connected, Tunnel10
C       10.10.99.0/30 is directly connected, Serial0/1/0
      10.10.50.0/28 is subnetted, 1 subnets
C       10.10.50.0 is directly connected, BVI20

Spanning-tree is blocking G0/0.20, fordwarding on G0/1.20.

There is only a single switch between the border router and final hop (10.10.50.5)

Thanks for at least looking!!

ANY thoughts?

Frank

[milan.kulik]

Hi,

I guess it's caused by the Tunnel10 you are using to connect to your border router.

RFC 1702 (http://www.ietf.org/rfc/rfc1702.txt) says:

"When IP is encapsulated in IP, the TTL, TOS, and IP security options
MAY be copied from the payload packet into the same fields in the
delivery packet.  The payload packet's TTL MUST be decremented when
   the packet is decapsulated to insure that no packet lives forever."

Which means:

An encapsulated traceroute packet with TTL = 0 comes to your Tunnel10 interface.

The router replies with an ICMP "TTL expired" packet using the Tunnel10 interface IP (10.10.40.4).

Following traceroute packet comes with TTL = 1.

When decapsulated, TTL is decremented and reaches 0 again.

So the router replies with ICMP "TTL expired", using Tunnel10 interface IP (10.10.40.4) again.

(I should have used 1 and 2 instead of 0 and 1 in the explanation above to be precise, but the principle is clear, I hope.)

Does it make sense?

HTH,

Milan

Message was edited by: milan.kulik

[fsebera]

Hi Milan,

I can follow the logic of an encrypted packet arriving at Tunnel10 with a TTL of 1 as this is normal traceroute behavior as I understand it. The router decrements the packet TTL by 1 which leave the TTL=0, so the router replies back to the original source address with "TTL expired" using the ingress interface 10.10.40.4. (By the way, mistake on my part and not a trick, just a typeo on my part - 10.10.40.4/30 is not valid should have been 10.10.40.2 or 3 but not an issue with understanding the logic in this example).

I understand and agree with this part.

This next step is a little murky for me.

The original source sends another packet towards the final destination with TTL set to 1 high than it took to reach Tunnel10 in last attempt. In this case the TTL value is now 2.  This time the packet is unencrypted at Tunnel10 and in this unencryption process the TTL is decremented by 1 which now leaves the TTL with a value of 1.

It seams the router would forward the packet to the next hop with TTL set to 1.

BUT I think you are indicating the router decrements the TTL BEFORE forwarding and since the TTL would equal 0, the router would reply back to the original source with the "TTL expired" message.  The ingress interface is always used for Traceroute and this is why Tunnel10 10.10.40.4 is seen twice?

So with point-to-point in-line encryption, each end point (unencapsulation point, end of tunnel) will always strip the TTL value by two?

Sorry took so long to respond, been little ill.

Thanks again for your assistance

Frank

[milan.kulik · Accepted solution]

Hi Frank,

my understanding is:

a) a router is replying with ICMP "TTL expired" when received a packet with TTL=1 (no matter of destination address).

b) a reouter is decreasing TTL value before forwarding an IP packet

c) GRE encapsulation is copying the TTL value from the original packet which is then encapsulted (including the original header with the original TTL).

So the GRE packet has an (external) TTL field within the IP header plus another (internal) TTL value within the encapsulated packet.

Now the source device is sending traceroute packets.

Starts with a packet with TTL = 1, then packet with TTL = 2 followed, then TTL = 3, etc.

Which means:

In your tracerout output:

3     10.10.30.3 - i.e., the router received a packet with TTL = 1 and replied with ICMP "TTL expired"

Next packet was received with TTL = 2 by your 10.10.3.3 router.

It decreases the TTL value to TTL = 1 before forwarding.

As there is a GRE tunnel used to forward it to the next hop, the router encapsulates the original packet to a GRE packet while copying TTL = 1 to the external TTL field.

When the next hop receives the GRE packet, it's seeing the external TTL = 1 and replies with ICMP "TTL expired".

And you see:

4     10.10.40.4

Next traceroute packet was recived with TTL = 3 by the 10.10.30.3 router.

And forwarded to the GRE tunnel with TTL = 2 within the external and internal TTL field.

As the external TTL is 2, the 10.10.40.4 router continues by decapsulating the packet.

When decapsulated, the extra step specified in the RFC follows: "The payload packet's TTL MUST be decremented when
   the packet is decapsulated to insure that no packet lives forever."

So the TTL of the decapsulated packet is decreased from 2 to 1.

And the router handles the packet like just received, i.e., it is seeing a packet arrived with TTL = 1.

So replies with ICMP "TTL expired" again (using the tunel port as the source IP), and you see:

5     10.10.40.4

The next packet comes with TTL=3 and is forwarded to the next hop with TTL=1, which has

6     10.10.50.5

as a result.

I hope to be clear now?

And you seem to be correct with "So with point-to-point in-line encryption, each end point (unencapsulation point, end of tunnel) will always strip the TTL value by two." - which is a side effect I did not notice originally.

BR,

Milan

[yenaungoo]

Hi All, I also encounter similar issue. The thing is it already reached to designated router but it hit next hop and return back. As in below, No.6 & No.7 are not needed. After No.5, it hits to that router via g0/0/2 but forward out via Tun1, then return via Tun1 again. And back to Loop0 IP.  Can help to suggest? Thanks.

C:\>tracert 192.168.13.9
Tracing route to 192.168.13.9 over a maximum of 30 hops
  1     1 ms    <1 ms    <1 ms  172.28.6.21
  2     1 ms    <1 ms    <1 ms  172.28.6.29
  3     1 ms     1 ms    <1 ms  172.31.1.145
  4     2 ms     1 ms    <1 ms  172.31.1.166
  5     1 ms    <1 ms     1 ms  192.168.12.241
  6    35 ms    36 ms    35 ms  192.168.16.186
  7    36 ms    36 ms    36 ms  192.168.16.186
  8    36 ms    36 ms    41 ms  192.168.13.9
Trace complete.

-------------------------------------------

interface Loopback0
 ip address 192.168.13.9 255.255.255.255
!
interface GigabitEthernet0/0/1
 mtu 1600
 ip address 10.200.32.25 255.255.255.252
!
interface GigabitEthernet0/0/2
 ip address 192.168.12.41 255.255.255.224
 no ip redirects
 no ip proxy-arp
 ip policy route-map PBR_THAILAND
!
interface Tunnel1
 bandwidth 20000
 ip address 192.168.16.185 255.255.255.252
 ip mtu 1443
 ip tcp adjust-mss 1300
 tunnel source GigabitEthernet0/0/1
 tunnel mode ipsec ipv4
 tunnel destination 10.200.32.26
 tunnel protection ipsec profile Pmpls
 service-policy output SHAPE-20M
!

[milan.kulik]

Hi, a) you are saying "...No.5, it hits to that router via g0/0/2." But I see 5 1 ms <1 ms 1 ms 192.168.12.241 while interface GigabitEthernet0/0/2 ip address 192.168.12.41 255.255.255.224 so a different IP address? b) I can see ip policy route-map PBR_THAILAND on the interface GigabitEthernet0/0/2. How does it look like? Isn't it forwarding the traffic to the tiunnel? Best regards. Milan

[yenaungoo]

a) After the router (192.168.12.241) it hits to that router (192.168.13.9) via g0/0/2

b) PBR on g0/0/2 is as in below.

route-map PBR_THAILAND permit 40
 match ip address DSCP_RTP
 set ip next-hop 192.168.16.186
!
route-map PBR_THAILAND permit 41
 match ip address VC_THAILAND
 set ip next-hop 192.168.16.186
!
route-map PBR_THAILAND permit 50
 match ip address DSCP_REPLICATION
 set ip next-hop 192.168.16.186
!
route-map PBR_THAILAND permit 51
 match ip address REPLICATION
 set ip next-hop 192.168.16.186
!
route-map PBR_THAILAND deny 55
 match ip address LAN_INTERFACE
!
route-map PBR_THAILAND permit 60
 match ip address DSCP_DEFAULT
 set ip next-hop 192.168.16.186
!
route-map PBR_THAILAND permit 61
 match ip address DEFAULT
 set ip next-hop 192.168.16.186
!
ip access-list extended DSCP_RTP
 permit ip any any dscp ef
 permit ip any any dscp cs3
 permit ip any any dscp af31
 permit ip any any dscp af32
ip access-list extended VC_THAILAND
 permit ip any host 10.244.30.200
 permit ip any host 10.244.30.201
 permit ip any host 10.244.36.210
ip access-list extended DSCP_REPLICATION
 permit ip any any dscp cs2
ip access-list extended REPLICATION
 permit tcp any any eq 8000
 permit tcp any eq 8000 any
 permit tcp any any eq 4545
 permit tcp any eq 4545 any
 permit ip 172.18.126.0 0.0.0.63 any
ip access-list extended LAN_INTERFACE
 permit ip any host 192.168.12.41
 permit ip any host 192.168.12.9
ip access-list extended DSCP_DEFAULT
 permit ip any any
ip access-list extended DEFAULT
! permit ip any any

[yenaungoo]

Hi Milan,

Now I realize that I should add the loopback IP (192.168.13.9) to the ACL LAN_INTERFACE.

So can we say that PBR prefer than the connected route?

Thanks,

[milan.kulik]

Hi, my understanding is: When a packet is received on an interface where a PBR is configured, PBR policy is checked as the first. If the packet matches it, it's forwarded based on the PBR policy. If the packet does not match it, then it's routed based on the router RIB (sent according to the connected route in this case). So yes, the priority is PBR first, connected route second. Best regards, Milan