Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

Very Odd Trace results

Hi,

Just wondering if anyone could shed light on the following trace results (from #9 down) taken from an Internet host to the inside interface of an Internet connected router.


TraceRoute to xx.56.137.249

Hop(ms)(ms)(ms)IP AddressHost name
11096xx.249.128.109-
2666xx.123.64.22-
310653xx.192.242.253-
44650257xx.229.0.193-
5259266260xx.229.0.193-
6269276370xx.229.1.186-
7303268264xx.170.0.182-
8266262Timed outxx.42.4.117-
9Timed out358390xxx.42.129.50-
10Timed out650Timed outxx.42.129.50-
11525504502xxx.42.129.50-
12506502503xxx.42.129.49-
13657681670xxx.42.129.50-
14328333340xxx.42.129.50-
15356374343xxx.42.129.50-
169468311021
xxx.42.129.50-
17725442414xxx.42.129.50-
18376347325xxx.42.129.49-
1996011741171xxx.42.129.49-
20Timed out1429Timed outxxx.42.129.50-
21Timed out1291Timed outxxx.42.129.49-
22Timed out1408Timed outxxx.42.129.49-
23Timed outTimed outTimed out-
24Timed outTimed out1467xxx.42.129.49-
25580535530xxx.42.129.50-
26479457409xxx.42.129.49-

As you can see, there seems to be a loop, but the routing table only shows a static default to the PE router, along with the inside and outside connected networks.  Access-lists have been removed for testing, so the only one remaining is used for debug purposes.  No routing protocols are running.

Traffic passing through the router, for example to the connected 'outside' firewall interface (xx.56.137.251), or to a static NAT address running on the firewall (xx.56.137.252) don't seem to be affected by the loop.

The network looks like this..

loop.JPG

And the config like this...

interface FastEthernet0/0   <<Inside

ip address xx.56.137.249 255.255.255.248

duplex auto

speed auto

!

interface Serial0/0/0:0   <<Outside

ip address xxx.42.129.50 255.255.255.252

ip access-group test in

!

ip forward-protocol nd

ip route 0.0.0.0 0.0.0.0 xxx.42.129.49

!

ip access-list extended test

permit ip any host xx.56.137.252 log

permit ip any any

!

Thanks,

Duncan

Everyone's tags (4)
7 REPLIES
VIP Super Bronze

Re: Very Odd Trace results

Hi Duncan,

Have you or the provider made any changes on the network that will cause this?

Another word, did this just happen recently or it has always been like this and you did not know about it?

You have a default route towards the provide and it seems that the provider has the same towards you, but in an L3-vpn/vrf.

Reza

Hall of Fame Super Silver

Re: Very Odd Trace results

Duncan

Can you clarify what was the target address in the traceroute in your post? You mention that traffic to some destination addresses does work. So perhaps it would help us if we knew what the traceroute was trying to get to.

HTH

Rick

New Member

Re: Very Odd Trace results

Hi Richard,

The traceroute was targeting the inside interface of the Internet router (xx.56.137.249).

I'm also seeing similar behaviour to certain NATed addresses on the firewall, behind the Internet router.

debug ip packet detail w/ an ACL filtering ICMP traffic to/from xx.56.137.248 /29 doesn't show the ICMP reaching the router for certain addresses (cef and fast switching disabled on both inside and outside interfaces to get complete packet-by-packet view), but for others it seems fine.

We tried restarting the router but to no avail.

The service provider recommends restarting the firewall too, which we'll try today, although the problem would seem to be in front of, rather than behind the Internet router.

Duncan

New Member

Re: Very Odd Trace results

It seems it has been an issue since the beginning.

The main problem is that we use a 3rd party mail filter (our MX record points to them and they forward email once filtered), half of who's servers can reach our email server in the DMZ (xx.56.137.252) and half who can't (their traffic gets pulled into a loop between the CE and PE routers).

VIP Super Bronze

Re: Very Odd Trace results

Is your connection to the service provider reside in a VRF?  To me. it seems that you are sending the provider a global default router and the service provider send it back to you in your own VRF.

Hall of Fame Super Silver

Re: Very Odd Trace results

Duncan

These symptoms are quite puzzling. In your earlier posts it seemed that the problem was that certain destination addresses would work and other addresses would not work. Now you seem to be saying that the same destination address may work or may not work depending on the source of the packet. Am I understanding this correctly?

That leads me to wonder whether there might be more than one network path to this router. Is there more than just the serial interface that you show that could be used to access this router?

And it also makes me that that the question from Reza is possibly useful. Are you doing any VRFs on this router?

HTH

Rick

New Member

Re: Very Odd Trace results

Richard,

We seem to get the same phenomenon in both directions (i.e. certain addresses are pingable from the Internet router and others aren't, and certain addresses in the xx.56.137.248/29 range are pingable from the Internet while others aren't.

As for other paths to the router, there are none (see below show ip int brief output):

IGD-RT01#sh ip int brief

Interface                  IP-Address      OK? Method Status                Prot

ocol

FastEthernet0/0            xx.56.137.249   YES NVRAM  up                    up

FastEthernet0/1            unassigned      YES NVRAM  up                    down

Serial0/0/0:0              xxx.42.129.50   YES NVRAM  up                    up

IGD-RT01#

Here's some sample output from a raw internet host pinging several addresses in the xx.56.137.248/29 range.

C:\Documents and Settings\Administrator>ping xx.56.137.253

Pinging xx.56.137.253 with 32 bytes of data:

Reply from xxx.42.129.50: TTL expired in transit.

Reply from xxx.42.129.50: TTL expired in transit.

Reply from xxx.42.129.50: TTL expired in transit.

Reply from xxx.42.129.50: TTL expired in transit.

Ping statistics for xx.56.137.253:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Documents and Settings\Administrator>ping xx.56.137.252

Pinging xx.56.137.252 with 32 bytes of data:

Reply from xxx.42.129.49: TTL expired in transit.

Reply from xxx.42.129.50: TTL expired in transit.

Reply from xxx.42.129.50: TTL expired in transit.

Reply from xxx.42.129.49: TTL expired in transit.

Ping statistics for xx.56.137.252:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Documents and Settings\Administrator>ping xx.56.137.249

Pinging xx.56.137.249 with 32 bytes of data:

Reply from xx.56.137.249: bytes=32 time=313ms TTL=239

Reply from xx.56.137.249: bytes=32 time=308ms TTL=239

Reply from xx.56.137.249: bytes=32 time=299ms TTL=238

Reply from xx.56.137.249: bytes=32 time=316ms TTL=238

Ping statistics for xx.56.137.249:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 299ms, Maximum = 316ms, Average = 309ms

I had 'debug ip packet detail' filtering based on ICMP traffic to/from this segment, with cef disabled globally and fast switching disabled on both interfaces, and for the 'TTL expired in transit' packets, I didn't get any output on the router.  For pings to .249 output came as expected.  Before you ask, the debug ACLs are fine.
Reza,
As for the VRF, I'm not very familiar with MPLS, but we certainly don't have any MPLS config on the CE side.
Basically the router config is as sparse as we can make it.  Take a look...
!
controller E1 0/0/0
channel-group 0 unframed
!
controller E1 0/0/1
!
!
!
!
!
interface FastEthernet0/0
ip address xx.56.137.249 255.255.255.248
duplex auto
speed auto
!
interface FastEthernet0/1
no ip address
duplex auto
speed auto
!
interface Serial0/0/0:0
ip address xxx.42.129.50 255.255.255.252
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 xxx.42.129.49
!
Everything else bar the above access-lists for 'debug ip packet detail' and the line config is default.
Thanks,
Duncan

598
Views
0
Helpful
7
Replies
CreatePlease to create content