Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Community Member

50% packet loss to/from 2621 router

I have a Cisco 2621 router in front of a Watchguard Firebox III 700. The interface (FastEthernet0/1) IP on the Cisco facing my LAN is 100.200.300.1, for example. The IP on the FBIII external interface is 100.200.300.2.

Using any computer behind the FBIII, if I ping the Cisco at 100.200.300.1, 50% of the packets are dropped. Likewise, from the Cisco, if I ping the FBIII at 100.200.300.2 50% of packets are dropped.

Any packets passing through the Cisco (the router is not the source or destination) seem to be fine, i.e. no packet loss.

As a result when I try to copy the system image from the Cisco to a TFTP server behind the FBIII, some data gets through but the copy eventually fails. The copy status on the Cisco console looks something like this

.!!.!...!.!...!...!!.....

A period represents a timeout and a bang represents 10 packets sent.

I'm leaning toward the issue being with the Cisco router but I'm not positive. I'm wondering if anyone has seen this behavior and has any helpful hints.

24 REPLIES
Hall of Fame Super Gold

Re: 50% packet loss to/from 2621 router

Chris

The symptom of 50 % packet loss (especially if it really is exactly 50 %) is frequently the result of having 2 routes in the routing table and one of them works and one does not. When the router is generating packets (traffic from the router not traffic through the router) it will send packet by packet over both routes and 50 % of the packets get lost.

Can you post the output of show ip route from the router?

HTH

Rick

Community Member

Re: 50% packet loss to/from 2621 router

Sure...

cisco2621#show ip route

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP

D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP

i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area

* - candidate default, U - per-user static route, o - ODR

P - periodic downloaded static route

Gateway of last resort is 71.128.141.41 to network 0.0.0.0

71.0.0.0/30 is subnetted, 1 subnets

C 71.128.141.40 is directly connected, Serial0/0.1

64.0.0.0/24 is subnetted, 1 subnets

C 64.171.123.0 is directly connected, FastEthernet0/1

67.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

S 67.153.169.20/30 [1/0] via 67.155.215.249

C 67.155.215.248/29 is directly connected, FastEthernet0/0

B* 0.0.0.0/0 [20/0] via 71.128.141.41, 5w3d

FastEthernet0/1 is the interface I'm referring to. Using real IP's, issuing ping from the 2621 to 64.171.123.2 I see 50% packet loss.

Hall of Fame Super Gold

Re: 50% packet loss to/from 2621 router

Chris

Thanks for posting the additional information. It does not show what I had thought it might and there is not an indication here that it might be the issue with 2 routes which I had thought it might be.

Would you post the output of show arp? It might also be helpful to turn on debug ip icmp, try the ping again, and post the debug output.

HTH

Rick

Community Member

Re: 50% packet loss to/from 2621 router

Hi cmcfarling :

the Firebox maybe not respond the icmp ack packet ?

Re: 50% packet loss to/from 2621 router

Did you ruled out the physical aspect? duplex transmission missmatch etc.. on all devices including your computer .

Community Member

Re: 50% packet loss to/from 2621 router

Here's the show arp output:

cisco2621#show arp

Protocol Address Age (min) Hardware Addr Type Interface

Internet 64.171.123.200 39 00a0.cc52.5ab3 ARPA FastEthernet0/1

Internet 64.171.123.201 20 00a0.cc52.5ab3 ARPA FastEthernet0/1

Internet 64.171.123.202 17 00a0.cc52.5ab3 ARPA FastEthernet0/1

Internet 64.171.123.1 - 0008.a3b3.b6a1 ARPA FastEthernet0/1

Internet 64.171.123.2 39 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.61 0 Incomplete ARPA

Internet 64.171.123.35 128 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.36 123 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.37 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.38 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.39 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.40 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.41 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.42 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 67.155.215.250 - 0008.a3b3.b6a0 ARPA FastEthernet0/0

Internet 67.155.215.249 2 00a0.c811.2ed0 ARPA FastEthernet0/0

Here's a ping attempt with the resulting log output with debugging on:

cisco2621#ping 64.171.123.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 64.171.123.2, timeout is 2 seconds:

!.!.!

Success rate is 60 percent (3/5), round-trip min/avg/max = 1/1/1 ms

cisco2621#show log

Syslog logging: enabled (0 messages dropped, 0 flushes, 0 overruns)

Console logging: level debugging, 375 messages logged

Monitor logging: level debugging, 0 messages logged

Buffer logging: level debugging, 346 messages logged

Trap logging: level informational, 36 message lines logged

Log Buffer (4096 bytes):

45w3d: ICMP: echo reply rcvd, src 64.171.123.2, dst 64.171.123.1

45w3d: ICMP: echo reply rcvd, src 64.171.123.2, dst 64.171.123.1

45w3d: ICMP: echo reply rcvd, src 64.171.123.2, dst 64.171.123.1

cisco2621#

I believe the physical connections are ruled out. Have verified the duplex settings are correct on all devices. Besides, only traffic to/from the 2621 seems to be affected. Traffic through the 2621 is not experiencing packet loss/performance issues.

Hall of Fame Super Gold

Re: 50% packet loss to/from 2621 router

Chris

Thanks for the additional information. Unfortunately it does not seem to point to the answer. I am surprised that it shows receiving a response but does not show sending the request. But while I think about that I will suggest something else that we can try. Would you turn on debugging for ip packet (with an access list), attempt the ping, and post the debug output.

- first create an access list to use with debug:

access-list 199 permit ip host 64.171.123.2 any

access-list 199 permit ip any host 64.171.123.2

- then run debug using the access list:

debug ip packet 199

- then try the ping

- then capture and post the debug output

- then remember to turn off the debug

HTH

Rick

Re: 50% packet loss to/from 2621 router

is the firewall & router directly connected ?

Try this if they are connected through a switch

connect another PC with the IP 64.171.123.3 & ping both the firewall interface & the router interface.

see what results you get

Community Member

Re: 50% packet loss to/from 2621 router

I'll work on the access lists. In the meantime I setup another computer on that network segment at 64.171.123.3 for example. From that host, when I ping the 2621 I get 50% packet loss. When I ping the Watchguard I get 0% loss. It definitely seems to be an issue with the router.

Hall of Fame Super Gold

Re: 50% packet loss to/from 2621 router

Chris

Perhaps it would help us if we knew a bit more about the topology of the network. In looking at the ARP table that you posted it looks like most of the addresses are behind the firewall from the router (they all have MAC of 0090.7f1f.ad22) and there are 3 addresses at MAC 00a0.cc52.5ab3. So what is the other box on the network?

When you connected another computer was it behind the firewall also? Do you have the same experience of packet loss if you ping to the .200 or .201 or .202 addresses?

cisco2621#show arp

Protocol Address Age (min) Hardware Addr Type Interface

Internet 64.171.123.200 39 00a0.cc52.5ab3 ARPA FastEthernet0/1

Internet 64.171.123.201 20 00a0.cc52.5ab3 ARPA FastEthernet0/1

Internet 64.171.123.202 17 00a0.cc52.5ab3 ARPA FastEthernet0/1

Internet 64.171.123.1 - 0008.a3b3.b6a1 ARPA FastEthernet0/1

Internet 64.171.123.2 39 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.61 0 Incomplete ARPA

Internet 64.171.123.35 128 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.36 123 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.37 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.38 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.39 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.40 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.41 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.42 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 67.155.215.250 - 0008.a3b3.b6a0 ARPA FastEthernet0/0

Internet 67.155.215.249 2 00a0.c811.2ed0 ARPA FastEthernet0/0

HTH

Rick

Community Member

Re: 50% packet loss to/from 2621 router

There's nothing on that segment other than the external Firebox interface, the FastEthernet0/1 Cisco interface and another host with the 3 IP's noted (.200,.201,.202). That segment is defined as a VLAN on an HP switch.

If I ping 64.171.123.1 from the host at 64.171.123.200 I get 50% packet loss

No packet loss when pinging .200 (or .201 or .202) from behind the Firebox. From the internet, if I ping the Firebox at 64.171.123.2 there is no packet loss. Pinging the 2621 at 64.171.123.1 from the internet results in packet loss though. If you were to ping that address you should see packet loss.

Community Member

Re: 50% packet loss to/from 2621 router

BTW, BGP routing is employed on this router. Could that have anything to do with it?

Hall of Fame Super Gold

Re: 50% packet loss to/from 2621 router

Chris

I had noticed the routing table default route was learned from BGP, so was aware that BGP was running. I believe that it is highly unlikely that BGP has anything to do with it.

I did take your suggestion and pinged the router interface. Actually I pinged both interfaces. And I am getting the same behavior on both of them. As a detail I am getting about 40% loss, not 50%. And that convinces me that it is not an extra route in the routing table as I had originally thought it might be.

I am wondering if the router is throttling its own packets for some reason. Is there any shaping or policing of traffic or any QOS configured on the router? Perhaps you can post the config of the router?

HTH

Rick

Community Member

Re: 50% packet loss to/from 2621 router

See attachment for config.

Re: 50% packet loss to/from 2621 router

Could you provide show interface fe0/1 statistics.. as a base line record the stats on Fe0/1 , then clear counters and note any crc errors on the interface, as well as your T1's.

Im just curious on the interface stats.

Community Member

Re: 50% packet loss to/from 2621 router

cisco2621#show int FastEthernet0/1

FastEthernet0/1 is up, line protocol is up

Hardware is AmdFE, address is 0008.a3b3.b6a1 (bia 0008.a3b3.b6a1)

Description: SBC primary T1

Internet address is 64.171.123.1/24

MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:02:30, output 00:00:00, output hang never

Last clearing of "show interface" counters never

Queueing strategy: fifo

Output queue 0/40, 0 drops; input queue 0/75, 0 drops

5 minute input rate 71000 bits/sec, 41 packets/sec

5 minute output rate 647000 bits/sec, 64 packets/sec

1516167236 packets input, 432529126 bytes

Received 743516 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast

0 input packets with dribble condition detected

1320432057 packets output, 4259644730 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

0 babbles, 0 late collision, 2452001 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

Re: 50% packet loss to/from 2621 router

you have high deferred transmissions on the interface.. whats the routers cpu utilization.

Is this packet loss anomaly suddently developed ? were you able to ping this interface witout any packet losses in the past?

Community Member

Re: 50% packet loss to/from 2621 router

It's been this way for a while. I can't say if it has always been this way though. It's just that I decided to finally try to get to the bottom of it. Attached is the cpu utilization. This output shows it being minimal. Even in this state I get the same packet loss.

Re: 50% packet loss to/from 2621 router

Im definately missing something , I don't see anything in the configuration you provided to all causing packet losses, we all know this is only happening on interfatce fe0/1, so it is narrowed to that interface, as a practice I always hardcode fe interfaces at both ends to 100 full duplex even if no crc's or other errors are seen..

I looked of any bugs on your ios 12.0 code but could not find anything on this anomaly.

I will take a second look on bugs.

I would do the following

1- Hardcode trans speed (both ends )

2- Upgrade code from 12.0

Community Member

Re: 50% packet loss to/from 2621 router

You need to use debug ip packet and debug ip icmp

to find out how those packets are being forwarded or dropped.

The simple solution, like someone said, is use these commands

debug ip packet

debug ip icmp

good luck to you :)

Community Member

Re: 50% packet loss to/from 2621 router

You should hard code the speed and duplex on both sides to prevent auto negotiation conflicts.

Scott

Community Member

Re: 50% packet loss to/from 2621 router

If this were a speed/duplex issue, wouldn't it stand to reason that all traffic through this interface would be affected and not just traffic to/from the 2621?

Community Member

Re: 50% packet loss to/from 2621 router

Problem solved.

By enabling debug ip packet I was able to gleen a little more info from the log. I saw that CEF switching was coming into play. Disabling CEF seems to have solved the problem. I don't know enough about CEF to determine why it would cause this behavior though. Should I expect any detrimental effects by disabling CEF?

Hall of Fame Super Gold

Re: 50% packet loss to/from 2621 router

Chris

I am glad that the debug that we suggested was able to help you find the issue. I am surprised that it was caused by CEF and it is likely that an upgrade to more recent code would resolve it.

CEF is an enhanced packet switching path in IOS. If CEF is disabled then you are left with switching paths of fast switching, process switching, etc. CEF has some performance advantage over fast switching (there is no need to process switch the first packet every time the router is forwarding to a destination not present in the fast switching cache) and fast switching certainly has a performance advantage over process switching. So if you have disabled CEF there will be some performance difference. Whether it will be enough to notice or enough to be detrimental is hard for us to tell without knowing a good deal more about the volume of traffic and kind of traffic being processed by the router.

HTH

Rick

2658
Views
5
Helpful
24
Replies
CreatePlease to create content