Packets dropping on a C4948

rvillavi · ‎01-08-2010

Hi all,

I'm having the strangest issue. I'm seen how some of my packets are dropping in one interface of the switch, giga 1/46.

I did some test with sniffers and pings. I set up a SPAM session to mirror the trafic on 1/46 and made some pings. I'm seeing on the pings that I get a 98% of success:

C4948_CARCOISP_MUR#ping 200.38.105.6 repeat 100

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 200.38.105.6, timeout is 2 seconds:
!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!
Success rate is 98 percent (98/100), round-trip min/avg/max = 1/3/12 ms

C4948_CARCOISP_MUR#ping 200.38.105.6 repeat 1000

Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 200.38.105.6, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!.!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!.!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!
!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 98 percent (988/1000), round-trip min/avg/max = 1/2/56 ms

But on the sniffer trace I'm seeing that all the pings are getting there. I'm attaching the sniffer trace for this 1200 pings.

How could it be that I'm seeing that the pings are getting into the switch interface but this pings are drop by the switch?

Am I missing something here? could this be a bug?

The version is: cat4000-i9k91s-mz.122-25.EWA7

I'm attching also, show intefaces, show logg, show run, show ver and the packet capture. The IP of the switch is: 200.38.105.4 and I'm pinging to 200.38.105.6

Any word on this will very helpful

I thank you in advance for all your time and help.

sachinraja · ‎01-08-2010

HI

What are you pinging to ? what device is 200.38.105.6 ? Sometimes on cisco devices i see ping drops, because of the icmp packets being treated with a lowww priority.. sometimes these drops are just informational, and not alarming.. are you having slowness because of this ?

Raj

rvillavi · ‎01-08-2010

Thanks For the answer Raj,

The 200.38.105.6 it's a firewall. What we are seeing is that some times we see that some packets are missing on the normal traffic. We start to acknowledge this when we start too send some streaming and the streaming start to freeze every once in a while.

We did a investigation and we realice that when every we see that the streaming start to failed the switch started to not respond some pings. Then we attached the sniffer and realice that everytime that the trafic on the network fails we see that all the pings get to the switch but the switch is not processing them. dont know why. I'm guessing that this is happening to the traffic also.

Some guys here believe that there's a process that is dropping the packets, so that's why we see the packet get to the interface but failed at the ping output.

What do you thing could be happening here?

I'm attaching you the show tech, maybe that could help.

sachinraja · ‎01-08-2010

Hi

Since the ping drops are towards the ASA, you should be inspecting the firewall, rather than the switch. just to test, can you try pinging the 4948 switch from ASA or other servers on the LAN, and see if you have drops ? I would see the "show interface" statistics on the switch port connecting the firewall, and on the ASA ethernet interfaces.. see if you have any errors, or drops counters on the Show command.. Is the video streaming passing the firewall ? how is the video streaming setup ?

Raj

rvillavi · ‎01-08-2010

Hi Raj,

Yes but I'm seeing in the packet capture that the firewall is replying the ping, and I'm seeing that is reaching the SW.

Yes if I ping from the Firewall to this switch or from other servers to this switch I see that some ping get lost. This happens only in this SW.

Under the show interfaces command I don't see any errors. It's at 0 all ther error counters.

Yes the video Streaming is passing the firewall. I see that the stream works fine almost all the time. The some time for an hour more or less we have the issue and some packets from the stream get lost. I ping all the devices involved and the only one who have problems replying the ping at the moment of the issue is this switch. Then when the issue stops, the swtich replies pings without problems.

I'm attaching a sniffer trace of a ping from the firewall to the switch. This sniffer is attached to a port on the switch and this port is a mirror of the port with the issue.

As you can see, I'm seeing that the switch gets requests but it doesn't replies nothing. I'm not sure why in the sniffer we see that the switch gets a request but it doens't reply, or the switch send a request then gets a reply but on the output we see that the ping failed.

Thanks for all your help, I really appreciate it.

sachinraja · ‎01-08-2010

Hi

The only bug i see with regards to this IOS is:

When VRF Packet Leaking is configured on a Catalyst 4900 series switch with a Supervisor Engine IV, a packet loss of 50 per cent occurs when you ping a Catalyst 4900 series switch VRF interface IP address from a device in the global table.

Packets forwarded by Catalyst 4900 series switch are not impacted.

Workaround: None. (CSCej36831)

but i dont think you are hitting this bug ! how is the cpu utilization of the switch when the issue happens ? when is this issue happening from ? did it start just now or been happening from the time video streaming was implemented ?

Regards

Raj

sachinraja · ‎01-08-2010

Hey

I just saw your show tech, and was going through the configuration.. it looks to me that qos might be dropping your icmp packets and also account for the slowness of your video streaming.. im not able to correlate the qos rules with the devices you have, but i see that there are strict policing configured for data traffic which could be the issue here.. run the command "show policy-map" and see if there are any drops with regards to each class.. since ping / icmp does not fall on any of the class, it goes towards the CLASS-DEFAULT.. Just to troubleshoot, you can remove the "service policy" command on the interface connecting to the fierwall, and see if it resolves your issues..

interface GigabitEthernet1/45

description hacia C7206_CARCO_MUR

no service-policy input SHAPING_IN

!

interface GigabitEthernet1/46

description HACIA_PPmur_PTO1/4

no service-policy input SHAPING

Hope this helps.. all the best..

Raj