Solved: Drop packet with Catalyst 4507R-E ??

trungfotech · ‎08-04-2010

Dear all,

I have a case that I don't have a way to resolve this, can anyone help me to troubleshoot this problem ?

In my DC, I have a Catalyst Switch 4507 connect to one ZTE Switch using port-channel5. And Server Farm connect to 4507 via linecard 24port 1G.

The topology of network in this picture.

When I ping from another local Lan in difference subnet with Server Farm, The delay of packet is very high, and some time the ping reply request timeout,althought all link is GigabitEthernet 1Gb. The Catalyst 4507 connect to ZTE Switch using Interface port-channel5 with 6 bundle link, up bandwidth to 6Gb. Then I log to Catalyst 4507, using "show interface port-channel5" command, we see that drop packet in port-channel5. This is the result of show interface port-channel5

Port-channel5 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0019.e734.3f2f (bia 0019.e734.3f2f)
MTU 1522 bytes, BW 6000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 64/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is N/A
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi1/19 Gi1/20 Gi1/21 Gi1/22 Gi1/23 Gi1/24
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 00:08:17
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 2960374
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 70075000 bits/sec, 88349 packets/sec
5 minute output rate 1509557000 bits/sec, 129272 packets/sec
     48154520 packets input, 4601539612 bytes, 0 no buffer
     Received 813 broadcasts (96 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 206 overrun, 0 ignored
     0 input packets with dribble condition detected
     70527852 packets output, 103036482692 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

But with the same configuration, I replace Catalyst 4507 using Catalyst 3750G, the problem don't appear. Don't have a lose packet in port-channel5. All ping command reply succesfull and time of ping is <1ms.

When I test with topo in picture bellow.

The drop packet error eapear again in port-channel6. there no problem between Catalyst 3750G and ZTE Switch. Only Problem with Catalyst 4507. Alway have a drop packet in Catalyst 4507 but with the same configuration, there no problem with Catalyst 3750.

When I enable netflow on Catalyst 4507, the errors eapear "Netflow stats lost either due to hardware hash collisions or full hardware flow table. Stats lost". But in Catalyst 3750, nothing appear.

So what is the problem ???? Can anyone help me ???

Thanks very much,

Trung.

jorge.calvo · ‎08-04-2010

Hello,

The linecard WS-X4424-GB-RJ45 you are using works with oversubscription 4:1. It has 6 ASICs grouping the ports this way:

ASIC 1 --> Gi1/1 - Gi1/4

ASIC 2 --> Gi1/5 - Gi1/8

ASIC 3 --> Gi1/9 - Gi1/12

ASIC 4 --> Gi1/13 - Gi1/16

ASIC 5 --> Gi1/17 - Gi1/20

ASIC 6 --> Gi1/21 - Gi1/24

Each ASIC shares 1Gb uplink from the linecard to the backplane so, each 4-ports groups share 1Gb of bandwidth.

In your case you are suffering oversubscription on your ports.

If you want to eliminate the drops I will suggest creating the etherchannel with ports:

Gi1/1, Gi1/5, Gi1/9, Gi1/13, Gi1/17 and Gi1/21 for instance. Or any other combination using one port from each group.

The C3750 works at wire speed with no oversubscription, hence you do not have drops.

Hope this helps.

View solution in original post

Jayakrishna Mada · ‎08-04-2010

Trung,

1. what model of linecard is it ?

2. Can you post the output of show interface for these ports,Gi1/19 Gi1/20 Gi1/21 Gi1/22 Gi1/23 Gi1/24.

3. And also the output of show etherchannel load-balance and show ver from the 4500 switch.

JayaKrishna

trungfotech · ‎08-04-2010

Dear JayaKrishna,

My Switch using linecard WS-X4424-GB-RJ45.

With Command show interface, I only have a show interface G1/21

GigabitEthernet1/21 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet Port, address is 0019.e734.3f2c (bia 0019.e734.3f2c)
MTU 1522 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 30/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, Auto-speed, link type is auto, media type is 10/100/1000-TX
input flow-control is off, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:03:01, output never, output hang never
Last clearing of "show interface" counters 00:03:46
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 277
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 10809000 bits/sec, 6670 packets/sec
5 minute output rate 117819000 bits/sec, 10219 packets/sec
     998799 packets input, 260308439 bytes, 0 no buffer
     Received 7 broadcasts (5 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 83 overrun, 0 ignored
     0 input packets with dribble condition detected
     1469094 packets output, 2087577692 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

This is the output of show ver.

Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500-ENTSERVICES-M), Version 12.2(54)SG, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Sun 27-Jun-10 04:53 by prod_rel_team
Image text-base: 0x10000000, data-base: 0x121F5B44

ROM: 12.2(31r)SGA3
Pod Revision 16, Force Revision 31, Tie Revision 32

SW4507_T14 uptime is 8 hours, 58 minutes
Uptime for this control processor is 8 hours, 37 minutes
System returned to ROM by reload
System image file is "bootflash:cat4500-entservices-mz.122-54.SG.bin"

cisco WS-C4507R-E (MPC8540) processor (revision 13) with 524288K bytes of memory.
Processor board ID FOX1412GCTW
MPC8540 CPU at 800Mhz, Supervisor V-10GE
Last reset from Reload
2 Virtual Ethernet interfaces
56 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.

The Catalyst 4507 I have temporary replace by Catalyst 3750, as now in Catalyst 4507 don't have any traffic, but when I integrated Catalyst 4507 in to network, I have configured "Port-channel load-balance src-dst-ip".

And I have a output of show interface summary when connected Catalyst 4507 to network

Interface               IHQ   IQD OHQ   OQD RXBS RXPS TXBS TXPS TRTL
-------------------------------------------------------------------------
GigabitEthernet1/11      0     0    0     0     0    0     0    0    0
GigabitEthernet1/12      0     0    0     0     0    0     0    0    0
GigabitEthernet1/13      0     0    0     0     0    0     0    0    0
GigabitEthernet1/14      0     0    0     0     0    0     0    0    0
GigabitEthernet1/15      0     0    0     0     0    0     0    0    0
GigabitEthernet1/16      0     0    0     0     0    0     0    0    0
* GigabitEthernet1/17      0     0    0     0     0    1 19000   48    0
* GigabitEthernet1/18      0     0    0     0 3512000 530 359000 409    0
* GigabitEthernet1/19      0     0    0    23 8505000 11297 306056000 26487    0
* GigabitEthernet1/20      0     0    0     0 8454000 11855 297997000 25383    0
* GigabitEthernet1/21      0     0    0 683333 8081000 9837 226494000 19306    0
* GigabitEthernet1/22      0     0    0 836787 9295000 12220 226123000 19345    0
* GigabitEthernet1/23      0     0    0 439225 15930000 20799 225069000 19363    0
* GigabitEthernet1/24      0     0    0 1053591 19426000 22368 229235000 19484    0

Thank you very much,

Trung.

jorge.calvo · ‎08-04-2010

Hello,

The linecard WS-X4424-GB-RJ45 you are using works with oversubscription 4:1. It has 6 ASICs grouping the ports this way:

ASIC 1 --> Gi1/1 - Gi1/4

ASIC 2 --> Gi1/5 - Gi1/8

ASIC 3 --> Gi1/9 - Gi1/12

ASIC 4 --> Gi1/13 - Gi1/16

ASIC 5 --> Gi1/17 - Gi1/20

ASIC 6 --> Gi1/21 - Gi1/24

Each ASIC shares 1Gb uplink from the linecard to the backplane so, each 4-ports groups share 1Gb of bandwidth.

In your case you are suffering oversubscription on your ports.

If you want to eliminate the drops I will suggest creating the etherchannel with ports:

Gi1/1, Gi1/5, Gi1/9, Gi1/13, Gi1/17 and Gi1/21 for instance. Or any other combination using one port from each group.

The C3750 works at wire speed with no oversubscription, hence you do not have drops.

Hope this helps.

trungfotech · ‎08-04-2010

Dear jorqe,

Thank you very much for your's reply, tonight I will apply your method for my switch at downtime and have a reply as soon as possible.

Thanks,

Trung.

trungfotech · ‎08-05-2010

Dear jorqe,

Thank you very much again for your's reply, you are very great. In yesterday, I have configured my Switch as your recommend and monitor for one day. There no drop packet, ping reply very good and network normal operate.

Thank you again

Trung.