Re: Packet Loss between switches

mshanmugasundaram · ‎07-17-2018

Sir,

I have switch Cisco WS-C3550-24 connected with cisco WS-C4507R.

These 2 having same subnet 10.19.250.0/24. In 4507 the switchport Gi5/30 in vlan 1 10.19.250.8/24 is having connectivity with 3550 switchport Fa0/4 in vlan 3 10.19.250.11/24.

I am facing the packet loss while ping from 4507 the IP 10.19.250.11 and vice versa.

Pls suggest to resolve the issue.

Ping from switch 4507

sw_4507>ping
Protocol [ip]:
Target IP address: 10.19.250.11
Repeat count [5]: 1000
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 10.19.250.11, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (999/1000), round-trip min/avg/max = 1/8/48 ms

The configuration in Switch 4507

sw_4507>sh int gi5/30
GigabitEthernet5/30 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet Port, address is 0011.bbf7.b0bd (bia 0011.bbf7.b0bd)
Description: *** To Firewall Switch Fa0/4 ***
MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec,
reliability 255/255, txload 156/255, rxload 214/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, link type is auto, media type is 10/100/1000-TX
input flow-control is off, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:18, output never, output hang never
Last clearing of "show interface" counters 5w5d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 30097465
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 83987000 bits/sec, 11477 packets/sec
5 minute output rate 61333000 bits/sec, 11011 packets/sec
30419571281 packets input, 30801012836912 bytes, 0 no buffer
Received 3306809 broadcasts (58876 multicast)
0 runts, 0 giants, 0 throttles
1 input errors, 1 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
27125693186 packets output, 16137908631863 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
sw_4507>sh int vlan1
Vlan1 is up, line protocol is up
Hardware is Ethernet SVI, address is 0012.7fc2.da7f (bia 0012.7fc2.da7f)
Internet address is 10.19.250.8/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 2y10w
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 791000 bits/sec, 405 packets/sec
5 minute output rate 2000 bits/sec, 2 packets/sec
L3 in Switched: ucast: 10648440547 pkt, 2097486896620 bytes - mcast: 0 pkt, 0 bytes
L3 out Switched: ucast: 28592065 pkt, 4545821658 bytes - mcast: 0 pkt, 0 bytes
10660353291 packets input, 2098543321632 bytes, 0 no buffer
Received 11912744 broadcasts (59 IP multicast)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
256239974 packets output, 16097529395 bytes, 0 underruns
0 output errors, 0 interface resets
0 output buffer failures, 0 output buffers swapped out

sw_4507#show running-config int gi5/30
Building configuration...

Current configuration : 134 bytes
!
interface GigabitEthernet5/30
description *** To Firewall Switch Fa0/4 ***
switchport mode access
logging event link-status
end

--------

The configuration in Switch 3550

switch_3550>show int vlan 3
Vlan3 is up, line protocol is up
Hardware is EtherSVI, address is 000e.8393.3500 (bia 000e.8393.3500)
Internet address is 10.19.250.11/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 225
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 12000 bits/sec, 15 packets/sec
5 minute output rate 1000 bits/sec, 6 packets/sec
57058721 packets input, 214006388 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
40276945 packets output, 2727081404 bytes, 0 underruns
0 output errors, 0 interface resets
0 output buffer failures, 0 output buffers swapped out

switch_3550>show int fa0/4
FastEthernet0/4 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 000e.8393.3504 (bia 000e.8393.3504)
Description: "PHASE4 10.19.250.8 switch Gi5/30"
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 214/255, rxload 154/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output 00:00:06, output hang never
Last clearing of "show interface" counters 11w6d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 60489000 bits/sec, 10947 packets/sec
5 minute output rate 83984000 bits/sec, 11453 packets/sec
2137050634 packets input, 4261119307 bytes, 0 no buffer
Received 13994229 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 5210588 multicast, 0 pause input
0 input packets with dribble condition detected
4207481560 packets output, 4176428799 bytes, 178621592 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
178621592 output buffer failures, 0 output buffers swapped out

switch_3550#show running-config int fa0/4
Building configuration...

Current configuration : 131 bytes
!
interface FastEthernet0/4
description "PHASE4 10.19.250.8 switch Gi5/30"
switchport access vlan 3
switchport mode access
end

Leo Laohoo · ‎07-17-2018

@mshanmugasundaram wrote:

reliability 255/255, txload 156/255, rxload 214/255

W

T

F

Georg Pauwen · ‎07-17-2018

Hello,

on the 3550, post the output of 'show buffers'...

mshanmugasundaram · ‎07-17-2018

Hello,

The Show Buffers output

switch_3550#show buffers
Buffer elements:
484 in free list (500 max allowed)
327434524 hits, 0 misses, 0 created

Public buffer pools:
Small buffers, 104 bytes (total 50, permanent 50, peak 110 @ 7w0d):
50 in free list (20 min, 150 max allowed)
225242774 hits, 20 misses, 60 trims, 60 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent 25, peak 40 @ 7w0d):
23 in free list (10 min, 150 max allowed)
66133 hits, 28 misses, 84 trims, 84 created
0 failures (0 no memory)
Big buffers, 1536 bytes (total 555, permanent 128, peak 573 @ 6w0d):
148 in free list (128 min, 512 max allowed)
171261771 hits, 55397 misses, 34756 trims, 35183 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 0, permanent 0):
0 in free list (0 min, 0 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
0 in free list (0 min, 0 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 0, permanent 0):
0 in free list (0 min, 0 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)

Interface buffer pools:
Ram Access buffers, 1524 bytes (total 104, permanent 100, peak 166 @ 7w0d):
104 in free list (100 min, 150 max allowed)
92446527 hits, 1631 misses, 6498 trims, 6502 created
0 failures (0 no memory)
CPU1 buffers, 1524 bytes (total 4, permanent 4):
4 in free list (0 min, 4 max allowed)
0 hits, 0 misses
CPU13 buffers, 1524 bytes (total 32, permanent 32):
1 in free list (0 min, 32 max allowed)
79 hits, 2 fallbacks
CPU11 buffers, 1524 bytes (total 32, permanent 32):
16 in free list (0 min, 32 max allowed)
77 hits, 0 misses
CPU6 buffers, 1524 bytes (total 10, permanent 10):
1 in free list (0 min, 10 max allowed)
4324436 hits, 432443 fallbacks
CPU12 buffers, 1524 bytes (total 20, permanent 20):
10 in free list (0 min, 20 max allowed)
10 hits, 0 misses
CPU3 buffers, 1524 bytes (total 20, permanent 20):
10 in free list (0 min, 20 max allowed)
10 hits, 0 misses
CPU0 buffers, 1524 bytes (total 200, permanent 200):
200 in free list (0 min, 200 max allowed)
0 hits, 0 misses
CPU5 buffers, 1524 bytes (total 200, permanent 200):
200 in free list (0 min, 200 max allowed)
0 hits, 0 misses
CPU8 buffers, 1524 bytes (total 32, permanent 32):
16 in free list (0 min, 32 max allowed)
1473826686 hits, 0 misses
CPU4 buffers, 1524 bytes (total 450, permanent 450):
222 in free list (0 min, 450 max allowed)
2168775078 hits, 0 misses
CPU2 buffers, 1524 bytes (total 10, permanent 10):
1 in free list (0 min, 10 max allowed)
23760127 hits, 4231692 fallbacks
CPU10 buffers, 1524 bytes (total 8, permanent 8):
0 in free list (0 min, 8 max allowed)
8 hits, 0 fallbacks
CPU9 buffers, 1524 bytes (total 32, permanent 32):
16 in free list (0 min, 32 max allowed)
16 hits, 0 misses
IPC buffers, 4096 bytes (total 3, permanent 2, peak 5 @ 7w0d):
2 in free list (1 min, 8 max allowed)
4 hits, 1 fallbacks, 1611 trims, 1612 created
0 failures (0 no memory)

Header pools:

Georg Pauwen · ‎07-17-2018

Hello,

try and configure the values below (all done in global configuration mode):

buffers small initial 80
buffers small permanent 120
buffers middle initial 50
buffers small permanent 80
buffers big initial 500
buffers small permanent 900
buffers FastEthernet0/4 initial 500
buffers FastEthernet0/4 permanent 1000

mshanmugasundaram · ‎07-17-2018

Hello,

Pls confirm buffers small/middle/big permanent values..

Thank you.

buffers small initial 80
buffers small permanent 120
buffers middle initial 50
buffers small permanent 80
buffers big initial 500
buffers small permanent 900
buffers FastEthernet0/4 initial 500
buffers FastEthernet0/4 permanent 1000

Seb Rupik · ‎07-17-2018

Hi there,
It looks as if the 3550 is strugling to cope with the volume of traffic leaving fa0/4:

...
reliability 255/255, txload 214/255, rxload 154/255
...
4207481560 packets output, 4176428799 bytes, 178621592 underruns
...
178621592 output buffer failures, 0 output buffers swapped out

Likewise the 3850 is showing output drops indicating the 3550 is not able to keep up with the incoming flow of traffic.

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 30097465

You have two options, either configure QoS to prioritise important traffic across this congested link, or create a port-channel between the two switches to increase the bandwidth available.

Or just replace the 3550 with a 1Gb switch.

cheers,
Seb.