11-09-2011 07:13 PM - edited 03-04-2019 02:13 PM
Hello Community,
I have installed a Catalyst 3560 as my internal router. I have created 4 vlans to route the traffic.
Here are the interface "show run"
vlan internal allocation policy ascending
!
interface FastEthernet0/1
switchport access vlan 2
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/2
switchport access vlan 3
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/3
switchport access vlan 4
switchport mode access
spanning-tree portfast
I have been getting notifications of large packet loss and latency for itnernal traffic going to servers. I have a simple setup of nagios who pings servers and will notify me of large packet loss or complete packet loss. I have implemented this L3 switch a few hours ago, I am currently running a constant ping to my servers, but I am not niticing packet loss right now.
This packet loss happened 4 time spans within the last hour of many large packet losses to all my servers within all subnets. It is now stopped.
I am not sure why this would be happening, the predessor of the internal router was just a normal linux box with 3 NICs on it. Nothing crazy going on since the business is closed.
Thanks in Advance!
Solved! Go to Solution.
11-09-2011 07:35 PM
John,
The issue may not be your switch. It might be the server or the NICs. Besides the usual ping, you can also ping using larger packet size and see if you see packet drops.
example:
first clear the interface
clear interface gigabitEthernet x/x/x
then
ping 192.168.1.10 repeat 500 size 1500
Then check the interface for packet drops
sh ip int gi x/x/x/
HTH
11-10-2011 03:27 PM
Hi John,
Your 3560 FE0/2 port stats are fine. I wouldn't change anything for now.
Are you still encountering the high latency and packet drop issue? I would suggest to perform SPAN and sniff with Whireshark if you do.
Sent from Cisco Technical Support iPhone App
11-09-2011 07:35 PM
John,
The issue may not be your switch. It might be the server or the NICs. Besides the usual ping, you can also ping using larger packet size and see if you see packet drops.
example:
first clear the interface
clear interface gigabitEthernet x/x/x
then
ping 192.168.1.10 repeat 500 size 1500
Then check the interface for packet drops
sh ip int gi x/x/x/
HTH
11-09-2011 08:06 PM
HTH,
I did not need to set a larger packet to see the issue. I have been watching it and noticed that all my pings to other subnets all went "out of wack" and started having response times up to 900 and even a couple time outs. This happened at the same time for all the constant pings to the other subnets.
I have a Constant. This constant is a ping to another server within the same subnet. That constant was not affected while all the other pings went nuts.
This never happened until I installed the L3.
I am seeing more and more sets of 10 packets that go from less than 1 up to 300-900 in a row. A few packets may drop too.
Thanks
John
11-09-2011 11:09 PM
hi john,
you may want to check the interface stats on the 3560 which connects to your servers. look under the txload and rxload. i suspect this maybe due to a link congestion. also, ensure 3560 FE ports and servers are hardcoded the same speed/duplex.
11-10-2011 07:32 AM
Hi John,
I have checked the ports and the txload and the rxload. Below is the output of my command: show interface FastEthernet 0/2
I have also inserted a picture of my Dell Powerconnect switch that is hooked up to my Catalyst. Is there something special I should do to connect the two. I noticed that my catalyst is using Full-Duplex 100Mb/s and my switch is using a 1000M, but shoudl that 1000M downgrade to 100M? My servers are mostly 100M too.
#show interface fastethernet 0/2
FastEthernet0/2 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 0007.0e76.0c84 (bia 0007.0e76.0c84)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 90/255, rxload 40/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output 00:00:05, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 15735000 bits/sec, 4477 packets/sec
5 minute output rate 35624000 bits/sec, 3555 packets/sec
169356269 packets input, 76279088303 bytes, 0 no buffer
Received 53247 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 36940 multicast, 0 pause input
0 input packets with dribble condition detected
132627013 packets output, 160530213511 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
11-10-2011 03:27 PM
Hi John,
Your 3560 FE0/2 port stats are fine. I wouldn't change anything for now.
Are you still encountering the high latency and packet drop issue? I would suggest to perform SPAN and sniff with Whireshark if you do.
Sent from Cisco Technical Support iPhone App
11-11-2011 06:12 AM
Hello John,
Yes I am still having issues.
Last night it was between 7:30 and 10:00. I get these Nagios packet loss notification emails and my blackberry blew up between those times. Then everything stopped. I noticed that the night before it happened at the same time.
Remember that all this happened after I installed this L3 switch which replaced a Linux Server with 3 NICs in it.
I am confused and dont know where to look. I do not know much about sniffing or wireshark.
John
11-11-2011 07:38 AM
Packet sniffing could be a lot of work for you if that's the case.
I would go back to what Reza had post. Did you find anything unusual with the server logs?
Sent from Cisco Technical Support iPhone App
11-11-2011 07:53 AM
Hi John,
I just experienced latency for a few minutes about 30 minutes ago. I have no idea what is going on. I was getting alerts from Nagios and users were experiencing latency to server.
I have checked event logs from servers and everything looks okay.
I am running wireshark on a server. Anything I should look for?
11-11-2011 08:05 AM
network acting up again. Large amounts of latency. Connections to servers freezing even
11-11-2011 08:21 AM
So things were getting so bad in my network that I had to revert to the old Linux server router.
I have a server constantly pinging all the subnets.
When on the L3 switch, most packets were <1ms, but a few packets would be <5ms - <50ms, it would be mixed.
But now that I reverted back to my old linux box, I see that all the packets are <1ms.
Things are normal now, but now I know that the L3 switch was the culprit.
Why would my L3 cause such issues? How can I diagnose it?
11-11-2011 08:27 AM
could my VLANS be causing this issue?
Below is my command output:
arisrouter>show vlan
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Fa0/5, Fa0/6, Fa0/7, Fa0/8
Fa0/9, Fa0/10, Fa0/11, Fa0/12
Gi0/1, Gi0/2
2 VLAN0002 active Fa0/1, Fa0/13, Fa0/14, Fa0/15
Fa0/16
3 VLAN0003 active Fa0/2, Fa0/17, Fa0/18, Fa0/19
Fa0/20
4 VLAN0004 active Fa0/3, Fa0/21, Fa0/22, Fa0/23
Fa0/24
5 VLAN0005 active Fa0/4
1002 fddi-default act/unsup
1003 token-ring-default act/unsup
1004 fddinet-default act/unsup
1005 trnet-default act/unsup
VLAN Type SAID MTU Parent RingNo BridgeNo Stp BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
1 enet 100001 1500 - - - - - 0 0
2 enet 100002 1500 - - - - - 0 0
3 enet 100003 1500 - - - - - 0 0
VLAN Type SAID MTU Parent RingNo BridgeNo Stp BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
4 enet 100004 1500 - - - - - 0 0
5 enet 100005 1500 - - - - - 0 0
1002 fddi 101002 1500 - - - - - 0 0
1003 tr 101003 1500 - - - - - 0 0
1004 fdnet 101004 1500 - - - ieee - 0 0
1005 trnet 101005 1500 - - - ibm - 0 0
Remote SPAN VLANs
------------------------------------------------------------------------------
Primary Secondary Type Ports
------- --------- ----------------- ------------------------------------------
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: