troubleshooting heavy I/O on 6513/Sup2

pfc-corporate · ‎12-14-2009

A few months ago, I moved back into one of my buildings where the entire building is ran through a 6513 with Sup2/PFC2/MSFC2. This switch is in hybrid mode and the Supervisors are on 8.5(4) yes, I know it's old and hybrid mode has gone the way of the dodo, but I'm stuck like this for now.. Prior to moving back in, the only thing running through the switch was an ASA 5520 firewall and some Dell servers.

Now I have users and phones connected to the switch on 10/100 PoE blades and we are running VoIP. Handsets are Polycom IP series, computers run through the switches in the phones.

Since deploying the phones and workstations, I've had average CPU loads in the 40-60% range. 40-50% of the usage shows from "kernel and idle" process, and an average of 20-25% is from SptBpduTx process

I'm seeing a lot of the following in the log history as well. It comes from all ports phones/devices are connected o.

2009 Dec 14 17:01:14 EST -05:00 %MCAST-5-RX_IGMPREPORT:IGMP: Rcvd IGMP Report 01-00-5e-00-00-fb on 13/22
2009 Dec 14 17:01:14 EST -05:00 %MCAST-5-RX_IGMPREPORT:IGMP: Rcvd IGMP Report 01-00-5e-00-00-fb on 13/2
2009 Dec 14 17:01:14 EST -05:00 %MCAST-5-RX_IGMPREPORT:IGMP: Rcvd IGMP Report 01-00-5e-00-00-fb on 12/30
2009 Dec 14 17:01:15 EST -05:00 %MCAST-5-RX_IGMPREPORT:IGMP: Rcvd IGMP Report 01-00-5e-00-00-fc on 12/47

The interesting thing, is that if we go back a few years, I was running the same phones and workstations off the same switch and linecards, and CPU loads where minimal, average of < 410%. I also didn't have these MCAST-5-RX_IGMPREPORT's in the log. The one difference is that the Polycom phones now hve newer BootROM and phone software on them.

Furthermore, I can't get my Polycom phones to run on the auxiliary/voice vlan and pull IP's, a sympton that occured as part of new BootROM/phone software.

I'm hoping someone can help me diagnose these problems in general.

glen.grant · ‎12-14-2009

How many vlans are defined on the switch ? Sounds like the spanning tree process is taking a lot of the cpu . Also on any trunks "manually prune of all vlans that are not needed on the trunks . If you do not prune it will allocate a spanning tree instance for all vlans on the box on each trunk and this will drive up the spanning tree process . Don't know about the voice vlan , have never used it on a catos box . You may have to figure out where all the multicast messages are coming from . Does multicast routing even need to be turned on ?

vvasisth · ‎12-15-2009

before i explain the error message let me set your expectations rite when we talk about CPU utilization anything under 60% is acceptable and is not an alarm. So you can take it easy for the time being.

coming back to the error message :-

%MCAST-5-RX_IGMPREPORT:IGMP: Rcvd IGMP Report

MCAST-4-RX_IGMPREPORT:IGMP: Rcvd IGMP Report [chars] on [dec]/[dec]

Explanation: This message indicates that the switch has received an IGMP report message for the group number on module number/port number; [chars] is the group number, and
[dec]/[dec] is the module number/port number. Action: This message is provided for information only.

The messages you are receiving are informational only and are generated when the switch receives IGMP report packets with a multicast MAC address that starts with
01-00-5e-00-00-xx. This Layer 2 range of addresses is equivalent to a Layer 3 multicast address range between 224.0.0.0 and 224.0.0.255. These addresses are reserved for the use
of routing protocols and other low-level topology discovery or maintenance protocols, such as gateway discovery and group membership reporting.  

This does not mean anything is wrong with the switch, these packets floating around from a rogue PC or server.

pfc-corporate · ‎12-15-2009

There are about 25 VLAN's on the switch, which is about the same I had in the past.

The heavy I/O is a new symptom from when I last used the switch and today. O nly major difference is new software o nthe phones (and my associated voice vlan issue). So I wonder if that's causing the problem.

This excess multicast traffic is also new, so I'm going to have to see if I can track that down. That could be the problem as well. I do have some permanent guest laptops on my network on an isolated VLAN which is restricted to get to the internet and my DHCP/DNS servers plus a few internal websites via ACL's.

vvasisth · ‎12-15-2009

get me the output of "show proc cpu"

this will tell us the reason for high cpu

pfc-corporate · ‎12-15-2009

Here is one as of a few seconds ago. It's rare for me to see processor EVER below 40%. it uses to be 5-10% average in the past.

CPU utilization for five seconds:  41.14%
                      one minute:  42.00%
                    five minutes:  41.02%

PID Runtime(ms) Invoked    uSecs    5Sec    1Min    5Min    TTY Process
--- ----------- ---------- -------- ------- ------- ------- --- ---------------
1   0           649277143  0         58.86%  58.00%  58.98% -2  Kernel and Idle
2   1           25         1000       0.00%   0.00%   0.00% -2  DiskInodeRefres
3   2           75         1000       0.00%   0.00%   0.00% -2  Flash MIB Updat
4   3980696     372457925  162000     0.04%   0.00%   0.00% -2  PowerMgmt      
5   0           1          0          0.00%   0.00%   0.00% -2  L2L3IntHdlr    
6   0           1          0          0.00%   0.00%   0.00% -2  RecoveryL2L3   
7   0           1          0          0.00%   0.00%   0.00% -2  L2L3PatchRev   
8   247736      37467401   1000       0.00%   0.00%   0.00% -2  EobRecoveryProc
9   1124        7989       1000       0.00%   0.00%   0.00% -2  SynDiags       
10  41912       121480     6000       0.00%   0.00%   0.00% -2  GenMsgHndlr    
11  3165890     384574115  130000     0.02%   0.00%   0.00% -2  SynConfig      
12  133194090   105900153  10000      0.00%   0.00%   0.00% -2  TempMon        
13  0           1          0          0.00%   0.00%   0.00% -2  System_CrashInf
14  2550309580  674557295  134000     7.58%   7.92%   7.98% -2  IDLE_Tasks     
15  11625137    82736038   3000       0.00%   0.00%   0.00% -2  Debug Port Coun
16  7700344     983648877  6000       0.00%   0.00%   0.00% -2  SL_TASK        
17  737337      224004486  118000     0.02%   0.00%   0.00% -2  RedundantTask  
18  50838442    329322116  3000       0.06%   0.00%   0.00% -2  Status Poll    
19  22650       4683907    1000       0.00%   0.00%   0.00% -2  SWPoll64bCnt   
20  3234926     2165453033 101000     0.00%   0.00%   0.00% -2  RouterConfig   
21  366465      56187594   1000       0.00%   0.00%   0.00% 0   Lyra Freeze Tas
22  1874        8284       232000     0.00%   0.00%   0.00% 0   HavailTask     
23  21114995    294458575  314000     0.00%   0.00%   0.00% -2  SyncTask       
24  1422734     117701409  4000       0.00%   0.00%   0.00% -2  Earl           
25  7633873     55161161   4000       0.06%   0.00%   0.00% -2  TablefullTask  
26  99718283    44550183   142000     0.40%   0.00%   0.00% -2  DTP_Rx         
27  948952      2486707    108000     0.00%   0.00%   0.00% -2  EthChnlRx      
28  0           1          0          0.00%   0.00%   0.00% -2  llcSSTPFlood   
29  15944898    2681734    102000     0.00%   0.00%   0.00% -2  EthChnlConfig  
30  167666      772529     4000       0.00%   0.00%   0.00% -2  FlexlinkRx     
31  3521        292472     4000       0.00%   0.00%   0.00% -2  SecurityRx     
32  168         2420       1000       0.00%   0.00%   0.00% -2  ACL            
33  2           5          1000       0.00%   0.00%   0.00% -2  VaclLog        
34  0           5          0          0.00%   0.00%   0.00% -2  ArpInspection  
35  385031640   370592078  5000       0.32%   0.00%   0.00% -2  AclStats       
36  10101365    836061020  3000       0.04%   0.00%   0.00% 0   L3Aging        
37  3379974     7448504    8000       0.00%   0.00%   0.00% 0   NetFlow        
38  314223394   77113303   501000     0.02%   0.00%   0.00% 0   Fib            
39  0           1          0          0.00%   0.00%   0.00% -2  Fib_bg_task    
40  1362659     97280252   2000       0.00%   0.00%   0.00% -2  DhcpSnooping   
41  260770      1134001    3000       0.00%   0.00%   0.00% -2  Authenticator_S
42  0           1          0          0.00%   0.00%   0.00% -2  Backend_SM     
43  94          701        1000       0.00%   0.00%   0.00% -2  dot1x_rx       
44  0           1          0          0.00%   0.00%   0.00% -2  Dot1x_Dhcp     
45  0           2          0          0.00%   0.00%   0.00% -2  GrpManager     
46  0           2          0          0.00%   0.00%   0.00% -2  Backend_Rx     
47  0           2          0          0.00%   0.00%   0.00% -2  Radius Acct Pro
48  397556212   119334712  18000      0.96%   0.00%   0.00% 0   SptTimer       
49  378482807   96754484   101000     2.96%   2.92%   2.98% 0   SptBpduRx      
50  4289601473  116635236  116000    18.44%  20.76%  20.95% 0   SptBpduTx      
51  1065238     1032136    3000       0.00%   0.00%   0.00% 0   sptTraps       
52  2           2          1000       0.00%   0.00%   0.00% -2  GL2Prot_Tunnel 
53  903294      115326425  6000       0.00%   0.00%   0.00% -2  VtpTimer       
54  105521448   631240374  3000       0.16%   0.00%   0.00% 0   Read Stats Task
55  34080       11240243   1000       0.00%   0.00%   0.00% 0   QDE Task       
56  5301283     24404209   2000       0.00%   0.00%   0.00% 0   BcastSuppr Task
57  676060681   209321023  6000       0.36%   0.00%   0.00% -2  oam_tx_proc    
58  4           30         1000       0.00%   0.00%   0.00% -2  oam_rx_proc    
59  78411417    20932105   5000       0.08%   0.00%   0.00% -2  oam_monitoring_
60  2552232     31231      114000     0.00%   0.00%   0.00% -2  NvramMonitor   
61  79669986    120204478  151000     0.08%   0.00%   0.00% -2  cdpd           
62  1667686298  112328103  55000      1.54%   1.00%   1.00% -2  cdpdtimer      
63  18893       1032135    1000       0.00%   0.00%   0.00% -2  sptTraps       
64  87807       11240222   1000       0.00%   0.00%   0.00% -2  ProtocolFilter 
65  0           1          0          0.00%   0.00%   0.00% -2  MacMoveNotTask 
66  3598        18737      1000       0.00%   0.00%   0.00% -2  telnetd        
67  1178        18737      1000       0.00%   0.00%   0.00% -2  connecthelper  
68  0           1          0          0.00%   0.00%   0.00% -2  tftpd          
69  73328       1497525    379000     0.00%   0.00%   0.00% -2  sntp_recd      
70  52416       1756676    2000       0.00%   0.00%   0.00% -2  sntp_xmitd     
71  690430141   3424680301 23000      1.86%   1.92%   1.98% -2  ProtocolTimer  
72  40366       11240233   1000       0.00%   0.00%   0.00% -2  ciscoRmonTimer 
73  36739       11240231   1000       0.00%   0.00%   0.00% -2  ciscoUsrHistory
74  0           1          0          0.00%   0.00%   0.00% -2  rmonMediaIndep 
75  76256       114409     2000       0.00%   0.00%   0.00% -2  SnmpTraps      
76  1237734     22482722   2000       0.00%   0.00%   0.00% -2  memPoolMain    
77  0           1          0          0.00%   0.00%   0.00% -2  Acct Send Bkg  
78  0           1          0          0.00%   0.00%   0.00% -2  l2t_server     
80  1383245280  30210382   168000     2.16%   2.92%   2.98% -2  SysLogTask     
81  277096      112215165  1000       0.00%   0.00%   0.00% -2  pinggateA      
82  5001186     21497251   9000       0.08%   0.00%   0.00% -2  McastRx        
83  0           1          0          0.00%   0.00%   0.00% -2  MacAuthProc    
84  77986       863606     2000       0.00%   0.00%   0.00% -2  AuthControlMgr 
85  76492013    83020221   2000       0.06%   0.00%   0.00% -2  McastIgmpTimer 
86  0           1          0          0.00%   0.00%   0.00% -2  RMON AlarmTimer
87  1828275809  110365259  21000      1.80%   1.00%   1.00% -2  DeviceLinkChk  
88  248         234        2000       0.00%   0.00%   0.00% -2  IGMPQuerierProc
89  1           1          0          0.00%   0.00%   0.00% -2  M-MLS_stats    
90  10          26         6000       0.00%   0.00%   0.00% -2  M-MLS_manager  
91  578         1350       67000      0.00%   0.00%   0.00% -2  QoSTask        
92  3474111     220590829  2000       0.00%   0.00%   0.00% -2  GBIC Process   
93  49655789    1427576131 2000       0.00%   0.00%   0.00% -2  VlanStatsTask  
94  4           1          3000       0.00%   0.00%   0.00% -2  HPActive       
95  138         1          1000       0.00%   0.00%   0.00% -2  HPTrapMgr      
97  1           2          0          0.00%   0.00%   0.00% -2  gentcp_act     
143 4359721     1460838    102000     0.00%   0.00%   0.00% -2  VtpRx          
144 131692      17701229   1457000    0.00%   0.00%   0.00% 0   Console        
145 212624474   1910800458 243000     0.00%   0.00%   0.00% -2  snmpdm         
146 745739851   1491093387 5000       0.00%   0.00%   0.00% -2  SNMP_THREAD    
147 1414467     473        61000      2.04%   2.92%   2.98% 2257468820 telnet147      
156 113108      108307     201000     0.00%   0.00%   0.00% -2  AclManager

sachinraja · ‎12-15-2009

Hi pfc

looks like there are excessive stpbpdu transmissions on your network... did you check the stp adjacencies on your network and see if there are loops ? did u introduce any switch/host new on the network, after which the cpu has gone up?

Raj