cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
919
Views
0
Helpful
9
Replies

Cisco 4900M High CPU

                   Hi,

We are observing high cpu of one of the backbone switch,

We enable eem script & below are the cpture

But we are not able to pin-point the issue.

event manager applet high_cpu

event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op gt entry-val "65" poll-interval 5 action 1.01 syslog msg "------HIGH CPU DETECTED----, CPU: $_snmp_oid_val %"

action 1.02 cli command "enable"

action 1.03 cli command "term len 0"

action 1.04 cli command "debug platform packet all receive buffer"

action 1.05 cli command "show platform health | redirect slot0:high_cpu1"

action 1.06 cli command "show proc cpu sort | redirect slot0:high_cpu2"

action 1.07 cli command "show platform cpu packet statistics | redirect slot0:high_cpu3"

action 1.08 cli command "show platform cpu packet buffered | redirect slot0:high_cpu4"

action 1.09 cli command "show platform health | redirect slot0:high_cpu5"

action 1.10 cli command "show proc cpu sort | redirect slot0:high_cpu6"

action 1.11 cli command "show platform cpu packet statistics | redirect slot0:high_cpu7"

action 1.12 cli command "show platform cpu packet buffered | redirect slot0:high_cpu8"

action 1.13 cli command "show clock | redirect slot0:high_cpu9"

action 1.14 cli command "undebug all"

action 1.15 cli command "conf t"

action 1.16 cli command "no event

Switch version: cat4500e-entservicesk9-mz.122-50.SG3.bin

Model:  WS-C4900M 3 slot switch

Pls suggest

Br/Subhojit

9 Replies 9

Karthick Murugan
Cisco Employee
Cisco Employee

Hi Subhojit,

I looks the EEM couldn't complete the task because of VTY session exhaustion. However, with the provided outupt I could see two misbehaving process.

K5QosDhmMan Rate DBL   2.00   3.95      7      8  100  500    4   4    3  24328:41

K5L3Unicast Adj Tabl         2.00   7.91     15      8  100  500   22   4    0  1453:32

Could you please provide the output of "show platform cpu packet statistics"?

Thanks & Regards,
Karthick Murugan
CCIE#39285

Thanks & Regards, Karthick Murugan CCIE#39285

HI,

I will share the cpature by tomorrow

But, Pls confirm is there any error on the script

Why the vty session exhausted

What does the misbehaving process means exactly

Br/Subhojit

HI,

I guess for every redirect command uses a vty session and thats why we might see this issue. I had a look at the logs again.

I see lots of SA Miss which is nothing but increase Source Address Miss. Have you verified if there are any topology changes(show spanning-tree detail | in ieee|exec|from|occurred)

?

switch##more slot0:high_cpu2   3

RkiosSysPacketMan:

Total                5 sec avg 1 min avg 5 min avg 1 hour avg

-------------------- --------- --------- --------- ----------

           641603851        50         9         0          0

Packets Dropped In Processing by CPU event

Event             Total                5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Sa Miss                      641599177        50         9         0          0

L3 Receive                           2         0         0         0          0

Input Acl Fwd                     4669         0         0         0          0

Sw Packet for Bridge                    3         0         0         0          0

Thanks & Regards,
Karthick Murugan
CCIE#39285

Thanks & Regards, Karthick Murugan CCIE#39285

Hi,

Spanning tree flap is happending for all the vlan on the time on the backbone switch & we try to find out the port but not yet successfull yet.

How to stop the exhaustion of the vty terminal when we are running the script.

Also we are getting vty 11-15 are showing 0.0 & occupied we are not be able to clear the same with disconnect/kill session

br/subhojit

You can use "show tcp brief" to identify the tcp connections with a TCB value associated to each.

Try to clear the session using the command "clear tcp tcb " to clear the lines.

High CPU was due to Spanning-tree state changes/

Thanks & Regards,
Karthick Murugan
CCIE#39285

**DO NOT FORGET TO RATE ALL USEFUL POSTS**

Thanks & Regards, Karthick Murugan CCIE#39285

HI,

Is there any way/command is there to check from where the stp changes are originating.

As the issue persiti on 2-3 sec, its very difficult to get the capture

also, pls conform, in the above eem script, each line is taking 1 vty session & all the above script is taking single vty

Br/Subhojit

HI,

Thanks for the details

Got the reply in your another post

!!!!!!

Thanks

Hi

It looks the EEM has an issue.

action 1.15 cli command "conf t"

action 1.16 cli command "no event"--> this is not removing the configure event manager applet change it to "event manager applet high_cpu"

It looks the CPU utilization on the switch was more than 65% for sometime the EEM tried to run the script more than once using different vty lines(as we have not removed the applet after first execution)

Thanks & Regards,
Karthick Murugan
CCIE#39285

Thanks & Regards, Karthick Murugan CCIE#39285

Hi,

Regarding the hung ssh session clear, I am not be able to find the tcb value pertaing to hung session in vty

Pls suggest any further command to remove the sessions.

Switch#sh user

  11 vty 10               SSH Process          00:00:00  
  12 vty 11               SSH Process          00:00:00  
  13 vty 12               SSH Process          00:00:00  
  14 vty 13               SSH Process          00:00:00  


Switch#sh tcp brief
TCB       Local Address           Foreign Address        (state)
13F75A7C  .22        .49016    ESTAB

Br/Subhojit

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: