11-12-2013 05:22 AM - edited 03-07-2019 04:33 PM
Hi,
We are observing high cpu of one of the backbone switch,
We enable eem script & below are the cpture
But we are not able to pin-point the issue.
event manager applet high_cpu
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op gt entry-val "65" poll-interval 5 action 1.01 syslog msg "------HIGH CPU DETECTED----, CPU: $_snmp_oid_val %"
action 1.02 cli command "enable"
action 1.03 cli command "term len 0"
action 1.04 cli command "debug platform packet all receive buffer"
action 1.05 cli command "show platform health | redirect slot0:high_cpu1"
action 1.06 cli command "show proc cpu sort | redirect slot0:high_cpu2"
action 1.07 cli command "show platform cpu packet statistics | redirect slot0:high_cpu3"
action 1.08 cli command "show platform cpu packet buffered | redirect slot0:high_cpu4"
action 1.09 cli command "show platform health | redirect slot0:high_cpu5"
action 1.10 cli command "show proc cpu sort | redirect slot0:high_cpu6"
action 1.11 cli command "show platform cpu packet statistics | redirect slot0:high_cpu7"
action 1.12 cli command "show platform cpu packet buffered | redirect slot0:high_cpu8"
action 1.13 cli command "show clock | redirect slot0:high_cpu9"
action 1.14 cli command "undebug all"
action 1.15 cli command "conf t"
action 1.16 cli command "no event
Switch version: cat4500e-entservicesk9-mz.122-50.SG3.bin
Model: WS-C4900M 3 slot switch
Pls suggest
Br/Subhojit
11-12-2013 07:07 AM
Hi Subhojit,
I looks the EEM couldn't complete the task because of VTY session exhaustion. However, with the provided outupt I could see two misbehaving process.
K5QosDhmMan Rate DBL 2.00 3.95 7 8 100 500 4 4 3 24328:41
K5L3Unicast Adj Tabl 2.00 7.91 15 8 100 500 22 4 0 1453:32
Could you please provide the output of "show platform cpu packet statistics"?
Thanks & Regards,
Karthick Murugan
CCIE#39285
11-12-2013 09:39 AM
HI,
I will share the cpature by tomorrow
But, Pls confirm is there any error on the script
Why the vty session exhausted
What does the misbehaving process means exactly
Br/Subhojit
11-12-2013 09:50 AM
HI,
I guess for every redirect command uses a vty session and thats why we might see this issue. I had a look at the logs again.
I see lots of SA Miss which is nothing but increase Source Address Miss. Have you verified if there are any topology changes(show spanning-tree detail | in ieee|exec|from|occurred)
?
switch##more slot0:high_cpu2 3
RkiosSysPacketMan:
Total 5 sec avg 1 min avg 5 min avg 1 hour avg
-------------------- --------- --------- --------- ----------
641603851 50 9 0 0
Packets Dropped In Processing by CPU event
Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Sa Miss 641599177 50 9 0 0
L3 Receive 2 0 0 0 0
Input Acl Fwd 4669 0 0 0 0
Sw Packet for Bridge 3 0 0 0 0
Thanks & Regards,
Karthick Murugan
CCIE#39285
11-12-2013 09:54 AM
Hi,
Spanning tree flap is happending for all the vlan on the time on the backbone switch & we try to find out the port but not yet successfull yet.
How to stop the exhaustion of the vty terminal when we are running the script.
Also we are getting vty 11-15 are showing 0.0 & occupied we are not be able to clear the same with disconnect/kill session
br/subhojit
11-12-2013 10:01 AM
You can use "show tcp brief" to identify the tcp connections with a TCB value associated to each.
Try to clear the session using the command "clear tcp tcb
High CPU was due to Spanning-tree state changes/
Thanks & Regards,
Karthick Murugan
CCIE#39285
**DO NOT FORGET TO RATE ALL USEFUL POSTS**
11-12-2013 10:06 AM
HI,
Is there any way/command is there to check from where the stp changes are originating.
As the issue persiti on 2-3 sec, its very difficult to get the capture
also, pls conform, in the above eem script, each line is taking 1 vty session & all the above script is taking single vty
Br/Subhojit
11-12-2013 10:07 AM
HI,
Thanks for the details
Got the reply in your another post
!!!!!!
Thanks
11-12-2013 10:11 AM
Hi
It looks the EEM has an issue.
action 1.15 cli command "conf t"
action 1.16 cli command "no event"--> this is not removing the configure event manager applet change it to "event manager applet high_cpu"
It looks the CPU utilization on the switch was more than 65% for sometime the EEM tried to run the script more than once using different vty lines(as we have not removed the applet after first execution)
Thanks & Regards,
Karthick Murugan
CCIE#39285
11-17-2013 08:44 PM
Hi,
Regarding the hung ssh session clear, I am not be able to find the tcb value pertaing to hung session in vty
Pls suggest any further command to remove the sessions.
Switch#sh user
11 vty 10 SSH Process 00:00:00
12 vty 11 SSH Process 00:00:00
13 vty 12 SSH Process 00:00:00
14 vty 13 SSH Process 00:00:00
Switch#sh tcp brief
TCB Local Address Foreign Address (state)
13F75A7C
Br/Subhojit
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: