EEM to check Process

Unanswered Question
Dec 31st, 2009
User Badges:

HI


How can i use EEM to capture 1st 5 process when cpu usage goes above 90 % (5 min avg).

I want to store output in a flash.


I tried generating trap when usage goes above 90% in 5 min and prepared applet based on output which i got.

biut looks like i am doing mistake in regular expression...


Is there any other / better way also to achive same?


THanks

Lokesh

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Joe Clarke Thu, 12/31/2009 - 13:51
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Post the policy you wrote.  What version of IOS are you running?

Lokesh.Khanna Thu, 12/31/2009 - 14:31
User Badges:

hi


snmp-server enable traps cpu threshold
process cpu threshold type total rising 90 interval 3


event manager applet CPU
event syslog pattern "CPURISINGTHRESHOLD"
action 1.0 cli command "enable"
action 1.1 cli command "show process cpu sorted 5min | include ^_[1-9]|^| append flash:cpu_info"


i am trying it on GNS3 - IOS is  12.4(25b) - 3600 series router. Not sure if this is not supposed to work on GNS3.

but i m sure i am doing mistake in my regular expresion since above command gives me everything instead of 1st 5 lines.

Lokesh.Khanna Thu, 12/31/2009 - 22:24
User Badges:

HI


I tried some thing else which partially worked


event manager applet CPU1
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.8.1 get-type exact entry-op ge entry-val "10" poll-interval 30
actiona 1.0 cli command "enable"
action 2.0 cli command "show process cpu | include ^__[1-9]|^__5|"
action 3.0 syslog priority warning msg "high cpu usage"


But this gives me all output of show process cpu, and i want only 1st few lines.

How do i achive this?

Joe Clarke Fri, 01/01/2010 - 09:12
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

What you want to do cannot be done using applets (at least not in your version of code).  You need to use a Tcl policy.  Attached is a policy which should do what you want.  The resulting flash:cpu_info file will contain the current date and time, the two "show proc cpu" header lines, and the first five processes.

Lokesh.Khanna Fri, 01/01/2010 - 10:16
User Badges:

Thanks Joe


Honestly i am finding it difficult since this is a new topic for me..


is there any easy way.

You mentioned this can't be done using regexp in my version of IOS... can this be done in other IOS, if yes then how? And which IOS do i need?

Joe Clarke Fri, 01/01/2010 - 10:22
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Getting the first five lines can be done easily using applets in IOS 12.4(22)T and higher.  However, writing that data to a file on flash is where it gets difficult.  You could easily print the lines, email them, or send them in a syslog message, but to write them out to a file, you need Tcl.  The reason for this is that IOS does not support the double pipe syntax.  That is, the following command line is invalid:


show proc cpu | exc 0.00 | append flash:/file


The second pipe is not interpreted the way you think it should be.

Lokesh.Khanna Fri, 01/01/2010 - 10:25
User Badges:

Thanks


Can you give me Sample config for sending mail and sending to Syslog for 1st 5 lines.


Regards

Lokesh

Joe Clarke Fri, 01/01/2010 - 10:35
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

event manager applet dump-procs

event syslog pattern "CPURISINGTHRESHOLD"

action 001 cli command "enable"

action 002 cli command "show proc cpu sorted 5min"

action 003 set lines 0

action 004 foreach line "$_cli_result" "\n"

action 005   if $lines gt 6

action 006     break

action 007   end

action 008   append output $line

action 009   increment lines

action 010 end

action 011 mail to [email protected] from [email protected] server 10.1.1.1 subject "Top five processes" body "$output"

action 012 syslog msg "Top five processes: $output"

Lokesh.Khanna Fri, 01/01/2010 - 10:45
User Badges:

Thanks Joe


What does increment lines do here?


And Which IOS must i have to run this?

Regards

Lokesh

Joe Clarke Fri, 01/01/2010 - 11:45
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

The "increment" command increments the value of the lines counter variable by one.  The idea is that after the first two header lines, and first five processes are captured, the loop should be exited.


This applet requires 12.4(22)T or higher (i.e. EEM 3.0 or higher).  You currently have EEM 2.1.

rockerptit Thu, 08/26/2010 - 05:18
User Badges:

Hi,


I tried your solution in both IOS 12.4T-22 & IOS 15.0 in C7206VXR platform, it doesn't work!

Joe Clarke Thu, 08/26/2010 - 10:10
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

You may need to add:


event manager session cli username USER


If you are using AAA command authorization.  If that does not work, start a new thread for your specific problem with the output of "debug event manager action cli".

ahmed.gadi Sun, 06/19/2011 - 17:14
User Badges:

Hi Joseph,


vent manager applet dump-procs
event syslog pattern "CPURISINGTHRESHOLD"
action 001 cli command "enable"
action 002 cli command "show proc cpu sorted 5min"
action 003 set lines 0
action 004 foreach line "$_cli_result" "\n"
action 005   if $lines gt 6
action 006     break
action 007   end
action 008   append output $line
action 009   increment lines
action 010 end
action 011 mail to [email protected] from [email protected] server 10.1.1.1 subject "Top five processes" body "$output"
action 012 syslog msg "Top five processes: $output"

Instead of using the programming logic for sending 10 lines via email, can i do some thing like  this


action 003 cli command "terminal width 100"<--since it has longer width for some process and to accomodate it in 1 line

action 004 cli command "terminal len 13" <-- it will count --more-- and 2 line after command

action 005  cli command "sh process cpu sorted 5min"


and then mail configuration.


Regards

Ahmed...

Joe Clarke Sun, 06/19/2011 - 19:17
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

No, this will not work.  The terminal used by EEM is specially designed to to disable the pager.  Plus, you need to be able to find the device prompt when the command finishes executing.  If you want to add more lines, just adjust the "$lines gt 6" piece.  Specify a line count of 10 or higher.  If you cannot use programmatic applets, then you will need to switch to Tcl to control the output.

itsnavee4 Tue, 04/17/2012 - 17:59
User Badges:

the applet prints 20 lines, not 5, any idea how to fix it?

Joe Clarke Tue, 04/17/2012 - 22:09
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

I'm not not sure what applet you're trying, but this one will print five lines of processes:


event manager applet dump-procs<br/> event syslog pattern "CPURISINGTHRESHOLD"<br/> action 001 cli command "enable"<br/> action 002 cli command "show proc cpu sorted 5min"<br/> action 003 set lines 0<br/> action 004 foreach line "$_cli_result" "\n"<br/> action 005   if $lines gt 6<br/> action 006     break<br/> action 007   end<br/> action 008   append output $line<br/> action 009   increment lines<br/> action 010 end<br/> action 011 mail to [email protected] from [email protected] server 10.1.1.1 subject "Top five processes" body "$output"<br/> action 012 syslog msg "Top five processes: $output"
krzysiek.zalewski Thu, 08/18/2011 - 03:39
User Badges:

Hi Joseph

Could you help me with this problem


- Data

> IOS 12.4(15)T13


- EEM config

CONFIG-SET

+---------------------------------------------------------------------------------------

event manager applet TEST trap

event syslog pattern "%SYS-5-CONFIG_I:"

action 1.0 cli command "enable"

action 2.0 cli command "sh run"

action 3.0 mail server "192.168.1.146" to "[email protected]" from "[email protected]" subject "B25 PBX Alert" body "$_cli_result"

+---------------------------------------------------------------------------------------


- Debug output

CONFIG-SET

+---------------------------------------------------------------------------------------

Rack1R1#debug event manager all

All possible Embedded Event Manager debugging has been turned on

Rack1R1#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

Rack1R1(config)#^Z

Rack1R1#

*Mar  1 23:51:34.346: cli_history_entry_add: free_hist_list size=0, hist_list size=7

*Mar  1 23:51:34.350: check_eem_cli_policy_handler: command_string=configure terminal

*Mar  1 23:51:34.350: check_eem_cli_policy_handler: num_matches = 0, response_code = 1

*Mar  1 23:51:35.354: %SYS-5-CONFIG_I: Configured from console by console

*Mar  1 23:51:35.362: fh_fd_syslog_event_match: num_matches = 1

*Mar  1 23:51:35.362: fh_fd_data_syslog: num_matches = 1

*Mar  1 23:51:35.366: fh_send_server_sig_hndlr: received a pulse from Syslog Event Detector on node0/0 with fdid: 2

*Mar  1 23:51:35.370: fh_send_syslog_fd_msg: msg_type=62

*Mar  1 23:51:35.370: fh_send_syslog_fd_msg: sval=0

*Mar  1 23:51:35.370: fh_send_server_sig_hndlr: received FH_MSG_EVENT_PUBLISH

*Mar  1 23:51:35.374: fh_schedule_callback: fh_schedule_callback: cc=66B81EDC prev_epc=0; epc=676B600C

*Mar  1 23:51:35.390: fh_schedule_callback: EEM callback policy TEST has been scheduled to run

Rack1R1#

*Mar  1 23:51:35.406: fh_io_msg: received FH_MSG_API_INIT; jobid=14, processid=230, client=4, job name=EEM Callback Thread

*Mar  1 23:51:35.406: fh_server: fh_io_msg: received msg FH_MSG_EVENT_REQINFO from client 4 pclient 1

*Mar  1 23:51:35.422: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : CTL : cli_open called.

*Mar  1 23:51:35.430: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : Rack1R1>

*Mar  1 23:51:35.430: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : IN  : Rack1R1>enable

*Mar  1 23:51:35.430: cli_history_entry_add: free_hist_list size=0, hist_list size=7

*Mar  1 23:51:35.430: eem_no_scan flag set, skipping scan of command_string=check_eem_cli_policy_handler

Rack1R1#

*Mar  1 23:51:35.510: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : Rack1R1#

*Mar  1 23:51:35.510: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : IN  : Rack1R1#sh run

*Mar  1 23:51:35.514: cli_history_entry_add: free_hist_list size=0, hist_list size=7

*Mar  1 23:51:35.514: eem_no_scan flag set, skipping scan of command_string=check_eem_cli_policy_handler

Rack1R1#

*Mar  1 23:51:40.226: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : Building configuration...

*Mar  1 23:51:40.226: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :

*Mar  1 23:51:40.226: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : Current configuration : 2710 bytes

*Mar  1 23:51:40.226: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : !

*Mar  1 23:51:40.230: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : version 12.4

*Mar  1 23:51:40.230: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : service timestamps debug datetime msec

*Mar  1 23:51:40.230: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : service timestamps log datetime msec

*Mar  1 23:51:40.230: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : no service password-encryption

*Mar  1 23:51:40.230: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : !

*Mar  1 23:51:40.230: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : hostname Rack1R1

*Mar  1 23:51:40.230: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : !

*Mar  1 23:51:40.230: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : boot-start-marker

*Mar  1 23:51:40.230: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : boot-end-marker

*Mar  1 23:51:40.230: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : !

*Mar  1 23:51:40.230: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : enable password cisco

*Mar  1 23:51:40.234: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : !

*Mar  1 23:51:40.234: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : no aaa new-model

*Mar  1 23:51:40.234: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : memory-size iomem 5

*Mar  1 23:51:40.234: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : ip cef

*Mar  1 23:51:40.234: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : !

*Mar  1 23:51:40.234: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : CTL : 20+ lines read from cli, debug output truncated

*Mar  1 23:51:40.234: %HA_EM-3-FMPD_UNKNOWN_ENV: fh_parse_var: could not find environment variable: _cli_r

*Mar  1 23:51:40.234: %HA_EM-3-FMPD_ERROR: Error executing applet TEST statement 3.0

*Mar  1 23:51:40.238: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : CTL : cli_close called.

*Mar  1 23:51:40.242: fh_server: fh_io_msg: received msg FH_MSG_CALLBACK_DONE from client 4 pclient 1

*Mar  1 23:51:40.242: fh_io_msg: EEM callback policy TEST has ended with abnormal exit status of 0xFFFFFFFF

*Mar  1 23:51:40.242: fh_schedule_callback: fh_schedule_callback: cc=66B81EDC prev_epc=676B600C; epc=0

*Mar  1 23:51:40.242: fh_schedule_policy: prev_epc=0x00000000; epc=0x00000000

*Mar  1 23:51:40.250: fh_server: fh_io_msg: received msg FH_MSG_API_CLOSE from client 4 pclient 1

*Mar  1 23:51:40.254: fh_io_msg: received FH_MSG_API_CLOSE client=4

+---------------------------------------------------------------------------------------


- Problem

> How generate more than 20 lines output?

*Mar  1 23:51:40.234: %HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : CTL : 20+ lines read from cli, debug output truncated

Joe Clarke Thu, 08/18/2011 - 13:13
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Please start a new thread for your question.

Joe Clarke Sat, 11/20/2010 - 09:32
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

This script should do what you want:


::cisco::eem::event_register_syslog pattern "CPURISINGTHRESHOLD"

namespace import ::cisco::eem::*
namespace import ::cisco::lib::*

if [catch {cli_open} result] {
    error $result $errorInfo
} else {
    array set cli1 $result
}

if [catch {cli_exec $cli1(fd) "enable"} _cli_result] {
    error $_cli_result $errorInfo
}

if [catch {cli_exec $cli1(fd) "show proc cpu sorted 5min"} _cli_result] {
    error $_cli_result $errorInfo
}

set lines 0
set _ts "$_cli_result"
while {$_ts != ""} {
    if {[regexp -indices "\n" $_ts _loc] == 0} {
        set line $_ts
        set _ts ""
    } else {
        set _mstart [lindex $_loc 0]
        set _mend [lindex $_loc 1]
        if {$_mstart == 0} {
            set line ""
        } else {
            set line [string range $_ts 0 [expr $_mstart - 1]]
        }
        set _ts [string range $_ts [expr $_mend + 1] end]
    }

    if {$lines > 25} {
        break
    }
    append output $line
    incr lines
}

set mail_pre "Mailservername: $_email_server\n"
append mail_pre "From: $_email_from\n"
append mail_pre "To: $_email_to\n"
append mail_pre
append mail_pre "Subject: CPU Alert\n\n"
append mail_pre "$output\n\n"
set mail_msg [uplevel #0 [list subst -nobackslashes -nocommands $mail_pre]]
if [catch {smtp_send_email $mail_msg} result] {
    error $result $errorInfo
}

# Close open cli before exit.
if [catch {cli_close $cli1(fd) $cli1(tty_id)} result] {
    error $result $errorInfo
}


Before registering this policy, you will need to configure the following EEM environment variables in "config t" mode:


event manager environment _email_server SERVER

event manager environment _email_from [email protected]

event manager environment _email_to [email protected]


Where SERVER is your SMTP server IP address.


You will also need to configure your CPU rising threshold accordingly:


process cpu threshold type total rising 80 ...

Actions

This Discussion