EEM - Applet Help Appreciated

Unanswered Question
Mar 28th, 2010
User Badges:

I am using the following applet to modify entries in two ACLs from a mobile device. Please see applet below with output from a test run:


3    applet    user    none                Off   Sun Mar 28 22:42:59 2010  rite1

policyname {rite1} sync {yes}

maxrun 60.000

action 1.0 puts "Set HostA:"

action 1.1 gets hosta

action 1.2 puts "Set HostB:"

action 1.3 gets hostb

action 1.4 cli command "enable"

action 1.5 cli command "config t"

action 1.6 cli command "ip access-list extended rite1_incoming"

action 1.7 cli command "no 10"

action 1.8 cli command "no 20"

action 1.9 cli command "10 permit ip host $hosta host $hostb"

action 2.0 cli command "20 permit ip host $hostb host $hosta"

action 2.1 cli command "ip access-list extended rite1_outgoing"

action 2.2 cli command "no 10"

action 2.3 cli command "no 20"

action 2.4 cli command "10 permit ip host $hosta host $hostb"

action 2.5 cli command "20 permit ip host $hostb host $hosta"

action 2.6 cli command "end"

action 2.7 cli command "show ip access-list | s rite1"

action 2.8 puts "$_cli_result"

action 2.9 puts ""

action 3.1 puts "-------------------------------------"

action 3.2 puts ""

action 3.3 puts "Use following packet_capture commands:"

action 3.4 puts " ritesh - show packet_capture"

action 3.5 puts " ritecl - clear packet_capture"

action 3.6 puts " ritestr - start packet_capture"

action 3.7 puts " ritestp - stop packet_capture"

action 3.8 puts ""

action 3.9 puts "-------------------------------------"


alias exec rite1 event manager run rite1


Router#rite1

Set HostA:

172.20.1.1

Set HostB:

20.1.1.1

Extended IP access list rite1_incoming

    10 permit ip host 172.20.1.1 host 20.1.1.1

    20 permit ip host 20.1.1.1 host 172.20.1.1

Extended IP access list rite1_outgoing

    10 permit ip host 172.20.1.1 host 20.1.1.1

    20 permit ip host 20.1.1.1 host 172.20.1.1


-------------------------------------


Use following packet_capture commands:

ritesh - show packet_capture

ritecl - clear packet_capture

ritestr - start packet_capture

ritestp - stop packet_capture


-------------------------------------



I use this for packet capturing with RITE on remote sites that do not warrant consistent samples. We get alerted when a specific top-talker is showing unusual high volumes of traffic and our techs are able to run the following script on the fly from mobile devices. The high-volume converstaion gets captured and we have another script that exports those onboard captures to centralized servers. The whole operation runs pretty great. The problem is that for some reason it eats a TON of process usage.



117852: Mar 28 23:03:54.197 DST: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 72%/1%, Top 3 processes(Pid/Util):  199/71%, 49/0%, 81/0%

117865: Mar 28 23:04:09.128 DST: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 1%/0%.


I"m assuming that process # 199 is dynamically created for this specific applet when it runs, as it does not seem to run on the routers at other times:


Router#show process cpu

....output omitted....

198           0           3          0  0.00%  0.00%  0.00%   0 EEM ED RPC

201       14371       99291        144  0.00%  0.00%  0.00%   0 DHCPD Receive

....output omitted....


For some reason, while the router is interacting with the user, waiting for input from the "action x.x get" commands, the processer is through the roof. Any ideas on how I could run this similar operation without taking up so much process overhead? I want to say that I could probably limit the resource usage of those specific processes maybe with ERM, but I'm currently not very familiar with ERM. Thanks for everyone's help.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Joe Clarke Mon, 03/29/2010 - 21:44
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

There appears to be a bug in the gets action which causes a CPU spike.  This does not happen if you use Tcl, though.  I converted your applet to a Tcl policy.  You should be able to see more reasonable gets behavior.

the_ios_inquisition Sun, 04/11/2010 - 19:22
User Badges:

Joe,


Thanks so much for converting our applet into a tcl script. It works great everywhere that we have deployed it. We did run into one issue deploying on a 3745 Router with 12.4(15)T9. When the policy was triggered, we received the following error:


error reading "stdin": Unknown error 134217728

    while executing

"gets stdin"

    invoked from within

"$slave eval $Contents"

    (procedure "eval_script" line 7)

    invoked from within

"eval_script slave $scriptname"

    invoked from within

"if {$security_level == 1} {       #untrusted script

     interp create -safe slave

     interp share {} stdin slave

     interp share {} stdout slave

..."

    (file "tmpsys:/lib/tcl/base.tcl" line 50)

Tcl policy execute failed: error reading "stdin": Unknown error 134217728

Tcl policy execute failed: error reading "stdin": Unknown error 134217728


I also pulled the specific script that it's referencing the error from:


# ----------------------------------

# base.tcl - Safe base script forinvoking safe-tcl on FM policies.

#

# August 2002, Jason Pfeifer

#

# Copyright (c) 2002, 2004-2006 bycisco Systems, Inc.

# All rights reserved.

# ----------------------------------

#


# Current security levels:

#

# 0 - trusted script, safe mode not enabled

# 1 - untrusted script, safe modeenabled


####################   Procedures  ##################


proc TraceVariable { fromInterp fromVartoInterp toVar } {

    set value [ interp eval $fromInterpset ::$fromVar]

    interp eval $toInterp [list set::$toVar $value]

}


proc eval_script { slave scriptname } {

    # Open script and evaluate it in slave

    set F [open $scriptname]

    set Contents [read $F]

    close $F


    $slave eval $Contents

}


####################  End of Procedures  ##################



# main


# Process argv input:

set scriptname [lindex $argv 0]

set security_level [lindex $argv 1]

if {$security_level != 0 && $security_level != 1} {

    set security_level 1

}



#Set Security level specific parameters


set errorInfo ""

set FM_TCLLIBDIR [info library]


if {$security_level == 1} {      #untrusted script

     interp create -safe slave

     interp share {} stdin slave

     interp share {} stdout slave

     interp share {} stderr slave


     lappend global_vars auto_path

     lappend global_vars auto_oldpath

     lappend global_vars tcl_library


     foreach entry [info vars] {

         if [array exists $entry] {

             continue

         }

         if {[lsearch -exact$global_vars $entry] >= 0} {

             continue;

         }

         TraceVariable {} $entry slave$entry

     }


     # uncomment to debug safe.tcl

     #::safe::setLogCmd puts stderr

     ::safe::interpInit slave

     # Hidden commands that are exposed to slave to ease restrictions.

     interp alias slave exit {} exit

     interp expose slave fconfigure

     interp expose slave socket

     eval_script slave $scriptname


     close stdin

     close stdout

     close stderr


     ::safe::interpDelete slave


} else {                          #trusted script

     interp create slave

     interp share {} stdin slave

     interp share {} stdout slave

     interp share {} stderr slave


     foreach entry [info vars] {

         if [array exists $entry] {

             continue

         }

         TraceVariable {} $entry slave$entry

     }


     if {[catch {::interp eval slave\

             {source [file join$FM_TCLLIBDIR init.tcl]}} msg]} {

         error "can't sourceinit.tcl into slave ($msg)"

     }


     eval_script slave $scriptname

     interp delete slave

}


What ended up happening is the script stayed stuck in pending(no 'scheduler' commands in this early version of eem) and we eventually had to reboot the router(we waited until a regularly scheduled maintenance window, so it worked out fine). Since receiving this error, we haven't tried to run this particular script on this particular router. All other devices that we have deployed this on, including other 3745's running 12.4(15)T, have worked great. If anyone has some idea as to exactly what happened and what adjustments we need to perform to correct the problem, we would greatly appreciate it.


Thanks again for everyone's help.

Joe Clarke Sun, 04/11/2010 - 21:49
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

There is typically some other error related to this (e.g. Process forced exit).  I have a 3745 running 12.4(15)T12, and it works just fine.  I will say that there were quite a few EEM bugs fixed around this version, and if possible, an upgrade might be helpful.  Yes, if a script gets stuck in EEM prior to 2.4 (and 12.4(15)T has 2.3), then a reload will be required to free the queue.


--


Please support CSC Helps Haiti


https://supportforums.cisco.com/docs/DOC-8895

https://supportforums.cisco.com

the_ios_inquisition Sun, 04/11/2010 - 22:14
User Badges:

Joe,


Thanks for such a quick response. I did upgrade this device to 12.4(15)T13 today, and we have had the script in question running on a few other 3745's without issue. Aside from CPU-hog protection, are there any other preventions that we can take when running a fair amount of scripts on a device that cannot upgrade past 12.4(15)T?


Thanks again for all of your help.

Joe Clarke Sun, 04/11/2010 - 22:46
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

You can increase the number of script threads assigned to the default scheduler class as a way to accommodate multiple parallel scripts (and potentially work around stuck threads).  However, the only way to clear such a thread will be with a reload.  To increase the number of threads, configure:


event manager scheduler script thread class default number X


Where X is a number between 1 (the default) and 65535.


--


Please support CSC Helps Haiti


https://supportforums.cisco.com/docs/DOC-8895

https://supportforums.cisco.com

Actions

This Discussion