Solved: EEM applet not working

deshaw · ‎03-04-2012

Hi team,

Okay, so what we have observed in our network is that flow-export sometimes causes high interface utilization and as a workaround, we remove and re-add flow export commands. Now there are short burstd of traffic as well, so I want to add a condition where "burst happens 3 times in 60s, then trigger the event". As I cannot test in production router, so I am testing on a lab switch and there is not much traffic on it. txload/rxload is 1 and hence my applet looks like this:

sw1-trn-mr3.hyd.shaw.net#sh run | b event manager
event manager applet test authorization bypass
 event tag 1 interface name GigabitEthernet0/4 parameter txload entry-op ge entry-val 1 entry-type value exit-op lt exit-val 1 exit-type value poll-interval 10
 event tag 2 interface name GigabitEthernet0/4 parameter rxload entry-op ge entry-val 1 entry-type value exit-op lt exit-val 1 exit-type value poll-interval 10
 trigger occurs 3 period 60
  correlate event 1 or event 2
 action 1.1 cli command "enable"
 action 1.2 syslog msg "Interface utilization has risen again. "
 action 1.3 cli command "show proc cpu | append flash:test"
 action 1.4 cli command "show interface GigabitEthernet0/4 | append flash:test"
 action 1.6 cli command "show ip cache flow | append flash:test"
 action 1.7 cli command "show tech | append flash:test"
action 2.2 cli command "no ip flow-export source GigabitEthernet0/4"
 action 2.3 cli command "no ip flow-export destination 192.168.1.1 2055"
 action 2.5 cli command "ip flow-export source GigabitEthernet0/4"
 action 2.6 cli command "ip flow-export destination 192.168.1.1 2055"
 action 2.9 cli command "end"
!
end

The thing is it does not work. I turned on debugs and I see this.


Mar  3 15:30:46: fh_send_intf_fd_msg: msg_type=114

Mar  3 15:30:46: fh_send_intf_fd_msg: sval=0

Mar  3 15:30:46: fh_send_intf_fd_msg: msg_type=114

Mar  3 15:30:46: fh_send_intf_fd_msg: sval=0

Mar  3 15:30:54: fh_fd_intf_process_async

Mar  3 15:30:54: fh_fd_intf_param_fetch:I/F=GigabitEthernet0/4 txload=1

Mar  3 15:30:54: intf_value_uint_compare:op1=1 op2=1 ret=TRUE

Mar  3 15:30:54: intf_entry_value_check:Returning TRUE

Mar  3 15:30:54: fh_fd_intf_event_match: re = 0x5849AA0, num_matches = 1



Mar  3 15:30:54: fh_fd_intf_start_poll_timer: start_t=10000

Mar  3 15:30:54: fh_fd_intf_process_poll_timer: update_t=0

Mar  3 15:30:54: fh_fd_intf_param_fetch:I/F=GigabitEthernet0/4 rxload=1

Mar  3 15:30:54: intf_value_uint_compare:op1=1 op2=1 ret=TRUE

Mar  3 15:30:54: intf_entry_value_check:Returning TRUE

Mar  3 15:30:54: fh_fd_intf_event_match: re = 0x56D793C, num_matches = 1



Mar  3 15:30:54: fh_fd_intf_start_poll_timer: start_t=10000

Mar  3 15:30:54: fh_fd_intf_process_poll_timer: update_t=0

Mar  3 15:30:54: fh_send_intf_fd_msg: msg_type=64

Mar  3 15:30:54: fh_send_intf_fd_msg: sval=63

Mar  3 15:30:54: fh_send_intf_fd_msg: msg_type=64

Mar  3 15:30:54: fh_send_intf_fd_msg: sval=0

Mar  3 15:31:04: fh_fd_intf_process_async

Mar  3 15:31:04: fh_fd_intf_param_fetch:I/F=GigabitEthernet0/4 txload=1

Mar  3 15:31:04: intf_value_uint_compare:op1=1 op2=1 ret=FALSE

Mar  3 15:31:04: intf_exit_value_check: re=0x5849AA0, returning=FALSE

Mar  3 15:31:04: intf_exit_comb_check:Returning FALSE

Mar  3 15:31:04: fh_fd_intf_event_match: re = 0x5849AA0, num_matches = 0



Mar  3 15:31:04: fh_fd_intf_start_poll_timer: start_t=10000

Mar  3 15:31:04: fh_fd_intf_process_poll_timer: update_t=0

Mar  3 15:31:04: fh_fd_intf_param_fetch:I/F=GigabitEthernet0/4 rxload=1

Mar  3 15:31:04: intf_value_uint_compare:op1=1 op2=1 ret=FALSE

Mar  3 15:31:04: intf_exit_value_check: re=0x56D793C, returning=FALSE

Mar  3 15:31:04: intf_exit_comb_check:Returning FALSE

Mar  3 15:31:04: fh_fd_intf_event_match: re = 0x56D793C, num_matches = 0

Only the first time, it returns TRUE. After that, it returns FALSE always and hence the condition "trigger occurs 3 period 60" does not match and hence no event is triggered. Where am I going wrong?
Appreciate your help on this.

Thanks.

Joe Clarke · ‎03-04-2012

You have an exit condition of "lt 1". That means that the load must drop below 1 before the event will rearm. In this case, it looks like the load is always 1, so you will not see the three events happen. You will need to push some traffic through that interface to complete your test.

Also, you're missing:

action 2.1 cli command "config t"

And likely you'll need to extend the first event with a maxrun parameter as I'm sure this policy will take more than 20 seconds to run.

View solution in original post

Joe Clarke · ‎03-04-2012

In this case, your design will not work. You don't want to use the exit condition. You want the policy to trigger on each occurrence of the event. That alone should be enough given that your window is 30 seconds. That would mean that each 10 seconds (when the policy checks the interface stats) the tx or rx load was above your threshold.

View solution in original post

Joe Clarke · ‎03-05-2012

That would have been the case if you didn't have your trigger occurs and period commands. If you do:

trigger occurs 30 period 300

That should only fire the actions when you've seen a consistent period of high utilization. Note: you will still need to remove the exit criteria.

View solution in original post

Joe Clarke · ‎03-04-2012

You have an exit condition of "lt 1". That means that the load must drop below 1 before the event will rearm. In this case, it looks like the load is always 1, so you will not see the three events happen. You will need to push some traffic through that interface to complete your test.

Also, you're missing:

action 2.1 cli command "config t"

And likely you'll need to extend the first event with a maxrun parameter as I'm sure this policy will take more than 20 seconds to run.

deshaw · ‎03-04-2012

Hi Joseph,

Thanks. Your suggestions make sense. I will try them and will let you know.

In the meantime, I have a question. How do I make sure that the actions are taken only after the condition persists for x time interval?

For example, in our production environment, if the interface txload is higher than > 200 for 10 minutes CONTINUOUSLY, take the EEM applet actions, otherwise don't do anything . The idea is to avoid situations where txload increases due to sudden bursts of traffic.

Thanks.

Joe Clarke · ‎03-04-2012

In this case, your design will not work. You don't want to use the exit condition. You want the policy to trigger on each occurrence of the event. That alone should be enough given that your window is 30 seconds. That would mean that each 10 seconds (when the policy checks the interface stats) the tx or rx load was above your threshold.

deshaw · ‎03-05-2012

Hi Joseph,

Thanks for the reply.

Nopes, I don't want the policy ( and hence the corresponding actions) to trigger on each occurrence of the event.

Suppose my configuration is:

event manager applet test authorization bypass

event tag 1 interface name FastEthernet0/10 parameter txload entry-op ge entry-val 10 entry-type value exit-op lt exit-val 10 exit-type value poll-interval 10

event tag 2 interface name FastEthernet0/10 parameter rxload entry-op ge entry-val 2 entry-type value exit-op le exit-val 1 exit-type value poll-interval 10

trigger

correlate event 1 or event 2

action 1.1 cli command "enable"

action 1.2 syslog msg "Interface utilization has risen again. "

action 1.3 cli command "show proc cpu | append flash:test"

action 1.4 cli command "show interface FastEthernet0/10 | append flash:test"

!

Now suppose at 00:00, txload is 11, event is true and actions are executed.

at 00:10, txload is 9,exit happens...

In the above case, actions are executed for a short burst of traffic.

My aim is NOT to execute the actions for short bursts of traffic.

What I want is, if the txload has been consistently high for 5 min, then execute the actions:

At 00:00, txload=12

At 00:10, txload=12

At 00:20, txload =13

..............................

At 05:00, txload=12,

So in the above case, for continuous 30 polls during 5-min interval, not even once has the txload been less than 12..Such types of events I want to capture and in such cases ONLY, the actions should be executed.

As per your suggestion, if I remove the exit condition altogether, then if the txload is consistently high for 5 min interval, then the actions will be executed thirty times!!! I want the actions to be executed only once and only when the utilization has been high for continuous 5 min.

Main goal is to avoid the actions when there are short bursts of traffic.

Let me know if anything stated is not clear.

Thanks,

Ritika

Joe Clarke · ‎03-05-2012

That would have been the case if you didn't have your trigger occurs and period commands. If you do:

trigger occurs 30 period 300

That should only fire the actions when you've seen a consistent period of high utilization. Note: you will still need to remove the exit criteria.

deshaw · ‎03-06-2012

Hi Joseph,

Thanks for the suggestion. That works.

So based on your suggestion, actions will be taken if the event is true for continuous 5 minutes.

Now if the txload remains high in between 00:05:00 and 00:10:00, actions will be taken again, right?

Is there a way where we can defer the action execution further? For example, after taking the actions after 5 min, take further actions after 10 minutes and only if the utilization has been high continuously for those 10 min.

So, first time actions are taken at 00:05:00.

Now keep checking the txload , but take same actions again after 10 minutes, ie at 00:15:00 ( and txload should be > 10 consistently during those 10 minutes).

Thanks.

Joe Clarke · ‎03-06-2012

Not with one policy. At this point, you would need to set an EEM counter each time the event occurs, then create another policy to monitor that counter. When the counter hits a specific value, then you can execute the actions. Then you can have a third policy that checks to see if rx and tx load fall below the threshold and reset the counter.

You may need to do some programmatic logic in your initial policy not to run the actions if the counter is above 5. This will prevent it from firing each time after it exceeds five minutes.