Solved: EEM QoS event

Florin Barhala · ‎04-07-2016

Hi guys,

I am couple weeks old now with an attempt for an EEM script to catch a QoS event. I have setup a basic interface policer on the Internet line:

int_r2#show policy-map
Policy Map BW_check
Class class-default
police cir 96000000 bc 3000000
conform-action transmit
exceed-action drop

int_r2#show run int gi0/0
Building configuration...

Current configuration : 242 bytes
!
interface GigabitEthernet0/0
ip flow ingress
duplex auto
speed auto
service-policy input BW_check
service-policy output BW_check
end

And here's the EEM script:

event manager applet QoS-Syslog
event snmp oid 1.3.6.1.4.1.9.9.166.1.17.1.1.9.32.736467 get-type exact entry-op ge entry-val "1000" entry-type rate average-factor 1 poll-interval 60
action 1.0 syslog msg "Bandwidth Exceeded within a 1 minute period"

Now I did check the SNMP OID with snmpwalk several times and seems it's the right one, still I can't make really catch the event so I can see the syslog message (trigger the action).

Any thoughts here?

Thanks,

Florin.

Joe Clarke · ‎04-12-2016

The "exceeded" line also has a bps measurement. If you don't use a gauge value, then you will have to add a lot of extra logic to the EEM policy to track deltas to the byte counter.

No, not if you use bytes. Once it hits 1000, it will always trigger. It's a monotonically increasing counter. The bps value is a gauge relative to time.

You need:

action 6 end

And:

event timer watchdog time 60

View solution in original post

Joe Clarke · ‎04-07-2016

I believe you have a TAC case open on this, and I have given your TAC engineer an alternative config that parses CLI output. The way you have this configured here, I would only expect the applet to trigger if the rate of increase of this counter is at our above 1000 since the last time the event occurred. It's not clear to me that is the case as I have not seen multiple queries of the object in question over a period of time.

Florin Barhala · ‎04-07-2016

Indeed a case is opened for some time now, I am probably at the third engineer.

The trouble is that exceeded packet counter increases more than 1000 at a time, but on the past three weeks syslog action was triggered only three times:

int_r2#show logging | i QoS
Mar 14 15:50:53 EET: %HA_EM-6-LOG: QoS-Syslog: Bandwidth Exceeded within a 1 minute period
Apr 3 09:47:24 EEST: %HA_EM-6-LOG: QoS-Syslog: Bandwidth Exceeded within a 1 minute period
Apr 3 09:48:24 EEST: %HA_EM-6-LOG: QoS-Syslog: Bandwidth Exceeded within a 1 minute period
int_r2#

while the real value is much higher:

int_r2#show policy-map interface gi0/0 | i exceeded
exceeded 605056 packets, 778041715 bytes; actions:
conformed 16416000 bps, exceeded 0000 bps
int_r2#

So what do you think we're missing here?

Joe Clarke · ‎04-07-2016

Since you're using the CLI to check I cannot say. But I would use the example I sent to your TAC engineer to use the CLI in EEM to trigger your event. It will at least line up to what you're using to verify so there should not be any surprises.

Florin Barhala · ‎04-11-2016

Thanks Joe! TAC didn't provide me that option until I specifically asked for. So here's your idea:

cli command "enable"

cli command "show policy-map interface gi0/0 input"

regexp "exceeded ([0-9]+) bps" "$_cli_result" match dropped if $dropped ge "1000"

syslog msg "Bandwidth Exceeded within a 1 minute period"

end

Now here's the interface's QoS output:

show policy-map interface gi0/0 input
GigabitEthernet0/0

Service-policy input: BW_check

Class-map: class-default (match-any)
2722712611 packets, 2182434740120 bytes
5 minute offered rate 52138000 bps, drop rate 0000 bps
Match: any
police:
cir 96000000 bps, bc 3000000 bytes
conformed 2722023817 packets, 2181554764144 bytes; actions:
transmit
exceeded 686391 packets, 879787523 bytes; actions:
drop
conformed 52137000 bps, exceeded 0000 bps

So based on that, here's what I added on the router:

event manager applet QoS-Syslog2
action 1 cli command "enable"
action 2 cli command "show policy-map interface gi0/0 input"
action 3 regexp "exceeded ([0-9]+) bytes" "$_cli_result" match dropped
action 4 if $dropped ge "1000"
action 5 syslog msg "Bandwidth exceeded"

And I cleared interface counters:

roma_int_r2#show policy-map interface gi0/0 input | i exceeded
exceeded 0 packets, 0 bytes; actions:
conformed 23040000 bps, exceeded 0000 bps

Anything that I missed on the script?

Joe Clarke · ‎04-11-2016

You need to match on bps, not bytes. Else once the counter hits 1000, it will always trigger. But, yes, that was my suggested change.

Florin Barhala · ‎04-12-2016

Hi Joe,

Couple questions if possible:

1. Why is it I have to match on bps rather than bytes, since exceeded line contains bytes keyword.

2. Exceeded counter is no 0; let's say it gets on 2233 and the message is triggered. Then next time it will increase from 2233 to 34000 will I get another syslog message?

3. Other than that bps/bytes keyword is there anything missing on the script? Here's what I could pick from router logs;

Apr 11 12:24:54 EEST: %HA_EM-6-FMPD_EEM_CONFIG: QoS-Syslog2: Missing "end" action in conditional block
Apr 11 12:24:54 EEST: %HA_EM-4-FMPD_NO_EVENT: No event configured for applet QoS-Syslog2

Thanks,

Florin.

Joe Clarke · ‎04-12-2016

The "exceeded" line also has a bps measurement. If you don't use a gauge value, then you will have to add a lot of extra logic to the EEM policy to track deltas to the byte counter.

No, not if you use bytes. Once it hits 1000, it will always trigger. It's a monotonically increasing counter. The bps value is a gauge relative to time.

You need:

action 6 end

And:

event timer watchdog time 60

Florin Barhala · ‎04-13-2016

I still don't get why is bps and not bytes, based on the command output:

show policy-map interface gi0/0 input
GigabitEthernet0/0

Service-policy input: BW_check

Class-map: class-default (match-any)
874284364 packets, 671907189880 bytes
5 minute offered rate 47470000 bps, drop rate 0000 bps
Match: any
police:
cir 96000000 bps, bc 3000000 bytes
conformed 874178735 packets, 671774990630 bytes; actions:
transmit
exceeded 105069 packets, 132164049 bytes; actions:
drop
conformed 47470000 bps, exceeded 0000 bps

I see only bytes after exceeded rather than bps.

While I was writing the above post I ran every 10-12s the command: show policy-map interface gi0/0 input and I immediately noticed that there was a traffic spike that populated the command right I cleared interface counters. Here's a sample of it:

11.01.10 am (my timezone):

show policy-map interface gi0/0 input
GigabitEthernet0/0

Service-policy input: BW_check

Class-map: class-default (match-any)
2561823 packets, 2163088096 bytes
5 minute offered rate 35526000 bps, drop rate 2000 bps
Match: any
police:
cir 96000000 bps, bc 3000000 bytes
conformed 2561457 packets, 2162623616 bytes; actions:
transmit
exceeded 368 packets, 464653 bytes; actions:
drop
conformed 35518000 bps, exceeded 9000 bps

11.01.22 am (my timezone):

show policy-map interface gi0/0 input
GigabitEthernet0/0

Service-policy input: BW_check

Class-map: class-default (match-any)
2561823 packets, 2163088096 bytes
5 minute offered rate 35526000 bps, drop rate 2000 bps
Match: any
police:
cir 96000000 bps, bc 3000000 bytes
conformed 2561457 packets, 2162623616 bytes; actions:
transmit
exceeded 368 packets, 464653 bytes; actions:
drop
conformed 35518000 bps, exceeded 8000 bps

and then 2 minutes later, this line was the same: exceeded 368 packets, 464653 bytes; actions:

while this line went to 0: conformed 35518000 bps, exceeded 0000 bps

So Joe, thanks again for your answer you nailed it as usual!