Best way to change config then change it back after a while.

Unanswered Question
Jul 1st, 2013
User Badges:

I spent some time today trying a couple approaches to this, but they all seemed a little klunky.


I'm looking for the best way to change the BGP prefixes which are advertised out to an ISP, based on some check.  I want to set a timeout so the router won't attempt to send the route again for say 30 minutes after it is triggered, but then will start advertising it again, and monitor to see if the trigger condition returns.  If the trigger condition returns then again withdrawl the route for 30 minutes and so on.


I'm using a prefix-list already to limit outbound route advertisments, so it seems simplest to just make a config change to remove one line in the prefix-list, then a few minutes later put it back.


I tried just using the "cli command wait", but if I set the wait period too long, the applet seemed to die, and never ran the later cli commands to put the prefix-list line back.  There is also a exit-time clause for the event, but I couldn't figure out how to put the line back after the exit-time expired.  Lastly I tried doing an event with a watchdog timer, but also couldn't get that to work either.  Before I spend too much time working on differant options, I wanted to see if anyone had any recommendations.


I've done some TCL scripting on Cisco routers, but that seemed to be overkill for this, and I wanted to keep the config easy to manage for peers who might not be as proficient in TCL scripting.


This is intended for ASR-1002X routers if it matters.


Any suggestions would be much appreciated.


Thanks

Derek

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Joe Clarke Mon, 07/01/2013 - 20:28
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

You can use a nested applet to do this.  Just make the change as you normally would, then add actions like the following:


event manager environment q "

!

event manager applet bgp-filter

...

action 000 cli command "config t"

action 001 cli command "event manager applet restore-prefix"

action 002 cli command "event timer countdown time 1800"

action 003 cli command "action 1.0 cli command enable"

action 004 cli command "action 2.0 cli command $q config t$q"

action 005 cli command "action 3.0 cli command $q PUT COMMANDS TO RESTORE PREFIX HERE$q"

...


This nested applet will get configured, countdown the desired amount of time, then perform whatever commands are required to restore your config.  Make sense?

derek.small Tue, 07/02/2013 - 05:40
User Badges:

Thanks for the response Joe.  I'm still getting up to speed on some of the newer EEM features, so I appologize for any ignorance.  If I am reading that correctly though, won't that put that applet into the config to restore the line, after the initial condition is met once?  What will happen when the event happens again a few hours/days later?


The sequence I am trying to automate is roughly this:

1. Detect condition (in this case a DDoS attack, via one of several possible triggers).

2. Remove a line from a prefix list, causing traffic attacking a public IP to no longer be sent to the router.

3. Wait for some period for the attack to subside (say 30 minutes)

4. Re-advertise the prefix which was being attacked.

5. Return to step 1


Will what you are suggesting return the router/config to the same state it was in before the initial trigger condition was detected?


Here is what I have got to work.  This uses polling of an SNMP OID for a rate limiter that I created to dump SYN traffic which exeeds our typical SYN traffic levels by a factor of 5.  (Might change the trigger later, but that isn't really important)  This seems to work, except that if I make the wait time more than 5 seconds it never gets past action 3.0.  I assume that is because the wait time is longer than the polling interval, and the whole thing "resets" on the polling interval.


event manager applet DDoS_RESPONSE01

event snmp oid 1.3.6.1.4.1.9.9.166.1.17.1.1.21.80.65538 get-type exact entry-op gt entry-val "0" entry-type increment exit-op eq exit-val "0" poll-interval 10

trigger

action 1.0 cli command "enable"

action 1.1 cli command "config term"

action 1.2 cli command "no ip prefix-list PUBLIC_NETWORKS seq 50 permit 216.1.1.0/24 le 32"

action 1.9 syslog msg "DDoS Attach Detected. Removing Web Srvr Subnet from PUBLIC_NETWORKS."

action 3.0 wait 5

action 4.0 cli command "enable"

action 4.1 cli command "config term"

action 4.2 cli command "ip prefix-list PUBLIC_NETWORKS seq 50 permit 216.1.1.0/24 le 32"

action 4.9 syslog msg "DDoS Attach Timeout reached. Re-adding Web Srvr Subnet to PUBLIC_NETWORKS."

!

Joe Clarke Tue, 07/02/2013 - 08:29
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

What I'm suggesting will do what you want.  It will also handle the case where the DDoS attack happens again and again.  As a final measure, you can unconfigure the timer policy as the last thing it does.  For example:


action 010 cli command "action 6.0 $q no event manager applet restore-prefix$q"


Just insert your one CLI command into the proper place in my example list of actions.  Be sure to watch those $q markers.  You'll need to keep them.  What you'll find is that this solution works exactly like what you're doing, only it gives you a larger timer that runs asynchronously.

derek.small Tue, 07/02/2013 - 10:55
User Badges:

Ok I got it to work the way you suggested.  But I have a question.  After the initial event is triggered, the line in my prefix-list is removed, and the restore applet is created starting the restore timer with a value of X.  After the timer X expires the line is put back, but what would happen if the initial event is triggered again, before the timer X reaches zero?


I thought about using an environment variable for the timer to put the line back, then using that same environment variable with the "exit-time $tout" clause to make it harder for someone to do that if they start adjusting timer values later, but I can't get the exit-time clause to accept an environment variable as a parameter.


One more question, There seems to be a lot of variability in when my event fires.  The first time it fires it usually does so within about 10-15 seconds of the OID I'm monitoring reaching a non-zero value. After the first time it fires though, it can be anywhere from [exit-time + 30 seconds] to [exit-time + 250 seconds]


Thanks for all your help!  Sorry I missed your session at CiscoLive this year. It is always one of my favorites!


Here is my complete script:

ip prefix-list PUBLIC_NETWORKS seq 140 permit 10.5.1..0/24 le 32
!
event manager environment q "
event manager environment ATimeout 30
!
no event manager applet RESTORE_PREFIX
no event manager applet DDOS_RESPONSE01
event manager applet DDOS_RESPONSE01
event snmp oid 1.3.6.1.4.1.9.9.166.1.17.1.1.21.80.65538 get-type exact entry-op gt entry-val "0" entry-type increment exit-time 60 poll-interval 10
trigger
action 001 cli command "enable"
action 002 cli command "config term"
action 003 cli command "no ip prefix-list PUBLIC_NETWORKS seq 140 permit 10.5.1..0/24 le 32"
action 004 syslog msg "DDoS Attack Detected. Removing Web Srvr Subnet from PUBLIC_NETWORKS for $ATimeout seconds."

action 006 cli command "config term"
action 007 cli command "event manager applet RESTORE_PREFIX"
action 008 cli command "event timer countdown time $ATimeout "
action 009 cli command "action 001 cli command enable"
action 010 cli command "action 002 cli command $q config term $q"
action 011 cli command "action 003 cli command $q ip prefix-list PUBLIC_NETWORKS seq 140 permit 10.5.1..0/24 le 32 $q"
action 014 cli command "action 004 syslog msg $q DDoS Attack Timeout ($ATimeout) reached. Re-adding Web Srvr Subnet to PUBLIC_NETWORKS. $q "
action 021 cli command "action 010 cli command enable"
action 022 cli command "action 011 cli command $q config term $q"
action 025 cli command "action 012 cli command $q no event manager applet RESTORE_PREFIX $q"
!

Joe Clarke Tue, 07/02/2013 - 17:46
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Change action 006 to:


action 006 cli command "no event manager applet RESTORE_PREFIX"


Remove actions 021 and 022.


By doing that, you will ensure that if something happens to trigger the first policy again before the restore policy has a chance to run, the timer will reset.


As for the trigger time, this will depend on the polling cycle and the exit-time.  You have set the entry-type to be increment, so the object will have to increment at least once in a polling cycle (compared to the last time the event fired).  And if it increments during the exit-time hold-down that won't count.

derek.small Wed, 07/03/2013 - 05:17
User Badges:

Thanks for all your help Joe. 


Ok, so here is my current script, which seems to be working pretty well (changing to entry-type "value" fixed the variability in detection times).  For testing in the script below, I'm using a 30 second timeout for when the line gets put back, and a 60 second timeout for when monitoring should resume after the event is triggered. The script checks the value of the OID every 5 seconds.


The only other thing I would like to do with it that I can't figure out, is how to use an environment variable for the exit-time.  Ideally, I would just add a value, like 10 seconds, to the ATimeout variable.  However I can't figure out the syntax to just use a var for the exit-time.  Anyone know the secret (or if it is possible?)


!

event manager environment ATimeout 30

event manager environment q "

!

no event manager applet DDOS_RESPONSE01

event manager applet DDOS_RESPONSE01

event snmp oid 1.3.6.1.4.1.9.9.166.1.17.1.1.21.80.65538 get-type exact entry-op gt entry-val "0" entry-type value exit-time 60 poll-interval 5

trigger

action 001 cli command "enable"

action 002 cli command "config term"

action 003 cli command "no ip prefix-list PUBLIC_NETWORKS seq 140 permit 10.4.1.0/24 le 32"

action 004 syslog msg "DDoS Attack Detected. Removing Web Srvr Subnet from PUBLIC_NETWORKS for ($ATimeout) seconds."

action 005 cli command "event manager applet RESTORE_PREFIX"

action 006 cli command "event timer countdown time $ATimeout "

action 007 cli command "action 101 cli command $q enable $q"

action 008 cli command "action 102 cli command $q config term $q"

action 009 cli command "action 103 cli command $q no event manager applet RESTORE_PREFIX $q"

action 010 cli command "action 104 cli command $q ip prefix-list PUBLIC_NETWORKS seq 140 permit 10.4.1.0/24 le 32$q"

action 011 cli command "action 105 syslog msg $q DDoS Attack Timeout ($ATimeout) reached. Re-adding Web Srvr Subnet to PUBLIC_NETWORKS. $q "

action 012 cli command "action 106 cli command $q no event manager applet RESTORE_PREFIX $q"

!

exit

! !
event manager environment ATimeout 30
event manager environment q "
!
event manager applet DDOS_RESPONSE01
event snmp oid 1.3.6.1.4.1.9.9.166.1.17.1.1.21.80.65538 get-type exact entry-op gt entry-val "0" entry-type value exit-time 60 poll-interval 5
trigger
action 001 cli command "enable"
action 002 cli command "config term"
action 003 cli command "no ip prefix-list PUBLIC_NETWORKS seq 140 permit 10.4.1.0/24 le 32"
action 004 syslog msg "DDoS Attack Detected. Removing Web Srvr Subnet from PUBLIC_NETWORKS for ($ATimeout) seconds."
action 005 cli command "event manager applet RESTORE_PREFIX"
action 006 cli command "event timer countdown time $ATimeout "
action 007 cli command "action 101 cli command $q enable $q"
action 008 cli command "action 102 cli command $q config term $q"
action 009 cli command "action 103 cli command $q no event manager applet RESTORE_PREFIX $q"
action 010 cli command "action 104 cli command $q ip prefix-list PUBLIC_NETWORKS seq 140 permit 10.4.1.0/24 le 32$q"
action 011 cli command "action 105 syslog msg $q DDoS Attack Timeout ($ATimeout) reached. Re-adding Web Srvr Subnet to PUBLIC_NETWORKS. $q "
action 012 cli command "action 106 cli command $q no event manager applet RESTORE_PREFIX $q"
!
exit
!

Joe Clarke Wed, 07/03/2013 - 08:38
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Likely this is not possible.  Only certain fields were made variable in applets, and I'm betting something as infrequently used as exit-time was not one of them.  If this must be a variable, convert your applet to a Tcl policy and then you can use the variable exit-time.

Actions

This Discussion