Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

EEM OSPF syslog

hi all,

I'm using EEM to be warn by mail when my OSPF adj changes ; I use OSPF in tunel int, so I can know when tunnel is down/up.

My script works well but sometimes I do not receive any mail whereas adj changes, so I can receive mail with "up" whereas I did not receive "down".

I ask me if there is no rate-limiter on syslog msg that could explain my issue. what else ? if you have an idea, I'm interested.

Other problem in my case, when my Internet access falls down, I receive tons of mail (one by tunnel interface ; I have 50 int tunnel !)... so I would like to improve my script to detect internet issue and in that case send only a mail to tell me : no internet. Same when Internet is back.

my script :

***************************
event manager environment mail_smtp aaa.bbb.ccc.ddd.eee
event manager environment mail_to aaaa@aaa.aa

event manager applet OSPFtunnelDOWN
   event syslog pattern "OSPF-5-ADJCHG.*FULL to DOWN"
   action 100 info type routername
   action 110 regexp "on ([_0-9a-zA-Z]*) from" "$_syslog_msg" result nominterface
   action 120 cli command "show interface $nominterface"
   action 130 regexp "Description: ([a-zA-Z0-9\-]+)" "$_cli_result" result site
   action 200 mail server "$mail_smtp" to "$mail_to" from "$_info_routername@aaa.aa" subject "down : $site" body "down $site"

event manager applet OSPFtunnelUP
   event syslog pattern "OSPF-5-ADJCHG.*to FULL"
   action 100 info type routername
   action 110 regexp "on ([_0-9a-zA-Z]*) from" "$_syslog_msg" result nominterface
   action 120 cli command "show interface $nominterface"
   action 130 regexp "Description: ([a-zA-Z0-9\-]+)" "$_cli_result" result site
   action 200 mail server "$mail_smtp" to "$mail_to" from "$_info_routername@aaa.aa" subject "up : $site" body "up $site"
*************************************************************

help is welcome

thanks

Niko

1 ACCEPTED SOLUTION

Accepted Solutions
Cisco Employee

Re: EEM OSPF syslog

I should say that the particular EEM CLASS will only need up to nine VTY lines.  If you have other policies configured in other thread classes, you may still require more free VTY lines.

--

Please support CSC Helps Haiti

https://supportforums.cisco.com/docs/DOC-8895

https://supportforums.cisco.com

22 REPLIES
Cisco Employee

Re: EEM OSPF syslog

For the first issue, you would need to confirm that the syslog message is being generated first.  If not, that could be an issue with the OSPF subsystem, and you might want to inquire on the routing forum.  If the message is being generated, you may need to run with "debug event manager detector syslog" for a while so that when it happens again, there is some additional data as to why the applet is not being triggered.

For the flood problem, you could add additional actions to your applets to check the state of the internet interface.  If the interface is down, then do not send the email.  Since you have programmatic applet capability, this would be easy to do.  Something like:

action 101 cli command "show interface Serial1/0 | inc line proto"

action 102 regexp "line protocol is up" $_cli_result

action 103 if $_regexp_result ne 1

action 104 exit 0

action 105 end

--

Please support CSC Helps Haiti

https://supportforums.cisco.com/docs/DOC-8895

https://supportforums.cisco.com

New Member

Re: EEM OSPF syslog

ok I will check syslog info when I have this issue... and back to you

about second demand, it's not so easy.. when my internet has problem my outside interface has no problem, it's more after that something is not ok

I thought to use event detector counter but I don't know how to use it in my case

regards

Niko

Cisco Employee

Re: EEM OSPF syslog

How do you know that the internet link is experiencing a problem then?  Is it not passing any traffic?  That is, would a ping to a known host on the internet fail in this "bad" scenario?

--

Please support CSC Helps Haiti

https://supportforums.cisco.com/docs/DOC-8895

https://supportforums.cisco.com

New Member

Re: EEM OSPF syslog

ok so I took 2 peers in my system :

on one of them, I done : clear ip ospf proc

with a debug on this side, I can see ospf down then up (full)

on the second, I added my EEM script + "debug event manager detector syslog"

look what I get :

Apr  4 08:59:59.224: %OSPF-5-ADJCHG: Process 1, Nbr 23.2.0.1 on Tunnel16 from LOADING to FULL, Loading Done
Apr  4 08:59:59.228: syslog_pubinfo_enqueue: matched pattern="OSPF-5-ADJCHG.*to FULL", matched message="
Apr  4 08:59:59.224: %OSPF-5-ADJCHG: Process 1, Nbr 23.2.0.1 on Tunnel16 from LOADING to FULL, Loading Done"
Apr  4 08:59:59.228: fh_fd_syslog_event_match: num_matches = 1
Apr  4 08:59:59.228: fh_fd_data_syslog: num_matches = 1
Apr  4 08:59:59.228: fh_send_syslog_fd_msg: msg_type=64

...

why there is no "FULL to DOWN" OSPF syslog msg ??? that's why I just got one mail with : tunnel16 UP whereas I missed a mail with tunnel16 DOWN

any idea ?

thanks

Niko

New Member

Re: EEM OSPF syslog

hmm maybe I found the problem but don't know how to solve it to work with monitoring :

sometimes there is no "full to down"

I added a "debug ip ospf adj"

1) when I shut tunnel int on other side :

Apr  4 09:19:48.618: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel16, changed state to down
Apr  4 09:19:48.618: fh_fd_syslog_event_match: num_matches = 0
Apr  4 09:19:48.618: fh_fd_data_syslog: num_matches = 0
Apr  4 09:19:48.622: OSPF: Interface Tunnel16 going Down
Apr  4 09:19:48.622: OSPF: 27.2.0.1 address 0.0.0.0 on Tunnel16 is dead, state DOWN
Apr  4 09:19:48.622: OSPF: 23.2.0.1 address 192.168.23.68 on Tunnel16 is dead, state DOWN
Apr  4 09:19:48.622: %OSPF-5-ADJCHG: Process 1, Nbr 23.2.0.1 on Tunnel16 from FULL to DOWN, Neighbor Down: Interface down or detached

....

Apr  4 09:20:05.314: %OSPF-5-ADJCHG: Process 1, Nbr 23.2.0.1 on Tunnel16 from LOADING to FULL, Loading Done
....

2) when I do "clear ip ospf proc" on other side :

Apr  4 09:21:13.066: OSPF: Cannot see ourself in hello from 23.2.0.1 on Tunnel16, state INIT
Apr  4 09:21:13.066: fh_fd_syslog_event_match: num_matches = 0
Apr  4 09:21:13.066: fh_fd_data_syslog: num_matches = 0
Apr  4 09:21:13.098: OSPF: 2 Way Communication to 23.2.0.1 on Tunnel16, state 2WAY
Apr  4 09:21:13.098: OSPF: Tunnel16 Nbr 23.2.0.1: Prepare dbase exchange
Apr  4 09:21:13.098: OSPF: Send DBD to 23.2.0.1 on Tunnel16 seq 0x1379 opt 0x52 flag 0x7 len 32
Apr  4 09:21:13.098: OSPF: Rcv DBD from 23.2.0.1 on Tunnel16 seq 0x153E opt 0x52 flag 0x7 len 32  mtu 1427 state EXSTART

....

Apr  4 09:21:13.359: %OSPF-5-ADJCHG: Process 1, Nbr 23.2.0.1 on Tunnel16 from LOADING to FULL, Loading Done

....

so any idea how to improve my script to be ok with my mail ?

Thanks

Niko

PS : I detected also several : "OSPF: Nbr 23.2.0.1 has larger interface MTU", can you tell me if it's important or not ? other than that all works well

Cisco Employee

Re: EEM OSPF syslog

As I said, if the messages are not being generated, you should ask in the routing forum.  They can also explain the full impact of the MTU mismatch.

As for EEM, you might have to convert to using a poll-based method to determine if the OSPF adjacency is down.  For example, parse the output of "show ip ospf neighbor" to see if you have lost a neighbor.  You might also consider using an SNMP ED to look at the OSPF-MIB, and see if there is a change there.  For example, perhaps ospfNbrState is transitioning to down.  The downside of this approach would be you would need one applet per neighbor.  Depending on your version of IOS, an SNMP trap might also be an option.  However, if the syslog is no being generated, a trap might not be generated either.

--

Please support CSC Helps Haiti

https://supportforums.cisco.com/docs/DOC-8895

https://supportforums.cisco.com

New Member

Re: EEM OSPF syslog

ok I will try on routing forum

about eem, seems to be more hard to do that :-(

I have an other idea, change ospf log to "detail" ; what do you think to add an other event :

event syslog pattern "OSPF-5-ADJCHG.*FULL to INIT" so I could generate mail to say DOWN in that case as well ?

is it possible in this case to create only one event with 2 syslogs msg ?

a kind of :

event syslog pattern "OSPF-5-ADJCHG.*FULL to INIT" or pattern "OSPF-5-ADJCHG.*FULL to DOWN"

about internet link, yes, more often issue is not on my router but an other that I don't manage... so yes, I could test with a ping to the net to know if there is problem ; I thought to use it

Niko

Cisco Employee

Re: EEM OSPF syslog

Sure, you can do that:

event syslog pattern "OSPF-5-ADJCHG.*FULL to (DOWN|INIT)"

You have two choices.  One is to execute a simple ping from within your applet, and check the resulting output to see if it worked.  For example:

action 104 cli command "ping x.x.x.x"

action 105 regexp "!!!!!" $_cli_result

action 106 if $_regexp_result ne 1

action 107 exit 0

action 108 end

The other option is to create an IP SLA collector which periodically pings a host on the Internet.  Then your policy could check the cli output of "show ip sla stat" to see what the latest return code is.  This could also be done with Enhanced Object Tracking, and even another EEM applet which watches the state of the tracked object, and sets a context variable if the state goes down.

--

Please support CSC Helps Haiti

https://supportforums.cisco.com/docs/DOC-8895

https://supportforums.cisco.com

New Member

Re: EEM OSPF syslog

nice !

now it works well

thanks for your help

regards

nicolas

New Member

Re: EEM OSPF syslog

hi,

it stays something wrong :-(

after doing "clear ip ospf proc" + debug event...

I have ton of same errors :

Apr  7 06:44:27.203: %HA_EM-3-FMPD_CLI_CONNECT: Unable to establish CLI session: no tty lines available, minimum of 2 required by EEM
Apr  7 06:44:27.203: %HA_EM-3-FMPD_ERROR: Error executing applet OSPFtunnelDOWN statement 120
Apr  7 06:44:27.211: %HA_EM-3-FMPD_CLI_CONNECT: Unable to establish CLI session: no tty lines available, minimum of 2 required by EEM
Apr  7 06:44:27.211: %HA_EM-3-FMPD_ERROR: Error executing applet OSPFtunnelDOWN statement 120
Apr  7 06:44:27.215: %HA_EM-3-FMPD_CLI_CONNECT: Unable to establish CLI session: no tty lines available, minimum of 2 required by EEM
Apr  7 06:44:27.215: %HA_EM-3-FMPD_ERROR: Error executing applet OSPFtunnelDOWN statement 120

.....

well, after reading forum and online books, I fixed it in increasing "line vty" to 50 ; what's your point about this change ?

FYI : I use ISR 3825 HSEC/K9 with IOS 12.4(24)T

remember you my script about OSPFtunnelDOWN :

event manager applet OSPFtunnelDOWN
event syslog pattern "OSPF-5-ADJCHG.*FULL to (DOWN|INIT)"
action 100 info type routername
action 110 regexp "on ([_0-9a-zA-Z]*) from" "$_syslog_msg" result nominterface
action 120 cli command "show interface $nominterface"
action 130 regexp "Description: ([a-zA-Z0-9\-]+)" "$_cli_result" result site
action 200 mail server "$mail_smtp" to "$mail_to" from "$_info_routername@toto.fr" subject "$site" body "$site down"

Thanks

Nicolas

Cisco Employee

Re: EEM OSPF syslog

You said you were getting one OSPF message per adjacency, per tunnel.  By default, there are 32 applet threads, so up to 32 applets can run in parallel.  This will exhaust the available VTY lines if all the tunnels go down at once.  A quick workaround would be either to do what you did with VTY lines, or reduce the number of applet threads to something like 8:

Router(config)#event manager scheduler applet thread class default number 8

Alternatively, you could create a second thread class for this script (say class A).  Only allow eight parallel executions of class A.  Then, tell your applet to execute within this class:

Router(config)#event manager scheduler applet thread class A number 8

Router(config)#event manager applet OSPFtunnelDOWN class A

...

If eight instances still overwhelms the number of VTYs, decrease the thread class until it works as desired.

--

Please support CSC Helps Haiti

https://supportforums.cisco.com/docs/DOC-8895

https://supportforums.cisco.com

New Member

Re: EEM OSPF syslog

ok, I tried to reduce the number of applet threads with 8, still same issue

I will try with less but if I have to add again tunnel interface I need to check again if this value is still correct :-( so maybe better to stay with "line vty 50"

FYI, I have more than 50 tunnel interfaces, so when there is a problem in link, I should receive 50 mails to warn me (DOWN) then when it is back 50 mails again (UP)

Niko

Cisco Employee

Re: EEM OSPF syslog

Given a tunnel count of 50, you will need around 50 VTY lines if all tunnels drop at one time.  With eight execution threads, you would only need about 10.  If you create a new thread class, be sure you assign your policy to that class, else it will still execute in the default class.

--

Please support CSC Helps Haiti

https://supportforums.cisco.com/docs/DOC-8895

https://supportforums.cisco.com

New Member

Re: EEM OSPF syslog

ok but I would like to know what's better to use ? is there any impact for one of them ?

just VTY 50

or a mix with number of thread + vty ?

thanks

nicolas

Cisco Employee

Re: EEM OSPF syslog

I think the thread option is best.  While you can increase the VTYs, you are also increasing the load on your device by running so many policy instances at once.

--

Please support CSC Helps Haiti

https://supportforums.cisco.com/docs/DOC-8895

https://supportforums.cisco.com

New Member

Re: EEM OSPF syslog

ok I just tested in putting :

thread 8 and vty 10 => no success (still problem)

thread 8 and vty 15 => seems to be ok :-)

sounds ok for you now ?

Niko

Cisco Employee

Re: EEM OSPF syslog

You just need to make sure you have enough VTY lines free to satisfy all your policy instances, all of your interactive sessions, then one extra as EEM will not take the last free VTY line.  So if 15 is the magic number for you, that sounds good.  That is the default.

--

Please support CSC Helps Haiti

https://supportforums.cisco.com/docs/DOC-8895

https://supportforums.cisco.com

New Member

Re: EEM OSPF syslog

ok so if I need to add again tunnel interface, I have to check if I still have enough vty lines

so if not enough, I increase it (vty) until it is ok ?

thanks for your big help

NIko

Cisco Employee

Re: EEM OSPF syslog

No.  By keeping your threads at eight, you ensure that EEM will only ever need up to nine VTY lines to be free.  So you can add as many tunnels as you want provided you can keep nine lines free at any given time.

--

Please support CSC Helps Haiti

https://supportforums.cisco.com/docs/DOC-8895

https://supportforums.cisco.com

Cisco Employee

Re: EEM OSPF syslog

I should say that the particular EEM CLASS will only need up to nine VTY lines.  If you have other policies configured in other thread classes, you may still require more free VTY lines.

--

Please support CSC Helps Haiti

https://supportforums.cisco.com/docs/DOC-8895

https://supportforums.cisco.com

New Member

Re: EEM OSPF syslog

Hi Joseph,

I'm back to you because now I have more than 40 tunnel interfaces and I get this message in log :

%HA_EM-3-FMPD_CLI_CONNECT: Unable to establish CLI session: no tty lines available, minimum of 2 required by EEM

remember you my configuration :

!

line vty 5 15
login
transport input none
!

!

event manager environment mail_smtp ****
event manager environment mail_to *****
event manager environment mail_from *****
event manager scheduler applet thread class default number 8
event manager applet OSPFtunnelUP
event syslog pattern "OSPF-5-ADJCHG.*to FULL"
action 100 info type routername
action 110 regexp "on ([_0-9a-zA-Z]*) from" "$_syslog_msg" result nominterface
action 120 cli command "show interface $nominterface"
action 130 regexp "Description: ([a-zA-Z0-9\-]+)" "$_cli_result" result site
action 200 mail server "$mail_smtp" to "$mail_to" from "$mail_from" subject "$site UP" body "$site is up"
event manager applet OSPFtunnelDOWN
event syslog pattern "OSPF-5-ADJCHG.*FULL to *"
action 100 info type routername
action 110 regexp "on ([_0-9a-zA-Z]*) from" "$_syslog_msg" result nominterface
action 120 cli command "show interface $nominterface"
action 130 regexp "Description: ([a-zA-Z0-9\-]+)" "$_cli_result" result site
action 200 mail server "$mail_smtp" to "$mail_to" from "$mail_from" subject "$site DOWN" body "$site is down."
!

so what I need to change again to not get this error in log and get mail alert ? increase vty lines => 5 20 ?

best regards

Nicolas

PS : this issue is present only in eem version ? maybe with new IOS it's ok and I don't need more vty lines ?

Cisco Employee

Re: EEM OSPF syslog

For this configuration, you could need at least nine free VTY lines.  Upgrading IOS will not change that requirement.  If you only have "line vty 0 5" configured, you need to increase that to at least "vty 0 8".  If all eight threads execute, then you can only have one user logged in.  Any more that that, and you will see the error you're seeing.  You can also reduce the thread count some more to limit the number of concurrent policies.

2455
Views
0
Helpful
22
Replies
CreatePlease to create content