EEM Call Home Engine aka Automating the Troubleshooting

Raphael Wouters · ‎06-14-2010

What's This About?

EEM Call Home Engine helps automating the troubleshooting or management task by providing a means for automatic execution of a sequence of commands, based on an event or time based trigger. This document explains the structure and use of EEM Call Home Engine.

Components

EEM Call Home Engine has three components:

EEM Tcl policy no_call_home_eng.tcl. This is the heart of the EEM Call Home Engine.
Policy configuration files (.csv). These files contain rules for creating dynamic policies.
Dynamically created EEM Tcl policies.

How It Works?

no_call_home_eng.tcl. creates, installs and uninstalls dynamic policies based on configuration files. This EEM policy has no other purpose. It does not execute unless explicitly invoked via CLI. This policy can and should remain registered even after troubleshooting is complete, it will not consume extra ressources after the registration. The no_call_home_eng.tcl policy will be provided by TAC engineers.

Dynamically created policies execute specified CLI commands when the specified condition is met. Collected output is sent via email to the email address specified in the .csv file that will be provided by TAC engineers. Source email address and SMTP server must be configured in the global router configuration.

Prerequisites

The device must support the show event manager environment EXEC level command.
IOS or IOS-XE with support for EEM (note: IOS-XR 3.6 may work but has not been fully tested).

Preparing the installation

If the device is configured for AAA command authorization (i.e. if aaa authorization commands line is present in device configuration), then the following must be configured:

Router#config t
Router(config)#event manager session cli username <USER>

where <USER> is a user authorized to run all privilege level 15 commands.

Configure sender email address and SMTP:
- If show call-home command is supported:

Router#config t
Router(config)#call-home
Router(cfg-call-home)#sender from <email_address>
Router(cfg-call-home)#mail-server <ip_address> priority <n>

Multiple mail servers may be specified, transfer via lowest priority server will be attempted first.

Otherwise, if Call Home is not supported, configure the following:

Router#config t
Router(config)#event manager environment _email_from <email_address>
Router(config)#event manager environment _email_server <ip_address>

Create a directory for policies and temporary files on one of the local file systems, e.g.:

Router#mkdir flash:/policies
Router#mkdir flash:/temp
Router#config t
Router(config)#event manager directory user policy flash:/policies
Router(config)#event manager environment che_temp_dir flash:/temp

show <filesystem> all

Download and register no_call_home_eng.tcl policy:

Router#copy tftp://<ip_address>/no_call_home_eng.tcl flash:/policies
Router#config t
Router(config)#event manager policy no_call_home_eng.tcl

Verify the installation:

Router#show event manager policy registered
No. Class Type Event Type Trap Time Registered Name
1 script user none Off Fri May 28 11:38:40 2010 no_call_home_eng.tcl
policyname {no_call_home_eng.tcl} sync {yes}
nice 0 queue-priority normal maxrun 90.000 scheduler rp_primary

Installing Dynamic Policies

Once you received policy configuration file(s) (.csv) from TAC, use the no_call_home_eng.tcl script to download them from your FTP,TFTP or SCP server:

Determine if your image support EEM version 2.4 or later by running show event manager version, if the command is supported the image support EEM 2.4 and later versions.

Install the policy, here is an example for a policy named high_cpu_tac for a TAC service request number 614300461 :

If your device support EEM version 2.4:

Router#event manager run no_call_home_eng.tcl tftp://<ip_address>/614300461-high_cpu_tac.csv
 Sending file modes: C0644 455 614300461-high_cpu_tac.csv
!INFO: Successfully installed and registered 614300461-high_cpu_tac.tcl.

Otherwise you must an environment variable to specify the path to the policy configuration file:

Router#config t
Router(config)#event manager environment che_install_file tftp://<ip_address>/614300461-high_cpu_tac.csv
Router(config)#end
Router#event manager run no_call_home_eng.tcl

Verify the installation:

Router#show event manager policy registered 

[...]
2    script    user    snmp         Off   Thu Jun 3 15:10:09 2010   614300461-high_cpu_tac.tcl
 oid {1.3.6.1.4.1.9.9.109.1.1.1.1.10.1} get_type exact entry_op ge entry_val {80}
entry_type value exit_op le exit_val {30} exit_type value poll_interval 5.000
 nice 0 queue-priority normal maxrun 21.000 scheduler rp_primary
[...]

Confirming the Policies Are Working

When a dynamically created policy runs, and collects the CLI data, it will do two things to notify the user that data has been collected:

The first thing will be to generate a syslog message. In the typical case when output data is sent directly to the Service Request, the syslog will look like the following.
*Jun 3 15:53:24.659: %HA_EM-6-LOG: 614300461-high_cpu_tac.tcl: Policy executed successfully; data was sent to 614300461.
The message may vary if the script have an additional Cc address, if the data are saved locally on a file or if no output are produced with this script.

The second thing that will be done is to send email to the addresses specified in To email address and Carbon-copy email address(es) fields. This will only be done if the output file has been specified, or if the output file value is NONE. The email will not contain the data.
In order for the notification emails to be sent, the sender email address and SMTP server must configured as explained in Preparing the installation paragraph.

Stopping and Removing Policies

It is VITAL that you uninstall, and stop your configured policy to prevent it from spamming the Service Request, or taking up unnecessary resources on the device. If a limited number of iterations for your policy has been specified, the policy will handle uninstalling itself. If not, then follow these steps to stop and uninstall the policies:

If your device support EEM 2.4 or later, use the same command you used at the installation and add --uninstall as first argument:
```
Router#event manager run no_call_home_eng.tcl --uninstall tftp://<ip_address>/614300461-high_cpu_tac.csv
```

If the device do not support EEM 2.4, we have to set the che_uninstall_file environment variable to the path to the CSV file which contains the policy definitions to uninstall.
```
Router#config t
Router(config)#event manager environment che_uninstall_file tftp://<ip_address>/614300461-high_cpu_tac.csv
 
```
Then run the no_call_home_eng.tcl script.
```
Router#event manager run no_call_home_eng.tcl
```

NOTE:

If you leave out the file name argument after --uninstall, no_call_home_eng.tcl will prompt you for the policy to uninstall.
The EEM Call Home Engine policy itself will not be uninstalled.

Uninstalling and removing the Call Home engine

There is no need to remove the Call Home engine after the troubleshooting is complete and the service request closed. The Call Home engine has been designed to be kept installed on the router, and keeping it will facilitate the future installation of customized EEM policies. If there is a need to uninstall the Call Home engine policy you may follow those steps:

Unregister the no_call_home_eng.tcl policy.

Router#config t
Router(config)#no event manager policy no_call_home_eng.tcl

Remove the no_call_home_eng.tcl script from the user policy directory.
```
Router#del /force flash:/policies/no_call_home_eng.tcl
```

Dictionary

Cisco TAC - Cisco Technical Assistance Centre

CLI - Command Line Interface

EEM - Embedded Event Manager

SMTP - Simple Mail Transfer Protocol