Solved: ESM Script caused switch reloads...

Tim Glen · ‎08-05-2014

Hi

A while back I wrote an ESM script to look at the log entries as they are created and if the script found the string "Line protocol" it would create a new log entry that would include the configured description for an interface. The idea behind this was to be able to see a more friendly name in the SYSLOG messages.

This tcl script has worked fine on several switch platforms, several IOS versions. Recently I configured a 2960G (IOS 12.2.(53)SE2) to use the script . Later when an interface went down the script ran but then something went awry and the switch started rebooting.

Here is the script. Is there anything that's obviously wrong or stands out with this code ?

# Setup time and date in a usable format
set time [clock seconds]
set time [clock format $time -format "%b %d %T"]

if [string match "*Line protocol*" $::orig_msg] {
set int_numb [lindex $::msg_args 0]
set int_stat [lindex $::msg_args 1]
set int_stat [string toupper $int_stat]
set int_desc [exec "show runn interface $int_numb | inc description"]
set int_desc [string trim $int_desc " description"]
set int_desc [string trimright $int_desc]
return "$time, LINK_STAT_CHG $::hostname $int_numb -- $int_stat -- $int_desc"
} else {
return "$::orig_msg"
}

Thanks for looking.

Tim

Joe Clarke · ‎08-07-2014

There was another duplicate bug of this one that used your exact same stack trace. Given the crash and the error, plus the fix, I think this is your bug. You could upgrade one of the switches to confirm, but I am highly confident this is what is causing your crash.

View solution in original post

Joe Clarke · ‎08-05-2014

Please post the "show ver" and "show stack" after the crash.

Tim Glen · ‎08-05-2014

Hi Joe,

Two switches rebooted over and over, both are using the same logging filter.

Thanks for looking.

Tim

Output from Switch 1

DMZ-Switch1#
DMZ-Switch1#sh ver
Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 05:52 by prod_rel_team
Image text-base: 0x00003000, data-base: 0x01500000

ROM: Bootstrap program is C2960 boot loader
BOOTLDR: C2960 Boot Loader (C2960-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

DMZ-Switch1 uptime is 23 hours, 13 minutes
System returned to ROM by power-on
System restarted at 13:24:09 edt Mon Aug 4 2014
System image file is "flash:c2960-lanbasek9-mz.122-53.SE2.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960G-48TC-L (PowerPC405) processor (revision A0) with 65536K bytes of memory.
Processor board ID FOC1026X8FK
Last reset from power-on
2 Virtual Ethernet interfaces
48 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

64K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:0A:B8:41:23:00
Motherboard assembly number : 73-10300-06
Power supply part number : 341-0098-02
Motherboard serial number : FOC102547AP
Power supply serial number : AZS10240371
Model revision number : A0
Motherboard revision number : B0
Model number : WS-C2960G-48TC-L
System serial number : FOC1026X8FK
Top Assembly Part Number : 800-27071-01
Top Assembly Revision Number : B0
Version ID : V01
CLEI Code Number : COM4A10BRA
Hardware Board Revision Number : 0x01

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 48 WS-C2960G-48TC-L 12.2(53)SE2 C2960-LANBASEK9-M

Configuration register is 0xF

DMZ-Switch1#
DMZ-Switch1#
DMZ-Switch1#sh stack
Minimum process stacks:
Free/Size Name
4972/6000 hulc_flash init
5536/6000 CDP BLOB
4164/6000 EEM Shell Director
5372/6000 vqpc_shim_create_addr_tbl
5360/6000 SPAN Subsystem
59536/60000 EEM Auto Registration Proc
5248/6000 SASL MAIN
5528/6000 hpm bulk_vlan_state process
8008/12000 Init
8516/9000 cdp init process
5184/6000 RADIUS INITCONFIG
5496/6000 Hulc Backup Interface Process
2532/3000 Rom Random Update Process
9416/12000 Exec
9292/12000 SSH Process

Interrupt level stacks:
Level Called Unused/Size Name
4 1335439380 8232/9000 NETWORK INTERFACE CHIP
5 0 9000/9000 SUPERVISOR EXCEPTIONS
6 6686 8912/9000 NS16550 VECTOR

***************************************************
******* Information of Last System Crash **********
***************************************************

Using flash:/crashinfo/crashinfo_6.
Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 05:52 by prod_rel_team

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x01152228 SRR1 = 0x00029230 SRR2 = 0x010E5828 SRR3 = 0x00029230
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x84000000 DBSR = 0x01000000

CPU Register Context:
Vector = 0x00002000 PC = 0x010E5828 MSR = 0x00029230 CR = 0x33000093
LR = 0x010E5820 CTR = 0x0115C628 XER = 0xC000004C
R0 = 0x010E5820 R1 = 0x0268FFE0 R2 = 0x00000000 R3 = 0x00000000
R4 = 0x021B8F7C R5 = 0x00000000 R6 = 0x00000000 R7 = 0x021E69DC
R8 = 0x00000000 R9 = 0x00000000 R10 = 0x01EC0000 R11 = 0x00000000
R12 = 0x00000000 R13 = 0x00110000 R14 = 0x001CE6EC R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000002
R20 = 0x0260DC4C R21 = 0x021E69DC R22 = 0x021B8F7C R23 = 0x00000002
R24 = 0x021B8F80 R25 = 0x0335EE61 R26 = 0x021B8F7C R27 = 0x02611F6C
R28 = 0x02673558 R29 = 0x02672DD8 R30 = 0x00000002 R31 = 0x021B8F7C

Stack trace:
PC = 0x010E5828, SP = 0x0268FFE0
Frame 00: SP = 0x026900D8 PC = 0x010E5820
Frame 01: SP = 0x02690108 PC = 0x010EBBE8
Frame 02: SP = 0x02690128 PC = 0x010C7AEC
Frame 03: SP = 0x026903A0 PC = 0x001CFBEC
Frame 04: SP = 0x026903D0 PC = 0x001CE0E0
Frame 05: SP = 0x026903F8 PC = 0x001CE748
Frame 06: SP = 0x02690400 PC = 0x011BB59C
Frame 07: SP = 0x00000000 PC = 0x011B2028

DMZ-Switch1#

Output from Switch 2

DMZ-Switch2#
DMZ-Switch2#sh ver
Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 05:52 by prod_rel_team
Image text-base: 0x00003000, data-base: 0x01500000

ROM: Bootstrap program is C2960 boot loader
BOOTLDR: C2960 Boot Loader (C2960-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

DMZ-Switch2 uptime is 23 hours, 16 minutes
System returned to ROM by power-on
System restarted at 13:24:10 edt Mon Aug 4 2014
System image file is "flash:c2960-lanbasek9-mz.122-53.SE2.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960G-48TC-L (PowerPC405) processor (revision A0) with 65536K bytes of memory.
Processor board ID FOC1026X8DC
Last reset from power-on
2 Virtual Ethernet interfaces
48 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

64K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:0A:B8:41:88:80
Motherboard assembly number : 73-10300-06
Power supply part number : 341-0098-02
Motherboard serial number : FOC102547VJ
Power supply serial number : AZS1024030L
Model revision number : A0
Motherboard revision number : B0
Model number : WS-C2960G-48TC-L
System serial number : FOC1026X8DC
Top Assembly Part Number : 800-27071-01
Top Assembly Revision Number : B0
Version ID : V01
CLEI Code Number : COM4A10BRA
Hardware Board Revision Number : 0x01

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 48 WS-C2960G-48TC-L 12.2(53)SE2 C2960-LANBASEK9-M

Configuration register is 0xF

DMZ-Switch2#
DMZ-Switch2#
DMZ-Switch2#sh stack
Minimum process stacks:
Free/Size Name
4972/6000 hulc_flash init
5532/6000 CDP BLOB
4164/6000 EEM Shell Director
5372/6000 vqpc_shim_create_addr_tbl
5360/6000 SPAN Subsystem
59536/60000 EEM Auto Registration Proc
5248/6000 SASL MAIN
5524/6000 hpm bulk_vlan_state process
8008/12000 Init
8520/9000 cdp init process
5180/6000 RADIUS INITCONFIG
5496/6000 Hulc Backup Interface Process
2536/3000 Rom Random Update Process
9928/12000 Exec
7708/12000 SSH Process

Interrupt level stacks:
Level Called Unused/Size Name
4 1278720048 8232/9000 NETWORK INTERFACE CHIP
5 0 9000/9000 SUPERVISOR EXCEPTIONS
6 16589 8912/9000 NS16550 VECTOR

***************************************************
******* Information of Last System Crash **********
***************************************************

Using flash:/crashinfo/crashinfo_6.
Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 05:52 by prod_rel_team

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x006A822C SRR1 = 0x00029230 SRR2 = 0x010E5828 SRR3 = 0x00029230
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x84000000 DBSR = 0x01000000

CPU Register Context:
Vector = 0x00002000 PC = 0x010E5828 MSR = 0x00029230 CR = 0x33000093
LR = 0x010E5820 CTR = 0x0115C628 XER = 0xC0000011
R0 = 0x010E5820 R1 = 0x02665350 R2 = 0x00000000 R3 = 0x00000000
R4 = 0x021B8F7C R5 = 0x00000000 R6 = 0x00000000 R7 = 0x021E69DC
R8 = 0x00000000 R9 = 0x00000000 R10 = 0x01EC0000 R11 = 0x00000000
R12 = 0x00000000 R13 = 0x00110000 R14 = 0x001CE6EC R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000002
R20 = 0x025E3160 R21 = 0x021E69DC R22 = 0x021B8F7C R23 = 0x00000002
R24 = 0x021B8F80 R25 = 0x03391329 R26 = 0x021B8F7C R27 = 0x025E72DC
R28 = 0x025DA164 R29 = 0x025D9444 R30 = 0x00000002 R31 = 0x021B8F7C

Stack trace:
PC = 0x010E5828, SP = 0x02665350
Frame 00: SP = 0x02665448 PC = 0x010E5820
Frame 01: SP = 0x02665478 PC = 0x010EBBE8
Frame 02: SP = 0x02665498 PC = 0x010C7AEC
Frame 03: SP = 0x02665710 PC = 0x001CFBEC
Frame 04: SP = 0x02665740 PC = 0x001CE0E0
Frame 05: SP = 0x02665768 PC = 0x001CE748
Frame 06: SP = 0x02665770 PC = 0x011BB59C
Frame 07: SP = 0x00000000 PC = 0x011B2028

DMZ-Switch2#
DMZ-Switch2#

Additionally I saw this in the crashinfo_ext on DMZ-Sw2 which was the first switch to go down.

Aug 4 12:56:23.231: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to down
CMD: 'show runn interface GigabitEthernet0/24 | inc description' 12:56:23 edt Mon Aug 4 2014

and

---- Partial decode of process block ----

Pid 161: Process "ESM Logger "
stack 0x3397C74 savedsp 0x21C0CDC
Flags: analyze prefers_new
Status 0x00000000 Orig_ra 0x00000000 Routine 0x00000000 Signal 0
Caller_pc 0x00000000 Callee_pc 0x00000000 Dbg_events 0x00000000 State 0
Totmalloc 148320 Totfree 161244 Totgetbuf 0
Totretbuf 0 Edisms 0x0 Eparm 0x0
Elapsed 0x6C Ncalls 0x62 Ngiveups 0x22
Priority_q 5 Ticks_5s 2 Cpu_5sec 0 Cpu_1min 0
Cpu_5min 0 Stacksize 0xEA60 Lowstack 0xEA60
Ttyptr 0x21AD098 Mem_holding 0x163D4 Thrash_count 0
Wakeup_reasons 0x0FFFFFFF Default_wakeup_reasons 0x0FFFFFFF
Direct_wakeup_major 0x00000000 Direct_wakeup_minor 0x00000000

Joe Clarke · ‎08-05-2014

This is CSCsx32049. It is fixed in 12.2(58)SE2.

Tim Glen · ‎08-06-2014

Joe,

Thanks for the tip. I read the bug document and I'm curious why you think it's that bug as my ESM script doesn't contain an ios_config statement.

CC: ESM logger cause tracebacks and RP reload at boot

CSCsx32049
Description
Symptoms: Traceback is observed and the system may reboot, depending on the
platform.

Conditions: The symptom is observed when the ESM filter is configured and
contains an ios_config statement.

Workaround: Remove ios_config statements from ESM filter.

Thanks

Tim

Joe Clarke · ‎08-07-2014

There was another duplicate bug of this one that used your exact same stack trace. Given the crash and the error, plus the fix, I think this is your bug. You could upgrade one of the switches to confirm, but I am highly confident this is what is causing your crash.