Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
New Member

ESM Script caused switch reloads...

Hi

A while back I wrote an ESM script to look at the log entries as they are created and if the script found the string "Line protocol" it would create a new log entry that would include the configured description for an interface.  The idea behind this was to be able to see a more friendly name in the SYSLOG messages.

This tcl script has worked fine on several switch platforms, several IOS versions.   Recently I configured a 2960G (IOS 12.2.(53)SE2)  to use the script .  Later when an interface went down the script ran but then something went awry and the switch started rebooting. 

Here is the script.     Is there anything that's obviously wrong or stands out with this code ?

 

#  Setup time and date in a usable format
set time [clock seconds]
set time [clock format $time -format "%b %d %T"]

if [string match "*Line protocol*" $::orig_msg] {
 set int_numb [lindex $::msg_args 0]
 set int_stat [lindex $::msg_args 1]
 set int_stat [string toupper $int_stat]
 set int_desc [exec "show runn interface $int_numb | inc description"]
 set int_desc [string trim $int_desc " description"]
 set int_desc [string trimright $int_desc]
 return "$time, LINK_STAT_CHG $::hostname $int_numb -- $int_stat -- $int_desc"
 } else  {
  return "$::orig_msg" 
}

 

 

Thanks for looking.

Tim

1 ACCEPTED SOLUTION

Accepted Solutions
Cisco Employee

There was another duplicate

There was another duplicate bug of this one that used your exact same stack trace.  Given the crash and the error, plus the fix, I think this is your bug.  You could upgrade one of the switches to confirm, but I am highly confident this is what is causing your crash.

5 REPLIES
Cisco Employee

Please post the "show ver"

Please post the "show ver" and "show stack" after the crash.

New Member

Hi Joe,Two switches rebooted

Hi Joe,

Two switches rebooted over and over, both are using the same logging filter. 

Thanks for looking.

Tim

 

Output from Switch 1

 

DMZ-Switch1#
DMZ-Switch1#sh ver
Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 05:52 by prod_rel_team
Image text-base: 0x00003000, data-base: 0x01500000

ROM: Bootstrap program is C2960 boot loader
BOOTLDR: C2960 Boot Loader (C2960-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

DMZ-Switch1 uptime is 23 hours, 13 minutes
System returned to ROM by power-on
System restarted at 13:24:09 edt Mon Aug 4 2014
System image file is "flash:c2960-lanbasek9-mz.122-53.SE2.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960G-48TC-L (PowerPC405) processor (revision A0) with 65536K bytes of memory.
Processor board ID FOC1026X8FK
Last reset from power-on
2 Virtual Ethernet interfaces
48 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

64K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 00:0A:B8:41:23:00
Motherboard assembly number     : 73-10300-06
Power supply part number        : 341-0098-02
Motherboard serial number       : FOC102547AP
Power supply serial number      : AZS10240371
Model revision number           : A0
Motherboard revision number     : B0
Model number                    : WS-C2960G-48TC-L
System serial number            : FOC1026X8FK
Top Assembly Part Number        : 800-27071-01
Top Assembly Revision Number    : B0
Version ID                      : V01
CLEI Code Number                : COM4A10BRA
Hardware Board Revision Number  : 0x01


Switch Ports Model              SW Version            SW Image                 
------ ----- -----              ----------            ----------               
*    1 48    WS-C2960G-48TC-L   12.2(53)SE2           C2960-LANBASEK9-M        


Configuration register is 0xF

DMZ-Switch1# 
DMZ-Switch1#
DMZ-Switch1#sh stack
Minimum process stacks:
 Free/Size   Name
 4972/6000   hulc_flash init
 5536/6000   CDP BLOB
 4164/6000   EEM Shell Director
 5372/6000   vqpc_shim_create_addr_tbl
 5360/6000   SPAN Subsystem
59536/60000  EEM Auto Registration Proc
 5248/6000   SASL MAIN
 5528/6000   hpm bulk_vlan_state process
 8008/12000  Init
 8516/9000   cdp init process
 5184/6000   RADIUS INITCONFIG
 5496/6000   Hulc Backup Interface Process
 2532/3000   Rom Random Update Process
 9416/12000  Exec
 9292/12000  SSH Process

Interrupt level stacks:
Level    Called Unused/Size  Name
  4  1335439380   8232/9000  NETWORK INTERFACE CHIP
  5           0   9000/9000  SUPERVISOR EXCEPTIONS
  6        6686   8912/9000  NS16550 VECTOR

***************************************************
******* Information of Last System Crash **********
***************************************************


Using flash:/crashinfo/crashinfo_6.
Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 05:52 by prod_rel_team

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x01152228  SRR1 = 0x00029230  SRR2 = 0x010E5828  SRR3 = 0x00029230
ESR = 0x00000000  DEAR = 0x00000000  TSR = 0x84000000  DBSR = 0x01000000

CPU Register Context:
Vector = 0x00002000  PC = 0x010E5828  MSR = 0x00029230  CR = 0x33000093
LR = 0x010E5820  CTR = 0x0115C628  XER = 0xC000004C
R0 = 0x010E5820  R1 = 0x0268FFE0  R2 = 0x00000000  R3 = 0x00000000
R4 = 0x021B8F7C  R5 = 0x00000000  R6 = 0x00000000  R7 = 0x021E69DC
R8 = 0x00000000  R9 = 0x00000000  R10 = 0x01EC0000  R11 = 0x00000000
R12 = 0x00000000  R13 = 0x00110000  R14 = 0x001CE6EC  R15 = 0x00000000
R16 = 0x00000000  R17 = 0x00000000  R18 = 0x00000000  R19 = 0x00000002
R20 = 0x0260DC4C  R21 = 0x021E69DC  R22 = 0x021B8F7C  R23 = 0x00000002
R24 = 0x021B8F80  R25 = 0x0335EE61  R26 = 0x021B8F7C  R27 = 0x02611F6C
R28 = 0x02673558  R29 = 0x02672DD8  R30 = 0x00000002  R31 = 0x021B8F7C

Stack trace:
PC = 0x010E5828, SP = 0x0268FFE0
Frame 00: SP = 0x026900D8    PC = 0x010E5820
Frame 01: SP = 0x02690108    PC = 0x010EBBE8
Frame 02: SP = 0x02690128    PC = 0x010C7AEC
Frame 03: SP = 0x026903A0    PC = 0x001CFBEC
Frame 04: SP = 0x026903D0    PC = 0x001CE0E0
Frame 05: SP = 0x026903F8    PC = 0x001CE748
Frame 06: SP = 0x02690400    PC = 0x011BB59C
Frame 07: SP = 0x00000000    PC = 0x011B2028


DMZ-Switch1#

 

Output from Switch 2

 

DMZ-Switch2#
DMZ-Switch2#sh ver
Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 05:52 by prod_rel_team
Image text-base: 0x00003000, data-base: 0x01500000

ROM: Bootstrap program is C2960 boot loader
BOOTLDR: C2960 Boot Loader (C2960-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

DMZ-Switch2 uptime is 23 hours, 16 minutes
System returned to ROM by power-on
System restarted at 13:24:10 edt Mon Aug 4 2014
System image file is "flash:c2960-lanbasek9-mz.122-53.SE2.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960G-48TC-L (PowerPC405) processor (revision A0) with 65536K bytes of memory.
Processor board ID FOC1026X8DC
Last reset from power-on
2 Virtual Ethernet interfaces
48 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

64K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 00:0A:B8:41:88:80
Motherboard assembly number     : 73-10300-06
Power supply part number        : 341-0098-02
Motherboard serial number       : FOC102547VJ
Power supply serial number      : AZS1024030L
Model revision number           : A0
Motherboard revision number     : B0
Model number                    : WS-C2960G-48TC-L
System serial number            : FOC1026X8DC
Top Assembly Part Number        : 800-27071-01
Top Assembly Revision Number    : B0
Version ID                      : V01
CLEI Code Number                : COM4A10BRA
Hardware Board Revision Number  : 0x01


Switch Ports Model              SW Version            SW Image                 
------ ----- -----              ----------            ----------               
*    1 48    WS-C2960G-48TC-L   12.2(53)SE2           C2960-LANBASEK9-M        


Configuration register is 0xF

DMZ-Switch2#
DMZ-Switch2#
DMZ-Switch2#sh stack
Minimum process stacks:
 Free/Size   Name
 4972/6000   hulc_flash init
 5532/6000   CDP BLOB
 4164/6000   EEM Shell Director
 5372/6000   vqpc_shim_create_addr_tbl
 5360/6000   SPAN Subsystem
59536/60000  EEM Auto Registration Proc
 5248/6000   SASL MAIN
 5524/6000   hpm bulk_vlan_state process
 8008/12000  Init
 8520/9000   cdp init process
 5180/6000   RADIUS INITCONFIG
 5496/6000   Hulc Backup Interface Process
 2536/3000   Rom Random Update Process
 9928/12000  Exec
 7708/12000  SSH Process

Interrupt level stacks:
Level    Called Unused/Size  Name
  4  1278720048   8232/9000  NETWORK INTERFACE CHIP
  5           0   9000/9000  SUPERVISOR EXCEPTIONS
  6       16589   8912/9000  NS16550 VECTOR

***************************************************
******* Information of Last System Crash **********
***************************************************


Using flash:/crashinfo/crashinfo_6.
Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 05:52 by prod_rel_team

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x006A822C  SRR1 = 0x00029230  SRR2 = 0x010E5828  SRR3 = 0x00029230
ESR = 0x00000000  DEAR = 0x00000000  TSR = 0x84000000  DBSR = 0x01000000

CPU Register Context:
Vector = 0x00002000  PC = 0x010E5828  MSR = 0x00029230  CR = 0x33000093
LR = 0x010E5820  CTR = 0x0115C628  XER = 0xC0000011
R0 = 0x010E5820  R1 = 0x02665350  R2 = 0x00000000  R3 = 0x00000000
R4 = 0x021B8F7C  R5 = 0x00000000  R6 = 0x00000000  R7 = 0x021E69DC
R8 = 0x00000000  R9 = 0x00000000  R10 = 0x01EC0000  R11 = 0x00000000
R12 = 0x00000000  R13 = 0x00110000  R14 = 0x001CE6EC  R15 = 0x00000000
R16 = 0x00000000  R17 = 0x00000000  R18 = 0x00000000  R19 = 0x00000002
R20 = 0x025E3160  R21 = 0x021E69DC  R22 = 0x021B8F7C  R23 = 0x00000002
R24 = 0x021B8F80  R25 = 0x03391329  R26 = 0x021B8F7C  R27 = 0x025E72DC
R28 = 0x025DA164  R29 = 0x025D9444  R30 = 0x00000002  R31 = 0x021B8F7C

Stack trace:
PC = 0x010E5828, SP = 0x02665350
Frame 00: SP = 0x02665448    PC = 0x010E5820
Frame 01: SP = 0x02665478    PC = 0x010EBBE8
Frame 02: SP = 0x02665498    PC = 0x010C7AEC
Frame 03: SP = 0x02665710    PC = 0x001CFBEC
Frame 04: SP = 0x02665740    PC = 0x001CE0E0
Frame 05: SP = 0x02665768    PC = 0x001CE748
Frame 06: SP = 0x02665770    PC = 0x011BB59C
Frame 07: SP = 0x00000000    PC = 0x011B2028


DMZ-Switch2#
DMZ-Switch2#

 

Additionally I saw this in the crashinfo_ext  on DMZ-Sw2 which was the first switch to go down. 

 

Aug  4 12:56:23.231: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to down
CMD: 'show runn interface GigabitEthernet0/24 | inc description' 12:56:23 edt Mon Aug 4 2014

 

and 

 

---- Partial decode of process block ----

Pid 161: Process "ESM Logger "
stack 0x3397C74  savedsp 0x21C0CDC 
Flags: analyze prefers_new 
Status     0x00000000 Orig_ra   0x00000000 Routine    0x00000000 Signal 0
Caller_pc  0x00000000 Callee_pc 0x00000000 Dbg_events 0x00000000 State  0
Totmalloc  148320     Totfree   161244     Totgetbuf  0       
Totretbuf  0          Edisms    0x0        Eparm 0x0       
Elapsed    0x6C       Ncalls    0x62       Ngiveups 0x22      
Priority_q 5          Ticks_5s  2          Cpu_5sec   0        Cpu_1min 0
Cpu_5min   0          Stacksize 0xEA60     Lowstack 0xEA60    
Ttyptr     0x21AD098  Mem_holding 0x163D4    Thrash_count 0
Wakeup_reasons      0x0FFFFFFF  Default_wakeup_reasons 0x0FFFFFFF
Direct_wakeup_major 0x00000000  Direct_wakeup_minor 0x00000000

 

 

Cisco Employee

This is CSCsx32049.  It is

This is CSCsx32049.  It is fixed in 12.2(58)SE2.

New Member

Joe,  Thanks for the tip.  I

Joe, 

 

Thanks for the tip.  I read the bug document and I'm curious why you think it's that bug as my ESM script doesn't contain an ios_config statement. 

 

CC: ESM logger cause tracebacks and RP reload at boot

CSCsx32049
Description
Symptoms: Traceback is observed and the system may reboot, depending on the
platform.

Conditions: The symptom is observed when the ESM filter is configured and
contains an ios_config statement.

Workaround: Remove ios_config statements from ESM filter.

Thanks


Tim

 

 

Cisco Employee

There was another duplicate

There was another duplicate bug of this one that used your exact same stack trace.  Given the crash and the error, plus the fix, I think this is your bug.  You could upgrade one of the switches to confirm, but I am highly confident this is what is causing your crash.

177
Views
0
Helpful
5
Replies
CreatePlease to create content