C6513 module(ACE) got rebooted...logs attached ...need Urgent help

Unanswered Question
Feb 23rd, 2010
User Badges:

Hi All,


need urgent help...


Our customer is having C6513 running with 12.2(18)SXF15a . One of the module (ACE20-MOD-K9) got rebooted with following error:

======================================================================================================

Feb 22 10:41:41.155 GMT: %OIR-SP-3-PWRCYCLE: Card in module 6, is being power-cycled off (Reset - Module Reloaded During Download)
Feb 22 10:41:41.183 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 6 set off (Reset - Module Reloaded During Download)
Feb 22 10:41:53.686 GMT: %OIR-SP-3-PWRCYCLE: Card in module 6, is being power-cycled off (Module not responding to Keep Alive polling)
Feb 22 10:41:53.686 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 6 set off (Module not responding to Keep Alive polling)

======================================================================================================

------------------ show version ------------------

Cisco Internetwork Operating System Software
IOS (tm) s72033_rp Software (s72033_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(18)SXF15a, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by cisco Systems, Inc.
Compiled Tue 21-Oct-08 00:41 by kellythw
Image text-base: 0x40101040, data-base: 0x42DD6470

ROM: System Bootstrap, Version 12.2(17r)S4, RELEASE SOFTWARE (fc1)
BOOTLDR: s72033_rp Software (s72033_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(18)SXF15a, RELEASE SOFTWARE (fc1)

BCLNDASEC59-6513INTPRI uptime is 50 weeks, 3 days, 10 hours, 43 minutes
Time since BCLNDASEC59-6513INTPRI switched to active is 50 weeks, 3 days, 10 hours, 47 minutes
System returned to ROM by reload (SP by reload)
System restarted at 02:12:07 GMT Sat Mar 7 2009
System image file is "sup-bootdisk:s72033-adventerprisek9_wan-mz.122-18.SXF15a.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
[email protected].

cisco WS-C6513 (R7000) processor (revision 1.1) with 983008K/65536K bytes of memory.
Processor board ID SAL104365G2
SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
Last reset from s/w reset
SuperLAT software (copyright 1990 by Meridian Technology Corp).
X.25 software, Version 3.0.0.
Bridging software.
TN3270 Emulation software.
34 Virtual Ethernet/IEEE 802.3 interfaces
40 Gigabit Ethernet/IEEE 802.3 interfaces
18 Ten Gigabit Ethernet/IEEE 802.3 interfaces
1917K bytes of non-volatile configuration memory.
8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102

Can any one help me to find the root cause as i need to share the RCA and neccessary steps to avoid such incident.

======================================================================================================

Need urgent help...


Regards

Madhu

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 3 (1 ratings)
Loading.
Giuseppe Larosa Mon, 03/01/2010 - 13:05
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Madhu,

may you post a sh module to see what image is running on ACE module?


some ACE images are deferred


Hope to help

Giuseppe

madhusudhan s Thu, 03/04/2010 - 23:44
User Badges:

------------------ show module ------------------



Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  1    1  Application Control Engine Module      ACE10-6500-K9      SAD1046038Z
  2   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP       SAD104608LF
  4    6  Firewall Module                        WS-SVC-FWM-1       SAD112902X7
  6    1  Application Control Engine Module      ACE20-MOD-K9       SAD124104DE
  7    2  Supervisor Engine 720 (Active)         WS-SUP720-3BXL     SAD104305MS
  8    2  Supervisor Engine 720 (Hot)            WS-SUP720-3BXL     SAD104305T6
10    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE      SAL1034YZ4B
11    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE      SAD104209MD
12    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE      SAL09084EGP
13    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE      SAD104209MK


Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  1  0019.aacc.a9d6 to 0019.aacc.a9dd   1.3   8.7(0.22)ACE A2(2.2)      Ok
  2  0019.aacc.a6a6 to 0019.aacc.a6bd   2.4   12.2(14r)S5  12.2(18)SXF1 Ok
  4  001c.5861.9318 to 001c.5861.931f   4.2   7.2(1)       4.0(5)       Ok
  6  0022.55b3.eb48 to 0022.55b3.eb4f   2.4   8.7(0.22)ACE A2(1.4a)     Ok
  7  0011.9202.8fc0 to 0011.9202.8fc3   5.2   8.4(2)       12.2(18)SXF1 Ok
  8  000a.b818.9fd4 to 000a.b818.9fd7   5.2   8.4(2)       12.2(18)SXF1 Ok
10  0018.b9c4.a86c to 0018.b9c4.a86f   2.4   12.2(14r)S5  12.2(18)SXF1 Ok
11  0019.561c.7198 to 0019.561c.719b   2.4   12.2(14r)S5  12.2(18)SXF1 Ok
12  0013.1972.5e60 to 0013.1972.5e63   2.3   12.2(14r)S5  12.2(18)SXF1 Ok
13  0019.561c.7178 to 0019.561c.717b   2.4   12.2(14r)S5  12.2(18)SXF1 Ok


Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
  2  Distributed Forwarding Card WS-F6700-DFC3BXL   SAD104500XT  5.3    Ok
  7  Policy Feature Card 3       WS-F6K-PFC3BXL     SAD10420AS3  1.8    Ok
  7  MSFC3 Daughterboard         WS-SUP720          SAD104309US  2.5    Ok
  8  Policy Feature Card 3       WS-F6K-PFC3BXL     SAD1042047M  1.8    Ok
  8  MSFC3 Daughterboard         WS-SUP720          SAD104309PP  2.5    Ok
10  Distributed Forwarding Card WS-F6700-DFC3BXL   SAL10425HLN  5.3    Ok
11  Distributed Forwarding Card WS-F6700-DFC3BXL   SAD1045010A  5.3    Ok
12  Distributed Forwarding Card WS-F6700-DFC3BXL   SAL10425HLE  5.3    Ok
13  Distributed Forwarding Card WS-F6700-DFC3BXL   SAD103808S0  5.3    Ok


Mod  Online Diag Status
---- -------------------
  1  Pass
  2  Pass
  4  Pass
  6  Pass
  7  Pass
  8  Pass
10  Pass
11  Pass
12  Pass
13  Pass


Regard

Amar

Giuseppe Larosa Fri, 03/05/2010 - 09:17
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Amar,

your ACE module is running A2(2) software


from A2.(2) release notes there are two open bugs that could have caused the module reload


CSCsv92321, CSCsx25981—The ACE module reboots unexpectedly and writes a core file to the disk. Workaround: None.


http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/vA2_2_x/release/note/RACEA2_2X.html#wp550550


both  have been closed and merged with


CSCsq38638            Bug Details



SRAM Parity Error ~LEGAL ADDR~sometimes IFMGR core at same time
Symptom:

The ACE blade cores indicating a SRAM Parity Error. Occasionally another type of process (such as IFMGR, etc.) core may accompany the SRAM error crash.

Conditions:

This is a rare condition where the ACE blade is running and performs an SRAM operation that detects an SRAM parity error. 

Workaround:

Reboot of the ACE will clear the state.  This reboot is accomplished automatically when the corefile is created.



Hope to  help

Giuseppe

Luis Trincheiras Mon, 07/23/2012 - 03:32
User Badges:

Giuseppe,


I have exactly the same problem, however my version is A2(3.5).

Can you help?


Tks

Luis

Actions

This Discussion