Solved: A Problem with VSS in the 6509

Bill_Yang1227 · ‎06-22-2014

Hello,

I have a problem with VSS in the 6500, when the switch 1 is active,probably a few months will switchover to switch 2 and switch 1 reload.

config:

switch virtual domain 1
switch mode virtual
switch 1 priority 110
switch 2 priority 110

interface Port-channel1
description To 6509-B-VSS
no switchport
no ip address
switch virtual link 1
mls qos trust cos
no mls qos channel-consistency
!
interface Port-channel2
description 6509-A-VSS
no switchport
no ip address
switch virtual link 2
mls qos trust cos
no mls qos channel-consistency
!

interface GigabitEthernet1/4/20
no switchport
no ip address
dual-active fast-hello

interface TenGigabitEthernet1/5/4
description To 6509-B-VSS
no switchport
no ip address
mls qos trust cos
channel-group 1 mode on
!
interface TenGigabitEthernet1/5/5
description To 6509-B-VSS
no switchport
no ip address
mls qos trust cos
channel-group 1 mode on
!

interface GigabitEthernet2/4/20
no switchport
no ip address
dual-active fast-hello
!

interface TenGigabitEthernet2/5/4
description To 6509-A-VSS
no switchport
no ip address
mls qos trust cos
channel-group 2 mode on
!
interface TenGigabitEthernet2/5/5
description To 6509-A-VSS
no switchport
no ip address
mls qos trust cos
channel-group 2 mode on
!

show switch virtual redundancy

                  My Switch Id = 2
                Peer Switch Id = 1
        Last switchover reason = active unit removed
    Configured Redundancy Mode = sso
     Operating Redundancy Mode = sso

Switch 2 Slot 5 Processor Information :
-----------------------------------------------
        Current Software state = ACTIVE
       Uptime in current state = 5 hours, 24 minutes
                 Image Version = Cisco IOS Software, s72033_rp Software (s72033_rp-IPBASEK9-M), Version 12.2(33)SXJ5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Thu 31-Jan-13 14:30 by prod_rel_team
                          BOOT = sup-bootdisk:/s72033-ipbasek9-mz.122-33.SXJ5.bin,12;
        Configuration register = 0x2102
                  Fabric State = ACTIVE
           Control Plane State = ACTIVE

Switch 1 Slot 5 Processor Information :
-----------------------------------------------
        Current Software state = STANDBY HOT (switchover target)
       Uptime in current state = 5 hours, 18 minutes
                 Image Version = Cisco IOS Software, s72033_rp Software (s72033_rp-IPBASEK9-M), Version 12.2(33)SXJ5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Thu 31-Jan-13 14:30 by prod_rel_team
                          BOOT = sup-bootdisk:/s72033-ipbasek9-mz.122-33.SXJ5.bin,12;
        Configuration register = 0x2102
                  Fabric State = ACTIVE
           Control Plane State = STANDBY

show log

Jun 21 11:20:52.742: %VSL-SW2_SPSTBY-5-VSL_CNTRL_LINK: New VSL Control Link Te2/5/4
Jun 21 11:20:52.786: %VSLP-SW2_SPSTBY-3-VSLP_LMP_FAIL_REASON: Te2/5/4: Link down
Jun 21 11:20:52.786: %VSLP-SW2_SPSTBY-2-VSL_DOWN: Last VSL interface Te2/5/4 went down

Jun 21 11:20:52.790: %VSLP-SW2_SPSTBY-2-VSL_DOWN: All VSL links went down while switch is in Standby role
Jun 21 11:20:52.790: %DUAL_ACTIVE-SW2_SPSTBY-1-VSL_DOWN: VSL is down - switchover, or possible dual-active situation has occurred
Jun 21 11:20:53.622: %SYS-SW2_SPSTBY-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
Jun 21 11:20:54.678: %C6KPWR-SP-4-PSOK: power supply 1 turned on.
Jun 21 11:20:54.682: %C6KPWR-SP-4-PSOK: power supply 2 turned on.
Jun 21 11:20:54.738: %SATVS_IBC-SW2_SP-5-VSL_DOWN_SCP_DROP: VSL inactive - dropping cached SCP packet: (SA/DA:0x4/0x4, SSAP/DSAP:0x19/0x0, OP/SEQ:0x320/0x8FD9, SIG/INFO:0x1/0x502, eSA:0000.0500.0000)
Jun 21 11:20:56.490: %SATVS_IBC-5-VSL_DOWN_SCP_DROP: VSL inactive - dropping cached SCP packet: (SA/DA:0x14/0x4, SSAP/DSAP:0x18/0x0, OP/SEQ:0x19/0x98, SIG/INFO:0x1/0x502, eSA:0000.1500.0000)

Jun 21 11:21:02.125: %VSDA-SW2_SP-3-LINK_DOWN: Interface Gi2/4/20 is no longer dual-active detection capable
Jun 21 11:23:27.719: %VSLP-SW2_SP-5-RRP_ROLE_RESOLVED: Role resolved as ACTIVE by VSLP
Jun 21 11:23:27.719: %VSL-SW2_SP-5-VSL_CNTRL_LINK: New VSL Control Link Te2/5/4

Jun 21 11:23:27.799: %VSLP-SW2_SP-5-VSL_UP: Ready for control traffic
Jun 21 11:24:43.346: %PFINIT-SW2_SP-5-CONFIG_SYNC: Sync'ing the startup configuration to the standby Router.
Jun 21 11:25:35.304: %VSLP-SW2_SP-5-VSL_UP: Ready for data traffic
Jun 21 11:25:14.946: %C6KPWR-SW1_SPSTBY-4-PSOK: power supply 1 turned on.
Jun 21 11:25:14.950: %C6KPWR-SW1_SPSTBY-4-PSOK: power supply 2 turned on.
Jun 21 11:25:16.975: %FABRIC-SW1_SPSTBY-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 5.
Jun 21 11:25:16.975: %FABRIC-SW1_SPSTBY-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 5 became active.
Jun 21 11:26:13.738: %SYS-SW1_SPSTBY-5-RESTART: System restarted --
Jun 21 11:26:20.446: %SYS-SW1_SPSTBY-3-LOGGER_FLUSHED: System was paused for 00:01:47 to ensure console debugging output.
Jun 21 11:26:32.056: %RF-SW2_SP-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)
Jun 21 11:27:56.962: %SYS-SW1_SPSTBY-3-LOGGER_FLUSHED: System was paused for 00:01:24 to ensure console debugging output.
Jun 21 11:27:53.970: %SYS-DFC4-5-RESTART: System restarted --
Jun 21 11:28:34.222: %VSDA-SW1_SPSTBY-5-LINK_UP: Interface Gi1/4/20 is now dual-active detection capable
Jun 21 11:28:36.236: %VSDA-SW2_SP-5-LINK_UP: Interface Gi2/4/20 is now dual-active detection capable

Best Regard

Rajeev Sharma · ‎06-24-2014

Hey,

So it did crash and the name of file tells the date and time - bootflash:crashinfo_SP_20140621-192052

I suggest opening a TAC case and get trace backs decoded for root cause of supervisor crash.

HTH.

Regards,

RS.

View solution in original post

Reza Sharifi · ‎06-22-2014

Hi,

Delete these 2 lines from you config and see if you still have the same issue.

switch 1 priority 110
switch 2 priority 110

By default when both switches boot up at the same time, switch 1 will always be the primary and switch 2 backup, so there no need for priority. If switch 1 fails switch 2 will take over and stay as primary.

HTH

Bill_Yang1227 · ‎06-23-2014

Thank you for providing the requested information.

But my question is why the switch 1 switchover and then reload.

Is it possible this problem related to VSL link?

Mitesh Manwatkar · ‎06-23-2014

Hi,

Instead of deleting..it is better to assign priority...u can keep like dis..

switch 1 priority 110
switch 2 priority 100

Also i wud like 2 suggest 2 have 2 members in dual-active fast-hello per switch.

We had faced similar issue...here is d description:

Problem Description:

The active supervisor engine crashed due to a parity error. Parity errors occur when a bit’s original value is changed to the opposite one (0 to 1, or vice versa). This problem can occur as one of two different types of parity errors, soft parity errors or hard parity errors. In our case, it looks like we are dealing with a soft parity error.

With a soft parity error, the bit’s value is changed due to an external influence on the switch, for example the most common are voltage variations, radiation of the environment, etc. While these types of problems are rare, they can happen and are transient issue that will not reoccur.

A hard parity error on the other hand are caused by an actual hardware failure. Parity errors are not bound to Cisco products, these can happen to any piece of electronics.

When the switch detected the parity error, it reloaded itself to purge the memory and continue with its normal behavior.

In our case the switch it has a modular version of the IOS, which means that if something fails it won’t reload the whole box. But, seeing that the problem happened in the kernel module, which is the “heart” of the switch, it did reload itself. The kernel module is very stable and a parity error is pretty much the only reason it could have failed. Here is the message depicting the parity error:

Rajeev Sharma · ‎06-23-2014

Hey,

As per logs looks like switch 1 lost the power and came back up, do you have any power outages on any circuit? Also look for any crashinfo file for switch 1; other than that it will be difficult to know the root cause.

HTH.

Regards,

RS

Bill_Yang1227 · ‎06-24-2014

hi,
Information of Last System Crash - SP

Writing crashinfo to bootflash:crashinfo_SP_20140621-192052-TW 00 07 00 00 00 00 00 00 00 0F 00 00 00 00 00 00 10 D6 0A 06 FD 37 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00
Jun 21 11:19:43.723: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.723: SW1_SP: IPC: Message 504CA554 timed out waiting for Ack
Jun 21 11:19:43.723: SW1_SP: IPC: MSG: ptr: 0x504CA554, flags: 0x24100, retries: 8, seq: 0x315E1ED, refcount: 1, rpc_result = 0x0, data_buffer = 0x504104BC, header = 0x8A453C8, data = 0x8A453E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57837, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF38, lo: 0x8A453E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 5E 0A 06 01 01 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 08 00 00
Jun 21 11:19:43.723: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504CA554, flags: 0x24100, retries: 8, seq: 0x315E1ED, refcount: 1, rpc_result = 0x0, data_buffer = 0x504104BC, header = 0x8A453C8, data = 0x8A453E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57837, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF38, lo: 0x8A453E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 5E 0A 06 01 01 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 08 00 00
Jun 21 11:19:43.727: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.727: SW1_SP: IPC: Message 4528FE20 timed out waiting for Ack
Jun 21 11:19:43.727: SW1_SP: IPC: MSG: ptr: 0x4528FE20, flags: 0x24100, retries: 8, seq: 0x315E1EE, refcount: 1, rpc_result = 0x0, data_buffer = 0x4520B23C, header = 0x8C884C8, data = 0x8C884E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57838, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF39, lo: 0x8C884E8 || DATA: 2C D5 00 01 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 87 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 40 0A 06 0A 15 01 04 00 00 00 87 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00
Jun 21 11:19:43.727: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x4528FE20, flags: 0x24100, retries: 8, seq: 0x315E1EE, refcount: 1, rpc_result = 0x0, data_buffer = 0x4520B23C, header = 0x8C884C8, data = 0x8C884E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57838, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF39, lo: 0x8C884E8 || DATA: 2C D5 00 01 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 87 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 40 0A 06 0A 15 01 04 00 00 00 87 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00
Jun 21 11:19:43.727: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.727: SW1_SP: IPC: Message 4529E380 timed out waiting for Ack
Jun 21 11:19:43.727: SW1_SP: IPC: MSG: ptr: 0x4529E380, flags: 0x24100, retries: 8, seq: 0x315E1EF, refcount: 1, rpc_result = 0x0, data_buffer = 0x50418920, header = 0x8A93DC8, data = 0x8A93DE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57839, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF3A, lo: 0x8A93DE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 BC 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 12 00 00 00 00 00 00 08 9E 0A 06 9A 0B 01 04 00 00 00 BC 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 16 00 00
Jun 21 11:19:43.727: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x4529E380, flags: 0x24100, retries: 8, seq: 0x315E1EF, refcount: 1, rpc_result = 0x0, data_buffer = 0x50418920, header = 0x8A93DC8, data = 0x8A93DE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57839, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF3A, lo: 0x8A93DE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 BC 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 12 00 00 00 00 00 00 08 9E 0A 06 9A 0B 01 04 00 00 00 BC 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 16 00 00
Jun 21 11:19:43.731: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.731: SW1_SP: IPC: Message 45238524 timed out waiting for Ack
Jun 21 11:19:43.731: SW1_SP: IPC: MSG: ptr: 0x45238524, flags: 0x24100, retries: 8, seq: 0x315E1F0, refcount: 1, rpc_result = 0x0, data_buffer = 0x50372694, header = 0x8468FC8, data = 0x8468FE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57840, sz: 1332, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF3B, lo: 0x8468FE8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 05 00 00 00 00 00 00 03 74 0A 06 FD 4C 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.731: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x45238524, flags: 0x24100, retries: 8, seq: 0x315E1F0, refcount: 1, rpc_result = 0x0, data_buffer = 0x50372694, header = 0x8468FC8, data = 0x8468FE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57840, sz: 1332, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF3B, lo: 0x8468FE8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 05 00 00 00 00 00 00 03 74 0A 06 FD 4C 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.731: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.731: SW1_SP: IPC: Message 4524C89C timed out waiting for Ack
Jun 21 11:19:43.731: SW1_SP: IPC: MSG: ptr: 0x4524C89C, flags: 0x24100, retries: 6, seq: 0x315E1F1, refcount: 1, rpc_result = 0x0, data_buffer = 0x451B45EC, header = 0x894FCC8, data = 0x894FCE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57841, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFE9, lo: 0x894FCE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 CF 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00
Jun 21 11:19:43.731: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x4524C89C, flags: 0x24100, retries: 6, seq: 0x315E1F1, refcount: 1, rpc_result = 0x0, data_buffer = 0x451B45EC, header = 0x894FCC8, data = 0x894FCE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57841, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFE9, lo: 0x894FCE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 CF 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00
Jun 21 11:19:43.731: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.731: SW1_SP: IPC: Message 45268348 timed out waiting for Ack
Jun 21 11:19:43.731: SW1_SP: IPC: MSG: ptr: 0x45268348, flags: 0x24100, retries: 6, seq: 0x315E1F2, refcount: 1, rpc_result = 0x0, data_buffer = 0x50462254, header = 0x8D4EFC8, data = 0x8D4EFE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57842, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFEA, lo: 0x8D4EFE8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 86 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 38 00 00 00 00 00 00 26 AB 0A 06 05 41 01 04 00 00 00 86 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 19 00 00
Jun 21 11:19:43.731: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x45268348, flags: 0x24100, retries: 6, seq: 0x315E1F2, refcount: 1, rpc_result = 0x0, data_buffer = 0x50462254, header = 0x8D4EFC8, data = 0x8D4EFE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57842, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFEA, lo: 0x8D4EFE8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 86 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 38 00 00 00 00 00 00 26 AB 0A 06 05 41 01 04 00 00 00 86 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 19 00 00
Jun 21 11:19:43.735: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.735: SW1_SP: IPC: Message 45281D58 timed out waiting for Ack
Jun 21 11:19:43.735: SW1_SP: IPC: MSG: ptr: 0x45281D58, flags: 0x24100, retries: 6, seq: 0x315E1F3, refcount: 1, rpc_result = 0x0, data_buffer = 0x45201860, header = 0x8C2CEC8, data = 0x8C2CEE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57843, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFEB, lo: 0x8C2CEE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 B9 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 04 24 0A 06 97 0D 01 04 00 00 00 BB 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.735: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x45281D58, flags: 0x24100, retries: 6, seq: 0x315E1F3, refcount: 1, rpc_result = 0x0, data_buffer = 0x45201860, header = 0x8C2CEC8, data = 0x8C2CEE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57843, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFEB, lo: 0x8C2CEE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 B9 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 04 24 0A 06 97 0D 01 04 00 00 00 BB 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.735: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.735: SW1_SP: IPC: Message 504A2D18 timed out waiting for Ack
Jun 21 11:19:43.735: SW1_SP: IPC: MSG: ptr: 0x504A2D18, flags: 0x24100, retries: 6, seq: 0x315E1F4, refcount: 1, rpc_result = 0x0, data_buffer = 0x451F3A88, header = 0x8BA92C8, data = 0x8BA92E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57844, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFEC, lo: 0x8BA92E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 05 00 00 00 00 00 00 01 48 0A 06 FD 2F 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00
Jun 21 11:19:43.735: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504A2D18, flags: 0x24100, retries: 6, seq: 0x315E1F4, refcount: 1, rpc_result = 0x0, data_buffer = 0x451F3A88, header = 0x8BA92C8, data = 0x8BA92E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57844, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFEC, lo: 0x8BA92E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 05 00 00 00 00 00 00 01 48 0A 06 FD 2F 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00
Jun 21 11:19:43.735: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.735: SW1_SP: IPC: Message 504AA128 timed out waiting for Ack
Jun 21 11:19:43.735: SW1_SP: IPC: MSG: ptr: 0x504AA128, flags: 0x24100, retries: 6, seq: 0x315E1F5, refcount: 1, rpc_result = 0x0, data_buffer = 0x45228364, header = 0x8D9C8C8, data = 0x8D9C8E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57845, sz: 568, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFED, lo: 0x8D9C8E8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 02 26 00 0D 00 10 02 20 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 08 00 00 00 00 00 00 02 3D 0A 06 FE 6C 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 2D 00 00
Jun 21 11:19:43.739: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504AA128, flags: 0x24100, retries: 6, seq: 0x315E1F5, refcount: 1, rpc_result = 0x0, data_buffer = 0x45228364, header = 0x8D9C8C8, data = 0x8D9C8E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57845, sz: 568, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFED, lo: 0x8D9C8E8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 02 26 00 0D 00 10 02 20 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 08 00 00 00 00 00 00 02 3D 0A 06 FE 6C 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 2D 00 00
Jun 21 11:19:43.739: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.739: SW1_SP: IPC: Message 504C15B4 timed out waiting for Ack
Jun 21 11:19:43.739: SW1_SP: IPC: MSG: ptr: 0x504C15B4, flags: 0x24100, retries: 4, seq: 0x315E1F6, refcount: 1, rpc_result = 0x0, data_buffer = 0x503C193C, header = 0x87593C8, data = 0x87593E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57846, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08B, lo: 0x87593E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 3B 00 00 00 00 00 00 10 A7 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 01 3B 00 00
Jun 21 11:19:43.739: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504C15B4, flags: 0x24100, retries: 4, seq: 0x315E1F6, refcount: 1, rpc_result = 0x0, data_buffer = 0x503C193C, header = 0x87593C8, data = 0x87593E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57846, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08B, lo: 0x87593E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 3B 00 00 00 00 00 00 10 A7 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 01 3B 00 00
Jun 21 11:19:43.739: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.739: SW1_SP: IPC: Message 45273050 timed out waiting for Ack
Jun 21 11:19:43.739: SW1_SP: IPC: MSG: ptr: 0x45273050, flags: 0x24100, retries: 4, seq: 0x315E1F7, refcount: 1, rpc_result = 0x0, data_buffer = 0x5044348C, header = 0x8C29BC8, data = 0x8C29BE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57847, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08C, lo: 0x8C29BE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 89 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 02 00 00 00 00 00 00 02 24 0A 06 0C 18 01 04 00 00 00 98 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.739: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x45273050, flags: 0x24100, retries: 4, seq: 0x315E1F7, refcount: 1, rpc_result = 0x0, data_buffer = 0x5044348C, header = 0x8C29BC8, data = 0x8C29BE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57847, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08C, lo: 0x8C29BE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 89 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 02 00 00 00 00 00 00 02 24 0A 06 0C 18 01 04 00 00 00 98 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.743: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.743: SW1_SP: IPC: Message 452A33E4 timed out waiting for Ack
Jun 21 11:19:43.743: SW1_SP: IPC: MSG: ptr: 0x452A33E4, flags: 0x24100, retries: 4, seq: 0x315E1F8, refcount: 1, rpc_result = 0x0, data_buffer = 0x50432830, header = 0x8B8A5C8, data = 0x8B8A5E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57848, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08D, lo: 0x8B8A5E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 C4 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 04 24 0A 06 A2 0B 01 04 00 00 00 C5 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.743: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x452A33E4, flags: 0x24100, retries: 4, seq: 0x315E1F8, refcount: 1, rpc_result = 0x0, data_buffer = 0x50432830, header = 0x8B8A5C8, data = 0x8B8A5E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57848, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08D, lo: 0x8B8A5E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 C4 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 04 24 0A 06 A2 0B 01 04 00 00 00 C5 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.743: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.743: SW1_SP: IPC: Message 4523FFA4 timed out waiting for Ack
Jun 21 11:19:43.743: SW1_SP: IPC: MSG: ptr: 0x4523FFA4, flags: 0x24100, retries: 4, seq: 0x315E1F9, refcount: 1, rpc_result = 0x0, data_buffer = 0x5036F47C, header = 0x844B3C8, data = 0x844B3E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57849, sz: 772, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08E, lo: 0x844B3E8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 02 F2 00 0D 00 16 02 EC 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00 00 00 00 00 01 46 0A 06 FE 5A 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00
Jun 21 11:19:43.743: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x4523FFA4, flags: 0x24100, retries: 4, seq: 0x315E1F9, refcount: 1, rpc_result = 0x0, data_buffer = 0x5036F47C, header = 0x844B3C8, data = 0x844B3E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57849, sz: 772, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08E, lo: 0x844B3E8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 02 F2 00 0D 00 16 02 EC 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00 00 00 00 00 01 46 0A 06 FE 5A 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00
Jun 21 11:19:43.743: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.743: SW1_SP: IPC: Message 45279200 timed out waiting for Ack
Jun 21 11:19:43.743: SW1_SP: IPC: MSG: ptr: 0x45279200, flags: 0x24100, retries: 2, seq: 0x315E1FA, refcount: 1, rpc_result = 0x0, data_buffer = 0x5039C3B0, header = 0x85F65C8, data = 0x85F65E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57850, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B123, lo: 0x85F65E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 CF 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 02 00 00
Jun 21 11:19:43.747: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x45279200, flags: 0x24100, retries: 2, seq: 0x315E1FA, refcount: 1, rpc_result = 0x0, data_buffer = 0x5039C3B0, header = 0x85F65C8, data = 0x85F65E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57850, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B123, lo: 0x85F65E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 CF 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 02 00 00
Jun 21 11:19:43.747: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.747: SW1_SP: IPC: Message 504D3544 timed out waiting for Ack
Jun 21 11:19:43.747: SW1_SP: IPC: MSG: ptr: 0x504D3544, flags: 0x24100, retries: 2, seq: 0x315E1FB, refcount: 1, rpc_result = 0x0, data_buffer = 0x5044AE34, header = 0x8C71FC8, data = 0x8C71FE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57851, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B124, lo: 0x8C71FE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 88 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00 00 00 00 00 04 48 0A 06 0B 24 01 04 00 00 00 89 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00
Jun 21 11:19:43.747: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504D3544, flags: 0x24100, retries: 2, seq: 0x315E1FB, refcount: 1, rpc_result = 0x0, data_buffer = 0x5044AE34, header = 0x8C71FC8, data = 0x8C71FE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57851, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B124, lo: 0x8C71FE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 88 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00 00 00 00 00 04 48 0A 06 0B 24 01 04 00 00 00 89 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00
Jun 21 11:19:43.747: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.747: SW1_SP: IPC: Message 504D5940 timed out waiting for Ack
Jun 21 11:19:43.747: SW1_SP: IPC: MSG: ptr: 0x504D5940, flags: 0x24100, retries: 2, seq: 0x315E1FC, refcount: 1, rpc_result = 0x0, data_buffer = 0x4521415C, header = 0x8CDD4C8, data = 0x8CDD4E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57852, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B125, lo: 0x8CDD4E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 BE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 0E 6D 0A 06 9C 5C 01 04 00 00 00 BF 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 08 00 00
Jun 21 11:19:43.747: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504D5940, flags: 0x24100, retries: 2, seq: 0x315E1FC, refcount: 1, rpc_result = 0x0, data_buffer = 0x4521415C, header = 0x8CDD4C8, data = 0x8CDD4E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57852, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B125, lo: 0x8CDD4E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 BE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 0E 6D 0A 06 9C 5C 01 04 00 00 00 BF 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 08 00 00
Jun 21 11:19:43.751: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.751: SW1_SP: IPC: Message 452989EC timed out waiting for Ack
Jun 21 11:19:43.751: SW1_SP: IPC: MSG: ptr: 0x452989EC, flags: 0x24100, retries: 2, seq: 0x315E1FD, refcount: 1, rpc_result = 0x0, data_buffer = 0x45172660, header = 0x86DCEC8, data = 0x86DCEE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57853, sz: 1060, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B126, lo: 0x86DCEE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 07 00 00 00 00 00 00 03 3E 0A 06 FE 1D 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 03 BF 00 00
Jun 21 11:19:43.751: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x452989EC, flags: 0x24100, retries: 2, seq: 0x315E1FD, refcount: 1, rpc_result = 0x0, data_buffer = 0x45172660, header = 0x86DCEC8, data = 0x86DCEE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57853, sz: 1060, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B126, lo: 0x86DCEE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 07 00 00 00 00 00 00 03 3E 0A 06 FE 1D 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 03 BF 00 00
Jun 21 11:19:43.751: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.751: SW1_SP: IPC: Message 504C9DAC timed out waiting for Ack
Jun 21 11:19:43.751: SW1_SP: IPC: MSG: ptr: 0x504C9DAC, flags: 0x24108, retries: 0, seq: 0x315E1FE, refcount: 1, rpc_result = 0x0, data_buffer = 0x451D3748, header = 0x8A772C8, data = 0x8A772E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57854, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D4, lo: 0x8A772E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 CF 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 97 00 00
Jun 21 11:19:43.751: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504C9DAC, flags: 0x24108, retries: 0, seq: 0x315E1FE, refcount: 1, rpc_result = 0x0, data_buffer = 0x451D3748, header = 0x8A772C8, data = 0x8A772E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57854, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D4, lo: 0x8A772E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 CF 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 97 00 00
Jun 21 11:19:43.751: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.751: SW1_SP: IPC: Message 4525926C timed out waiting for Ack
Jun 21 11:19:43.751: SW1_SP: IPC: MSG: ptr: 0x4525926C, flags: 0x24108, retries: 0, seq: 0x315E1FF, refcount: 1, rpc_result = 0x0, data_buffer = 0x50405568, header = 0x89DD1C8, data = 0x89DD1E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57855, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D5, lo: 0x89DD1E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 88 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 80 0A 06 0B 24 01 04 00 00 00 8A 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 02 00 00
Jun 21 11:19:43.751: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x4525926C, flags: 0x24108, retries: 0, seq: 0x315E1FF, refcount: 1, rpc_result = 0x0, data_buffer = 0x50405568, header = 0x89DD1C8, data = 0x89DD1E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57855, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D5, lo: 0x89DD1E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 88 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 80 0A 06 0B 24 01 04 00 00 00 8A 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 02 00 00
Jun 21 11:19:43.755: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.755: SW1_SP: IPC: Message 452512E0 timed out waiting for Ack
Jun 21 11:19:43.755: SW1_SP: IPC: MSG: ptr: 0x452512E0, flags: 0x24108, retries: 0, seq: 0x315E200, refcount: 1, rpc_result = 0x0, data_buffer = 0x451AF00C, header = 0x891CCC8, data = 0x891CCE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57856, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D6, lo: 0x891CCE8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 C0 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0B 00 00 00 00 00 00 04 B2 0A 06 9E 0B 01 04 00 00 00 C3 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 17 00 00
Jun 21 11:19:43.755: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x452512E0, flags: 0x24108, retries: 0, seq: 0x315E200, refcount: 1, rpc_result = 0x0, data_buffer = 0x451AF00C, header = 0x891CCC8, data = 0x891CCE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57856, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D6, lo: 0x891CCE8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 C0 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0B 00 00 00 00 00 00 04 B2 0A 06 9E 0B 01 04 00 00 00 C3 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 17 00 00
Jun 21 11:19:43.755: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.755: SW1_SP: IPC: Message 504A9C90 timed out waiting for Ack
Jun 21 11:19:43.755: SW1_SP: IPC: MSG: ptr: 0x504A9C90, flags: 0x24108, retries: 0, seq: 0x315E201, refcount: 1, rpc_result = 0x0, data_buffer = 0x45158028, header = 0x85E22C8, data = 0x85E22E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57857, sz: 736, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D7, lo: 0x85E22E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 02 D0 00 0D 00 15 02 CA 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 03 4E 0A 06 FE 4C 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 01 C2 00 00
Jun 21 11:19:43.755: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504A9C90, flags: 0x24108, retries: 0, seq: 0x315E201, refcount: 1, rpc_result = 0x0, data_buffer = 0x45158028, header = 0x85E22C8, data = 0x85E22E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57857, sz: 736, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D7, lo: 0x85E22E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 02 D0 00 0D 00 15 02 CA 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 03 4E 0A 06 FE 4C 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 01 C2 00 00
Jun 21 11:19:43.755: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:52.687: %CPU_MONITOR-SW1_SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 120 seconds [21/1]
Jun 21 11:20:22.687: %CPU_MONITOR-SW1_SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 150 seconds [21/1]
Jun 21 11:20:33.175: SW1_SP: IPC: Message 504F68E8 timed out waiting for Ack
Jun 21 11:20:33.175: SW1_SP: IPC: MSG: ptr: 0x504F68E8, flags: 0x34101, retries: 21, seq: 0x315AABD, refcount: 2, rpc_result = 0x0, data_buffer = 0x4517147C, header = 0x86D24C8, data = 0x86D24E8 || HDR: src: 0x10000, dst: 0x315000E, index: 1, seq: 43709, sz: 80, type: 1, flags: 0x1404, ext_flags: 0x0, hi: 0xA56AE85, lo: 0x86D24E8 || DATA: 00 00 00 15 00 00 00 00 00 00 07 D1 00 00 00 0A 00 00 00 0C 00 00 00 0F 00 00 00 05 00 00 00 0A 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Jun 21 11:20:33.175: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504F68E8, flags: 0x34101, retries: 21, seq: 0x315AABD, refcount: 2, rpc_result = 0x0, data_buffer = 0x4517147C, header = 0x86D24C8, data = 0x86D24E8 || HDR: src: 0x10000, dst: 0x315000E, index: 1, seq: 43709, sz: 80, type: 1, flags: 0x1404, ext_flags: 0x0, hi: 0xA56AE85, lo: 0x86D24E8 || DATA: 00 00 00 15 00 00 00 00 00 00 07 D1 00 00 00 0A 00 00 00 0C 00 00 00 0F 00 00 00 05 00 00 00 0A 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Jun 21 11:20:33.175: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:20:33.179: %C6K_PROCMIB-SW1_SP-3-IPC_TRANSMIT_FAIL: Failed to send process statistics update : error code = timeout
-Traceback= 40BF9E18 40BF9E68 40BFA070 40BFA2D4 408AAFE8 408AAFD4
Jun 21 11:20:52.687: %CPU_MONITOR-SW1_SP-3-TIMED_OUT: CPU_MONITOR messages have failed, resetting system [21/1]

Rajeev Sharma · ‎06-24-2014

Hey,

So it did crash and the name of file tells the date and time - bootflash:crashinfo_SP_20140621-192052

I suggest opening a TAC case and get trace backs decoded for root cause of supervisor crash.

HTH.

Regards,

RS.