9001 XR Interface Failure

vcldataservice · ‎06-30-2013

Hi Guys, i am new to XR IOS, just recently deployed some 9001 XR on our network, had a prblem recently where one devices interfaces loose traffic, all the neighbours go down as per the below log, i am trying to find the reason for this as it keeps happening on 5Hour intervals. Thanks

RP/0/RSP0/CPU0:Jun 30 03:25:08.437 : eem_ed_oir[190]: Messge received content : Event 1 NodeId: 0x828

RP/0/RSP0/CPU0:Jun 30 03:25:08.440 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB35: Bundle-Ether1.35: state Standby -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.442 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB32: Bundle-Ether1.32: state Active -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.443 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB31: Bundle-Ether1.31: state Standby -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.444 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB15: Bundle-Ether1.15: state Standby -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.447 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB30: Bundle-Ether1.307: state Active -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.448 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB34: Bundle-Ether1.34: state Active -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.449 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB33: Bundle-Ether1.33: state Standby -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.449 : invmgr[255]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/0/CPU0, state: BRINGDOWN

RP/0/RSP0/CPU0:Jun 30 03:25:08.450 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB60: Bundle-Ether1.360: state Standby -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.452 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB49: Bundle-Ether1.49: state Active -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.453 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB35: Bundle-Ether1.305: state Standby -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.455 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB12: Bundle-Ether1.312: state Active -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.455 : mpls_ldp[1043]: %ROUTING-LDP-5-GR : GR session 172.29.0.8:0 (instance 1) disconnected

RP/0/RSP0/CPU0:Jun 30 03:25:08.457 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB11: Bundle-Ether1.311: state Standby -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.457 : mpls_ldp[1043]: %ROUTING-LDP-5-NBR_CHANGE : Neighbor 172.29.0.8:0, DOWN (Interface state down)

RP/0/RSP0/CPU0:Jun 30 03:25:08.458 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB16: Bundle-Ether1.16: state Active -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.460 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB29: Bundle-Ether1.29: state Active -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.461 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB27: Bundle-Ether1.27: state Standby -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:08.463 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB79: Bundle-Ether1.709: state Standby -> Init

RP/0/RSP0/CPU0:Jun 30 03:25:38.635 : shelfmgr[395]: %PLATFORM-SHELFMGR_HAL-6-BOOT_REQ_RECEIVED : Boot Request from 0/0/CPU0, RomMon Version: 1.30

RP/0/RSP0/CPU0:Jun 30 03:25:38.637 : shelfmgr[395]: %PLATFORM-MBIMGR-7-IMAGE_VALIDATED : Remote location 0/0/CPU0: : MBI tftp:/disk0/asr9k-os-mbi-4.2.3.CSCue55886-1.0.0/lc/mbiasr9k-lc.vm validated

RP/0/RSP0/CPU0:Jun 30 03:25:38.640 : shelfmgr[395]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/0/CPU0 ASR9001-LC state:MBI-BOOTING

RP/0/RSP0/CPU0:Jun 30 03:26:56.660 : shelfmgr[395]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/0/CPU0 ASR9001-LC state:MBI-RUNNING

RP/0/RSP0/CPU0:Jun 30 03:27:13.194 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.11 Down - BGP Notification sent, hold time expired (VRF: default)

RP/0/RSP0/CPU0:Jun 30 03:27:19.774 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.6 Down - BGP Notification sent, hold time expired (VRF: default)

RP/0/RSP0/CPU0:Jun 30 03:27:40.594 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.1 Down - BGP Notification sent, hold time expired (VRF: default)

RP/0/RSP0/CPU0:Jun 30 03:27:40.806 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.2 Down - BGP Notification sent, hold time expired (VRF: default)

RP/0/RSP0/CPU0:Jun 30 03:27:43.563 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.8 Down - BGP Notification sent, hold time expired (VRF: default)

RP/0/RSP0/CPU0:Jun 30 03:27:45.142 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.12 Down - BGP Notification sent, hold time expired (VRF: default)

LC/0/0/CPU0:10: init[65540]: %OS-INIT-7-MBI_STARTED : total time 5.229 seconds

LC/0/0/CPU0:Jun 30 03:27:00.002 : insthelper[61]: %INSTALL-INSTHELPER-7-PKG_DOWNLOAD : MBI running; starting software download

LC/0/0/CPU0:Jun 30 03:27:41.274 : sysmgr[90]: %OS-SYSMGR-5-NOTICE : Card is COLD started

LC/0/0/CPU0:Jun 30 03:27:41.757 : init[65540]: %OS-INIT-7-INSTALL_READY : total time 51.189 seconds

LC/0/0/CPU0:Jun 30 03:27:42.383 : sysmgr[352]: %OS-SYSMGR-6-INFO : Backup system manager is ready

RP/0/RSP0/CPU0:Jun 30 03:27:51.664 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.5 Down - BGP Notification sent, hold time expired (VRF: default)

RP/0/RSP0/CPU0:Jun 30 03:27:55.337 : shelfmgr[395]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/0/0 A9K-MPA-20X1GE state:UNPOWERED

RP/0/RSP0/CPU0:Jun 30 03:27:55.449 : shelfmgr[395]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/0/0 A9K-MPA-20X1GE state:POWER UP

LC/0/0/CPU0:Jun 30 03:27:56.275 : lda_server[64]: %L2-SPA-5-STATE_CHANGE : SPA in bay 0 type A9K-MPA-20X1GE Initing

RP/0/RSP0/CPU0:Jun 30 03:27:57.309 : shelfmgr[395]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/0/0 A9K-MPA-20X1GE state:INITIALIZED

LC/0/0/CPU0:Jun 30 03:27:57.435 : kdscrape-lc[62]: KERNEL-DUMP: Copied LC1.130630-012501.e500 to /harddisk:/dumper

LC/0/0/CPU0:Jun 30 03:27:57.542 : kdscrape-lc[62]: KERNEL-DUMP: Copied LC1.130630-012501.crashinfo.by.p40x0mc to /harddisk:/dumper

RP/0/RSP0/CPU0:Jun 30 03:27:57.585 : invmgr[255]: %PLATFORM-INV-6-IF_OIRIN : xFP OIR: 0/0/0 port_num: 0 is inserted, state: 1

RP/0/RSP0/CPU0:Jun 30 03:27:59.193 : invmgr[255]: %PLATFORM-INV-6-IF_OIRIN : xFP OIR: 0/0/0 port_num: 1 is inserted, state: 1

RP/0/RSP0/CPU0:Jun 30 03:28:00.795 : invmgr[255]: %PLATFORM-INV-6-IF_OIRIN : xFP OIR: 0/0/0 port_num: 2 is inserted, state: 1

LC/0/0/CPU0:Jun 30 03:28:01.269 : pfm_node_lc[291]: %FABRIC-FIA-1-DATA_SB_SERDES_1_FAIL_0 : Set|fialc[139329]|Fabric Interface(0x103b000)|Data SB Serdes Link 1 Failure on FIA 0

LC/0/0/CPU0:Jun 30 03:28:01.271 : pfm_node_lc[291]: %FABRIC-FIA-1-DATA_SB_SERDES_0_FAIL_0 : Set|fialc[139329]|Fabric Interface(0x103b000)|Data SB Serdes Link 0 Failure on FIA 0

RP/0/RSP0/CPU0:Jun 30 03:28:01.974 : invmgr[255]: %PLATFORM-INV-6-IF_OIRIN : xFP OIR: 0/0/0 port_num: 3 is inserted, state: 1

RP/0/RSP0/CPU0:Jun 30 03:28:03.577 : invmgr[255]: %PLATFORM-INV-6-IF_OIRIN : xFP OIR: 0/0/0 port_num: 4 is inserted, state: 1

RP/0/RSP0/CPU0:Jun 30 03:28:05.181 : invmgr[255]: %PLATFORM-INV-6-IF_OIRIN : xFP OIR: 0/0/0 port_num: 5 is inserted, state: 1

RP/0/RSP0/CPU0:Jun 30 03:28:06.785 : invmgr[255]: %PLATFORM-INV-6-IF_OIRIN : xFP OIR: 0/0/0 port_num: 6 is inserted, state: 1

RP/0/RSP0/CPU0:Jun 30 03:28:08.386 : mpls_ldp[1043]: %ROUTING-LDP-5-GR : GR session 172.29.0.1:0 (instance 1) timed out

RP/0/RSP0/CPU0:Jun 30 03:28:08.389 : invmgr[255]: %PLATFORM-INV-6-IF_OIRIN : xFP OIR: 0/0/0 port_num: 7 is inserted, state: 1

RP/0/RSP0/CPU0:Jun 30 03:28:08.458 : mpls_ldp[1043]: %ROUTING-LDP-5-GR : GR session 172.29.0.8:0 (instance 1) timed out

LC/0/0/CPU0:Jun 30 03:28:09.559 : prm_server_ty[303]: %PLATFORM-QOSHAL-6-QOS_MODE_CHANGE : Line-card QoS mode set to child-shaping-enabled.

LC/0/0/CPU0:Jun 30 03:28:09.563 : prm_server_ty[303]: %PLATFORM-QOSHAL-6-QOS_MODE_CHANGE : Line-card QoS mode set to lowburst-disabled.

RP/0/RSP0/CPU0:Jun 30 03:28:09.993 : invmgr[255]: %PLATFORM-INV-6-IF_OIRIN : xFP OIR: 0/0/0 port_num: 8 is inserted, state: 1

LC/0/0/CPU0:Jun 30 03:28:10.011 : sysmgr[90]: %OS-SYSMGR-7-DEBUG : sysmgr_admin_plane_check:SYSMGR_PLANE_ADMIN Notification sent.

RP/0/RSP0/CPU0:Jun 30 03:28:11.180 : invmgr[255]: %PLATFORM-INV-6-IF_OIRIN : xFP OIR: 0/0/0 port_num: 9 is inserted, state: 1

LC/0/0/CPU0:Jun 30 03:28:13.005 : syslog_dev[88]: hsbi_fpd_agent[198]: FPD ltrace_file_name => fpd-agent/hsbi_fpd_agent

LC/0/0/CPU0:Jun 30 03:28:15.980 : rsi_agent[318]: %OS-RSI_AGENT-6-CARD_ROLE_CHANGE : Based on the card configuration/type, the AFI IPv4 role of the card has changed from Invalid to Customer Facing

LC/0/0/CPU0:Jun 30 03:28:15.984 : rsi_agent[318]: %OS-RSI_AGENT-6-CARD_ROLE_CHANGE : Based on the card configuration/type, the AFI IPv6 role of the card has changed from Invalid to Not Interested

RP/0/RSP0/CPU0:Jun 30 03:28:16.600 : invmgr[255]: %PLATFORM-INV-6-IF_OIRIN : xFP OIR: 0/0/0 port_num: 11 is inserted, state: 1

RP/0/RSP0/CPU0:Jun 30 03:28:33.471 : shelfmgr[395]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/0/0 A9K-MPA-20X1GE state:OK

RP/0/RSP0/CPU0:Jun 30 03:28:33.488 : invmgr[255]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/0/0, state: OK

RP/0/RSP0/CPU0:Jun 30 03:28:33.495 : invmgr[255]: %PLATFORM-INV-6-OIRIN : OIR: Node 0/0/0 inserted

RP/0/RSP0/CPU0:Jun 30 03:28:33.496 : eem_ed_oir[190]: Messge received content : Event 0 NodeId: 0x828

RP/0/RSP0/CPU0:Jun 30 03:28:33.498 : eem_ed_oir[190]: Messge received content : Event 0 NodeId: 0x828

LC/0/0/CPU0:Jun 30 03:28:34.330 : pfm_node_lc[291]: %FABRIC-FIA-1-DATA_SB_SERDES_1_FAIL_0 : Clear|fialc[139329]|Fabric Interface(0x103b000)|Data SB Serdes Link 1 Failure on FIA 0

LC/0/0/CPU0:Jun 30 03:28:34.331 : pfm_node_lc[291]: %FABRIC-FIA-1-DATA_SB_SERDES_0_FAIL_0 : Clear|fialc[139329]|Fabric Interface(0x103b000)|Data SB Serdes Link 0 Failure on FIA 0

LC/0/0/CPU0:Jun 30 03:28:44.631 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface TenGigE0/0/2/3, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:44.632 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface TenGigE0/0/2/2, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:44.633 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface TenGigE0/0/2/1, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:44.633 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface TenGigE0/0/2/0, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:44.637 : cfgmgr-lc[134]: %MGBL-CONFIG-6-OIR_RESTORE : Configuration for node '0/0/2' has been restored.

LC/0/0/CPU0:Jun 30 03:28:47.602 : rsi_agent[318]: %OS-RSI_AGENT-6-CARD_ROLE_CHANGE : Based on the card configuration/type, the AFI IPv4 role of the card has changed from Customer Facing to Core Facing

LC/0/0/CPU0:Jun 30 03:28:47.603 : rsi_agent[318]: %OS-RSI_AGENT-6-CARD_ROLE_CHANGE : Based on the card configuration/type, the AFI IPv6 role of the card has changed from Not Interested to Core Facing

RP/0/RSP0/CPU0:Jun 30 03:28:51.005 : shelfmgr[395]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/0/CPU0 ASR9001-LC state:IOS XR RUN

RP/0/RSP0/CPU0:Jun 30 03:28:51.010 : invmgr[255]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/0/CPU0, state: IOS XR RUN

LC/0/0/CPU0:Jun 30 03:28:51.022 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/19, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:51.022 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/18, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:51.022 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/17, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:51.023 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/16, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:51.023 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/15, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:51.024 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/14, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:51.024 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/13, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:51.024 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/12, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:51.025 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/11, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:51.025 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/10, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:51.025 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/9, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:51.026 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/8, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:51.026 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/7, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:51.030 : ifmgr[201]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/6, changed state to Down

LC/0/0/CPU0:Jun 30 03:28:51.031 : ifmgr[201]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/6, changed state to Down

LC/0/0/CPU0:Jun 30 03:28:51.031 : ifmgr[201]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/5, changed state to Administratively Down

LC/0/0/CPU0:Jun 30 03:28:51.032 : ifmgr[201]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/4, changed state to Down

LC/0/0/CPU0:Jun 30 03:28:51.032 : ifmgr[201]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/4, changed state to Down

LC/0/0/CPU0:Jun 30 03:28:51.032 : ifmgr[201]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/3, changed state to Down

LC/0/0/CPU0:Jun 30 03:28:51.032 : ifmgr[201]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/3, changed state to Down

LC/0/0/CPU0:Jun 30 03:28:51.033 : ifmgr[201]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/2, changed state to Down

LC/0/0/CPU0:Jun 30 03:28:51.034 : ifmgr[201]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/2, changed state to Down

LC/0/0/CPU0:Jun 30 03:28:51.034 : ifmgr[201]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/1, changed state to Down

LC/0/0/CPU0:Jun 30 03:28:51.034 : ifmgr[201]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/1, changed state to Down

LC/0/0/CPU0:Jun 30 03:28:51.035 : ifmgr[201]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/0, changed state to Down

LC/0/0/CPU0:Jun 30 03:28:51.035 : ifmgr[201]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/0, changed state to Down

LC/0/0/CPU0:Jun 30 03:28:51.043 : cfgmgr-lc[134]: %MGBL-CONFIG-6-OIR_RESTORE : Configuration for node '0/0/0' has been restored.

LC/0/0/CPU0:Jun 30 03:28:57.029 : rsi_agent[318]: %OS-RSI_AGENT-6-CARD_ROLE_CHANGE : Based on the card configuration/type, the AFI IPv4 role of the card has changed from Core Facing to Standard

LC/0/0/CPU0:Jun 30 03:28:57.030 : rsi_agent[318]: %OS-RSI_AGENT-6-CARD_ROLE_CHANGE : Based on the card configuration/type, the AFI IPv6 role of the card has changed from Core Facing to Standard

LC/0/0/CPU0:Jun 30 03:28:58.163 : ifmgr[201]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/6, changed state to Up

LC/0/0/CPU0:Jun 30 03:28:58.165 : ifmgr[201]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/6, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.171 : BM-DISTRIB[1148]: %L2-BM-6-ACTIVE : GigabitEthernet0/0/0/6 is Active as part of Bundle-Ether1

RP/0/RSP0/CPU0:Jun 30 03:29:00.171 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.174 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.178 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.35, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.178 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.32, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.179 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.31, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.179 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.15, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.180 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.307, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.180 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.34, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.181 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.33, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.181 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.360, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.182 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.49, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.182 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.305, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.183 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.40, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.183 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.312, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.183 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.311, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.184 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.16, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.184 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.29, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.184 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.27, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.185 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether1.709, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.192 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.35, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.192 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.32, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.192 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.31, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.192 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.15, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.193 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.307, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.193 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.34, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.193 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.33, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.194 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.360, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.194 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.49, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.194 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.305, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.195 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.40, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.195 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.312, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.195 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.311, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.195 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.16, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.195 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.29, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.196 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.27, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:00.196 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether1.709, changed state to Up

LC/0/0/CPU0:Jun 30 03:29:03.825 : ifmgr[201]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/0, changed state to Up

LC/0/0/CPU0:Jun 30 03:29:03.827 : ifmgr[201]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/0, changed state to Up

LC/0/0/CPU0:Jun 30 03:29:03.840 : ifmgr[201]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/1, changed state to Up

LC/0/0/CPU0:Jun 30 03:29:03.841 : ifmgr[201]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/1, changed state to Up

LC/0/0/CPU0:Jun 30 03:29:03.855 : ifmgr[201]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/2, changed state to Up

LC/0/0/CPU0:Jun 30 03:29:03.857 : ifmgr[201]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/2, changed state to Up

LC/0/0/CPU0:Jun 30 03:29:04.016 : ifmgr[201]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/4, changed state to Up

LC/0/0/CPU0:Jun 30 03:29:04.018 : ifmgr[201]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/4, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:04.022 : BM-DISTRIB[1148]: %L2-BM-6-ACTIVE : GigabitEthernet0/0/0/4 is Active as part of Bundle-Ether1

RP/0/RSP0/CPU0:Jun 30 03:29:04.349 : ospf[1011]: %ROUTING-OSPF-5-ADJCHG : Process 200, Nbr 172.29.0.1 on GigabitEthernet0/0/0/0 in area 0 from LOADING to FULL, Loading Done,vrf default vrfid 0x60000000

LC/0/0/CPU0:Jun 30 03:29:05.811 : pfm_node_lc[291]: %PLATFORM-SFP-3-HIGH_RX_POWER_WARNING : Set|envmon_lc[143443]|SFP(0x1029001)|Port_0/01

LC/0/0/CPU0:Jun 30 03:29:05.811 : pfm_node_lc[291]: %PLATFORM-SFP-3-HIGH_RX_POWER_WARNING : Set|envmon_lc[143443]|SFP(0x1029002)|Port_0/02

RP/0/RSP0/CPU0:Jun 30 03:29:05.849 : BM-DISTRIB[1148]: %L2-BM-6-ACTIVE : GigabitEthernet0/0/0/1 is Active as part of Bundle-Ether2

RP/0/RSP0/CPU0:Jun 30 03:29:05.849 : BM-DISTRIB[1148]: %L2-BM-6-ACTIVE : GigabitEthernet0/0/0/2 is Active as part of Bundle-Ether2

RP/0/RSP0/CPU0:Jun 30 03:29:05.850 : ifmgr[243]: %PKT_INFRA-LINK-3-UPDOWN : Interface Bundle-Ether2, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:05.852 : ifmgr[243]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface Bundle-Ether2, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:05.877 : ospf[1011]: %ROUTING-OSPF-5-ADJCHG : Process 200, Nbr 172.29.0.8 on Bundle-Ether2 in area 0 from LOADING to FULL, Loading Done,vrf default vrfid 0x60000000

RP/0/RSP0/CPU0:Jun 30 03:29:05.928 : mpls_ldp[1043]: %ROUTING-LDP-5-NBR_CHANGE : Neighbor 172.29.0.8:0, UP

RP/0/RSP0/CPU0:Jun 30 03:29:06.730 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.1 Up (VRF: default)

LC/0/0/CPU0:Jun 30 03:29:06.914 : ifmgr[201]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/3, changed state to Up

LC/0/0/CPU0:Jun 30 03:29:06.915 : ifmgr[201]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/3, changed state to Up

RP/0/RSP0/CPU0:Jun 30 03:29:06.984 : mpls_ldp[1043]: %ROUTING-LDP-5-NBR_CHANGE : Neighbor 172.29.0.11:0, UP

RP/0/RSP0/CPU0:Jun 30 03:29:07.079 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.12 Up (VRF: default)

RP/0/RSP0/CPU0:Jun 30 03:29:07.894 : mpls_ldp[1043]: %ROUTING-LDP-5-NBR_CHANGE : Neighbor 172.29.0.1:0, UP

RP/0/RSP0/CPU0:Jun 30 03:29:08.134 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.2 Up (VRF: default)

RP/0/RSP0/CPU0:Jun 30 03:29:08.924 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.11 Up (VRF: default)

RP/0/RSP0/CPU0:Jun 30 03:29:09.511 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.5 Up (VRF: default)

RP/0/RSP0/CPU0:Jun 30 03:29:10.033 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.6 Up (VRF: default)

RP/0/RSP0/CPU0:Jun 30 03:29:12.443 : ospf[1011]: %ROUTING-OSPF-5-ADJCHG : Process 200, Nbr 172.29.0.11 on GigabitEthernet0/0/0/3 in area 0 from LOADING to FULL, Loading Done,vrf default vrfid 0x60000000

RP/0/RSP0/CPU0:Jun 30 03:29:13.192 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.8 Up (VRF: default)

RP/0/RSP0/CPU0:Jun 30 03:29:21.056 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB32: Bundle-Ether1.32: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.056 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB32: Bundle-Ether1.32: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.063 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB29: Bundle-Ether1.29: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.064 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB29: Bundle-Ether1.29: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.074 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB79: Bundle-Ether1.709: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.074 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB79: Bundle-Ether1.709: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.105 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB34: Bundle-Ether1.34: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.105 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB34: Bundle-Ether1.34: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.123 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB12: Bundle-Ether1.312: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.123 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB12: Bundle-Ether1.312: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.123 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB16: Bundle-Ether1.16: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.124 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB16: Bundle-Ether1.16: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.139 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB31: Bundle-Ether1.31: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.139 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB31: Bundle-Ether1.31: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.145 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB30: Bundle-Ether1.307: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.146 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB30: Bundle-Ether1.307: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.152 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB27: Bundle-Ether1.27: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.153 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB27: Bundle-Ether1.27: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.172 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB49: Bundle-Ether1.49: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.172 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB49: Bundle-Ether1.49: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.172 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB11: Bundle-Ether1.311: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.173 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB11: Bundle-Ether1.311: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.180 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB35: Bundle-Ether1.35: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.180 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB35: Bundle-Ether1.35: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.187 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB35: Bundle-Ether1.305: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.187 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB35: Bundle-Ether1.305: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.196 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB33: Bundle-Ether1.33: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.196 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB33: Bundle-Ether1.33: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.201 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB15: Bundle-Ether1.15: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.202 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB15: Bundle-Ether1.15: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:21.212 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB60: Bundle-Ether1.360: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:21.212 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB60: Bundle-Ether1.360: state Standby -> Active

RP/0/RSP0/CPU0:Jun 30 03:29:37.354 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB35: Bundle-Ether1.305: state Active -> Speak

RP/0/RSP0/CPU0:Jun 30 03:29:37.452 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB31: Bundle-Ether1.31: state Active -> Speak

RP/0/RSP0/CPU0:Jun 30 03:29:37.606 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB79: Bundle-Ether1.709: state Active -> Speak

RP/0/RSP0/CPU0:Jun 30 03:29:37.631 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB33: Bundle-Ether1.33: state Active -> Speak

RP/0/RSP0/CPU0:Jun 30 03:29:37.727 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB27: Bundle-Ether1.27: state Active -> Speak

RP/0/RSP0/CPU0:Jun 30 03:29:37.815 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB11: Bundle-Ether1.311: state Active -> Speak

RP/0/RSP0/CPU0:Jun 30 03:29:37.862 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB60: Bundle-Ether1.360: state Active -> Speak

RP/0/RSP0/CPU0:Jun 30 03:29:38.037 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB15: Bundle-Ether1.15: state Active -> Speak

RP/0/RSP0/CPU0:Jun 30 03:29:38.349 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB35: Bundle-Ether1.35: state Active -> Speak

RP/0/RSP0/CPU0:Jun 30 03:29:47.355 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB35: Bundle-Ether1.305: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:47.453 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB31: Bundle-Ether1.31: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:47.607 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB79: Bundle-Ether1.709: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:47.632 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB33: Bundle-Ether1.33: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:47.728 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB27: Bundle-Ether1.27: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:47.815 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB11: Bundle-Ether1.311: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:47.863 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB60: Bundle-Ether1.360: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:48.038 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB15: Bundle-Ether1.15: state Speak -> Standby

RP/0/RSP0/CPU0:Jun 30 03:29:48.350 : hsrp[1085]: %IP-STANDBY-6-INFO_STATECHANGE : SB35: Bundle-Ether1.35: state Speak -> Standby

skondala · ‎07-02-2013

Looks like the whole card went down..

LC/0/0/CPU0:Jun 30 03:27:41.274 : sysmgr[90]: %OS-SYSMGR-5-NOTICE : Card is COLD started

LC/0/0/CPU0:Jun 30 03:27:41.757 : init[65540]: %OS-INIT-7-INSTALL_READY : total time 51.189 seconds

LC/0/0/CPU0:Jun 30 03:27:42.383 : sysmgr[352]: %OS-SYSMGR-6-INFO : Backup system manager is ready

RP/0/RSP0/CPU0:Jun 30 03:27:51.664 : bgp[1048]: %ROUTING-BGP-5-ADJCHANGE : neighbor 172.29.0.5 Down - BGP Notification sent, hold time expired (VRF: default)

RP/0/RSP0/CPU0:Jun 30 03:27:55.337 : shelfmgr[395]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/0/0 A9K-MPA-20X1GE state:UNPOWERED

RP/0/RSP0/CPU0:Jun 30 03:27:55.449 : shelfmgr[395]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/0/0 A9K-MPA-20X1GE state:POWER UP

LC/0/0/CPU0 30 03:27:56.275 : lda_server[64]: %L2-SPA-5-STATE_CHANGE : SPA in bay 0 type A9K-MPA-20X1GE Initing

RP/0/RSP0/CPU0:Jun 30 03:27:57.309 : shelfmgr[395]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/0/0 A9K-MPA-20X1GE state:INITIALIZED

LC/0/0/CPU0:Jun 30 03:27:57.435 : kdscrape-lc[62]: KERNEL-DUMP: Copied LC1.130630-012501.e500 to /harddisk:/dumper

LC/0/0/CPU0:Jun 30 03:27:57.542 : kdscrape-lc[62]: KERNEL-DUMP: Copied LC1.130630-012501.crashinfo.by.p40x0mc to

get sh reboot history location o/x/cpu0 and check why the card went down. also try getting the crash files from hardisk to check why the LC went down

Reagrds, Satish.

vcldataservice · ‎07-03-2013

Hi Please have a look at this:

RP/0/RSP0/CPU0:HOSTNAME#sh reboot history location 0/0/CPU0

Wed Jul 3 22:49:17.379 gmt

No Time Cause Code Reason

--------------------------------------------------------------------------------

01 Tue Jul 2 14:56:09 2013 0x2c000015 Cause: Fatal Eror Condition

Process: p40x0mc

02 Tue Jul 2 14:56:09 2013 0x0c000015 Cause: Fatal Eror Condition

Process: p40x0mc

03 Tue Jul 2 07:59:57 2013 0x2c000015 Cause: Fatal Eror Condition

Process: p40x0mc

04 Tue Jul 2 07:59:57 2013 0x0c000015 Cause: Fatal Eror Condition

Process: p40x0mc

05 Sun Jun 30 12:20:38 2013 0x2c000015 Cause: Fatal Eror Condition

Process: p40x0mc

06 Sun Jun 30 12:20:38 2013 0x0c000015 Cause: Fatal Eror Condition

Process: p40x0mc

07 Sun Jun 30 03:24:57 2013 0x2c000015 Cause: Fatal Eror Condition

Process: p40x0mc

08 Sun Jun 30 03:24:57 2013 0x0c000015 Cause: Fatal Eror Condition

Process: p40x0mc

vcldataservice · ‎07-03-2013

Hi Also the Crash dump looks like this:

RP/0/RSP0/CPU0:HOSTNAME#more LC1.130630-012501.crashinfo.by.p40x0mc

LC/0/0/CPU0:Jun 30 03:24:57.075 : p40x0mc[74]: ====== L3 CACHE ERROR #1 ======

LC/0/0/CPU0:Jun 30 03:24:57.080 : p40x0mc[74]: CPC Registers: OFF CPC1 CPC2

LC/0/0/CPU0:Jun 30 03:24:57.084 : p40x0mc[74]: CPCERRDET* e40 0x80000008 0x00000000

LC/0/0/CPU0:Jun 30 03:24:57.088 : p40x0mc[74]: CPCCAPTECC e28 0xc9000009 0x00000000

LC/0/0/CPU0:Jun 30 03:24:57.093 : p40x0mc[74]: CPCERRINTEN e48 0x000000ec 0x000000ec

LC/0/0/CPU0:Jun 30 03:24:57.097 : p40x0mc[74]: CPCERRATTR e4c 0x7082011d 0x00000000

LC/0/0/CPU0:Jun 30 03:24:57.102 : p40x0mc[74]: CPCERRADDR e54 0x0fef1210 0x00000000

LC/0/0/CPU0:Jun 30 03:24:57.106 : p40x0mc[74]: CPCCSR0 000 0xc0000000 0xc0000000

LC/0/0/CPU0:Jun 30 03:24:57.110 : p40x0mc[74]: CPCCFG0 008 0x50b7c010 0x50b7c010

LC/0/0/CPU0:Jun 30 03:24:57.115 : p40x0mc[74]: CPCHDBCR0 f00 0x015e0000 0x015e0000

LC/0/0/CPU0:Jun 30 03:24:57.119 : p40x0mc[74]: CPCCAPTDTHI e20 0x01001001 0x00000000

LC/0/0/CPU0:Jun 30 03:24:57.124 : p40x0mc[74]: CPCCAPTDTLO e24 0x79d54656 0x00000000

LC/0/0/CPU0:Jun 30 03:24:57.128 : p40x0mc[74]: CPCPIR0x0 200 0xffffffff 0xffffffff

LC/0/0/CPU0:Jun 30 03:24:57.132 : p40x0mc[74]: CPCPAR0x0 208 0x000003fb 0x000003fb

LC/0/0/CPU0:Jun 30 03:24:57.137 : p40x0mc[74]: CPCPWR0x0 20c 0xffffffff 0xffffffff

LC/0/0/CPU0:Jun 30 03:24:57.140 : p40x0mc[74]: ====== DDR CONTROLLER ERROR #1 ======

LC/0/0/CPU0:Jun 30 03:24:57.144 : p40x0mc[74]: DDR Registers: OFF DDR1 DDR2

LC/0/0/CPU0:Jun 30 03:24:57.149 : p40x0mc[74]: ERR_DETECT* e40 0x80000008 0x00000000

LC/0/0/CPU0:Jun 30 03:24:57.153 : p40x0mc[74]: ERR_INT_EN e48 0x0000018d 0x0000018d

LC/0/0/CPU0:Jun 30 03:24:57.158 : p40x0mc[74]: CAPTURE_ATTRIBUTES e4c 0x10832001 0x00000000

LC/0/0/CPU0:Jun 30 03:24:57.162 : p40x0mc[74]: CAPTURE_ADDRESS e50 0x0fef1200 0x00000000

LC/0/0/CPU0:Jun 30 03:24:57.167 : p40x0mc[74]: CAPTURE_DATA_HI e20 0x03001001 0x00000000

LC/0/0/CPU0:Jun 30 03:24:57.171 : p40x0mc[74]: CAPTURE_DATA_LO e24 0x77a7f528 0x00000000

LC/0/0/CPU0:Jun 30 03:24:57.175 : p40x0mc[74]: CAPTURE_DATA_ECC e28 0x21212121 0x00000000

LC/0/0/CPU0:Jun 30 03:24:57.317 : p40x0mc[74]: EISR0: 0x00000110 [DDR1|CPC1]

LC/0/0/CPU0:Jun 30 03:24:57.321 : p40x0mc[74]: DDR1 ERR_DETECT : 0x80000008 [MME|MBE] Count = 1

LC/0/0/CPU0:Jun 30 03:24:57.327 : p40x0mc[74]: CPC1 CPCERRDET : 0x80000008 [MULLERR|DMBECCERR] Count = 1

LC/0/0/CPU0:Jun 30 03:24:57.330 : p40x0mc[74]: Shutting down node due to fatal error

LC/0/0/CPU0:Jun 30 03:24:59.343 : p40x0mc[74]: reboot_internal: Incomplete graceful reboot cleanup (Connection timed out)

LC/0/0/CPU0:Jun 30 03:24:59.343 : p40x0mc[74]: Sun Jun 30 03:24:57 2013:sync start

LC/0/0/CPU0:Jun 30 03:24:59.343 : p40x0mc[74]: Sun Jun 30 03:24:57 2013:sync end

LC/0/0/CPU0:Jun 30 03:24:59.343 : p40x0mc[74]: Sun Jun 30 03:24:57 2013:platform_reboot_op start

RP/0/RSP0/CPU0:HOSTNAME#more LC1.130630-102042.crashinfo.by.p40x0mc

LC/0/0/CPU0:Jun 30 01:27:00.002 : insthelper[61]: %INSTALL-INSTHELPER-7-PKG_DOWNLOAD : MBI running; starting software download

LC/0/0/CPU0:Jun 30 01:27:41.274 : sysmgr[90]: %OS-SYSMGR-5-NOTICE : Card is COLD started

LC/0/0/CPU0:Jun 30 01:27:41.757 : init[65540]: %OS-INIT-7-INSTALL_READY : total time 51.189 seconds

LC/0/0/CPU0:Jun 30 01:27:42.383 : sysmgr[352]: %OS-SYSMGR-6-INFO : Backup system manager is ready

LC/0/0/CPU0:Jun 30 01:27:56.275 : lda_server[64]: %L2-SPA-5-STATE_CHANGE : SPA in bay 0 type A9K-MPA-20X1GE Initing

LC/0/0/CPU0:Jun 30 01:27:57.435 : kdscrape-lc[62]: KERNEL-DUMP: Copied LC1.130630-012501.e500 to /harddisk:/dumper

LC/0/0/CPU0:Jun 30 01:27:57.542 : kdscrape-lc[62]: KERNEL-DUMP: Copied LC1.130630-012501.crashinfo.by.p40x0mc to /harddisk:/dumper

LC/0/0/CPU0:Jun 30 01:28:01.269 : pfm_node_lc[291]: %FABRIC-FIA-1-DATA_SB_SERDES_1_FAIL_0 : Set|fialc[139329]|Fabric Interface(0x103b000)|Data SB Serdes Link 1 Failure on FIA 0

LC/0/0/CPU0:Jun 30 01:28:01.271 : pfm_node_lc[291]: %FABRIC-FIA-1-DATA_SB_SERDES_0_FAIL_0 : Set|fialc[139329]|Fabric Interface(0x103b000)|Data SB Serdes Link 0 Failure on FIA 0

LC/0/0/CPU0:Jun 30 01:28:09.559 : prm_server_ty[303]: %PLATFORM-QOSHAL-6-QOS_MODE_CHANGE : Line-card QoS mode set to child-shaping-enabled.

LC/0/0/CPU0:Jun 30 01:28:09.563 : prm_server_ty[303]: %PLATFORM-QOSHAL-6-QOS_MODE_CHANGE : Line-card QoS mode set to lowburst-disabled.

LC/0/0/CPU0:Jun 30 01:28:10.011 : sysmgr[90]: %OS-SYSMGR-7-DEBUG : sysmgr_admin_plane_check:SYSMGR_PLANE_ADMIN Notification sent.

LC/0/0/CPU0:Jun 30 01:28:13.005 : syslog_dev[88]: hsbi_fpd_agent[198]: FPD ltrace_file_name => fpd-agent/hsbi_fpd_agent