cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
797
Views
0
Helpful
9
Replies

many interfaces down on CRS-1 and we haven't find DWDM problem

fly
Level 2
Level 2

LC/1/1/CPU0:Aug 24 11:00:35.617 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/1/5/0, changed state to Down
LC/1/1/CPU0:Aug 24 11:00:35.639 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/1/5/0, changed state to Up
LC/1/1/CPU0:Aug 24 11:00:35.643 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/1/5/0, changed state to Down
LC/1/1/CPU0:Aug 24 11:00:35.652 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/1/5/0, changed state to Up
LC/1/1/CPU0:Aug 24 11:00:35.664 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/1/5/0, changed state to Up
LC/1/6/CPU0:Aug 24 11:00:36.364 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/6/3/0, changed state to Up
LC/1/6/CPU0:Aug 24 11:00:36.366 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/6/3/0, changed state to Up
LC/1/5/CPU0:Aug 24 11:00:39.258 : spa_xge_v2(2)[301]: %L2-ether_spa_plugin-6-LINK_EVENT : Interface TenGige 1/5/2/0, Rx Fault
LC/1/5/CPU0:Aug 24 11:00:39.258 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/5/2/0, changed state to Down
LC/1/5/CPU0:Aug 24 11:00:39.260 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/5/2/0, changed state to Down
LC/1/5/CPU0:Aug 24 11:00:39.273 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/5/2/0, changed state to Up
LC/1/5/CPU0:Aug 24 11:00:39.276 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/5/2/0, changed state to Up
LC/1/0/CPU0:Aug 24 11:00:39.960 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/0/3/0, changed state to Down
LC/1/0/CPU0:Aug 24 11:00:39.961 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/0/3/0, changed state to Down
LC/1/0/CPU0:Aug 24 11:00:39.968 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/0/3/0, changed state to Up
LC/1/0/CPU0:Aug 24 11:00:39.976 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/0/3/0, changed state to Up
LC/1/9/CPU0:Aug 24 11:00:43.504 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/9/2/0, changed state to Down
LC/1/9/CPU0:Aug 24 11:00:43.505 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/9/2/0, changed state to Down
LC/1/9/CPU0:Aug 24 11:00:43.512 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/9/2/0, changed state to Up
LC/1/9/CPU0:Aug 24 11:00:43.520 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/9/2/0, changed state to Up
LC/1/6/CPU0:Aug 24 11:00:47.726 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/6/5/0, changed state to Down
LC/1/6/CPU0:Aug 24 11:00:47.726 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/6/5/0, changed state to Down
LC/1/6/CPU0:Aug 24 11:00:47.734 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/6/5/0, changed state to Up
LC/1/6/CPU0:Aug 24 11:00:47.744 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/6/5/0, changed state to Up
LC/0/10/CPU0:Aug 24 11:00:48.240 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE0/10/5/0, changed state to Down
LC/0/10/CPU0:Aug 24 11:00:48.241 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE0/10/5/0, changed state to Down
LC/0/10/CPU0:Aug 24 11:00:48.248 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE0/10/5/0, changed state to Up
LC/0/10/CPU0:Aug 24 11:00:48.257 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE0/10/5/0, changed state to Up
LC/0/10/CPU0:Aug 24 11:00:48.367 : spa_xge_v2(5)[302]: %L2-ether_spa_plugin-6-LINK_EVENT : Interface TenGige 0/10/5/0, Rx Fault
LC/0/10/CPU0:Aug 24 11:00:48.367 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE0/10/5/0, changed state to Down
LC/0/10/CPU0:Aug 24 11:00:48.369 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE0/10/5/0, changed state to Down
LC/0/10/CPU0:Aug 24 11:00:48.381 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE0/10/5/0, changed state to Up
LC/0/10/CPU0:Aug 24 11:00:48.385 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE0/10/5/0, changed state to Up
LC/0/11/CPU0:Aug 24 11:00:50.238 : spa_xge_v2(5)[301]: %L2-ether_spa_plugin-6-LINK_EVENT : Interface TenGige 0/11/5/0, Rx Fault
 --More--           LC/0/11/CPU0:Aug 24 11:00:50.239 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE0/11/5/0, changed state to Down
LC/0/11/CPU0:Aug 24 11:00:50.240 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE0/11/5/0, changed state to Down
LC/0/11/CPU0:Aug 24 11:00:50.253 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE0/11/5/0, changed state to Up
LC/0/11/CPU0:Aug 24 11:00:50.257 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE0/11/5/0, changed state to Up
LC/0/11/CPU0:Aug 24 11:00:50.375 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE0/11/5/0, changed state to Down
LC/0/11/CPU0:Aug 24 11:00:50.376 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE0/11/5/0, changed state to Down
LC/0/11/CPU0:Aug 24 11:00:50.385 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE0/11/5/0, changed state to Up
LC/0/11/CPU0:Aug 24 11:00:50.396 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE0/11/5/0, changed state to Up
LC/1/3/CPU0:Aug 24 11:00:51.118 : spa_xge_v2(5)[303]: %L2-ether_spa_plugin-6-LINK_EVENT : Interface TenGige 1/3/5/0, Rx Fault
LC/1/3/CPU0:Aug 24 11:00:51.119 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/3/5/0, changed state to Down
LC/1/3/CPU0:Aug 24 11:00:51.120 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/3/5/0, changed state to Down
LC/1/3/CPU0:Aug 24 11:00:51.133 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/3/5/0, changed state to Up
LC/1/3/CPU0:Aug 24 11:00:51.136 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/3/5/0, changed state to Up
LC/1/6/CPU0:Aug 24 11:00:52.262 : spa_xge_v2(3)[301]: %L2-ether_spa_plugin-6-LINK_EVENT : Interface TenGige 1/6/3/0, Rx Fault
LC/1/6/CPU0:Aug 24 11:00:52.262 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/6/3/0, changed state to Down
LC/1/6/CPU0:Aug 24 11:00:52.264 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/6/3/0, changed state to Down
LC/1/6/CPU0:Aug 24 11:00:52.276 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/6/3/0, changed state to Up
LC/1/6/CPU0:Aug 24 11:00:52.280 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/6/3/0, changed state to Up
LC/1/1/CPU0:Aug 24 11:00:55.927 : spa_xge_v2(5)[303]: %L2-ether_spa_plugin-6-LINK_EVENT : Interface TenGige 1/1/5/0, Rx Fault
LC/1/1/CPU0:Aug 24 11:00:55.927 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/1/5/0, changed state to Down
LC/1/1/CPU0:Aug 24 11:00:55.929 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/1/5/0, changed state to Down
RP/1/RP0/CPU0:Aug 24 11:00:55.936 : ospf[1009]: %ROUTING-OSPF-5-ADJCHG : Process 100, Nbr 122.192.52.1 on TenGigE1/1/5/0 in area 0 from 2WAY to DOWN, Neighbor Down: interface down or detached,vrf default vrfid 0x60000000
LC/1/1/CPU0:Aug 24 11:00:55.943 : ifmgr[177]: %PKT_INFRA-LINK-3-UPDOWN : Interface TenGigE1/1/5/0, changed state to Up
LC/1/1/CPU0:Aug 24 11:00:55.947 : ifmgr[177]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface TenGigE1/1/5/0, changed state to Up
RP/1/RP0/CPU0:Aug 24 11:01:37.720 : ospf[1009]: %ROUTING-OSPF-5-ADJCHG : Process 100, Nbr 122.192.52.1 on TenGigE1/1/5/0 in a

9 Replies 9

osman
Cisco Employee
Cisco Employee

Please open a TAC case for us so that we can troubleshoot this. Rx fault indicates some issue on the remote end so it might be a good idea to identify if there might be any transport issues/L1 problems on the link.

Hi,osman,

   Thank you!

   How about Tx Fault.

   I saw few Tx Fault in the log. but just only few times. does that mean XFP hardware problem?

Best

Tom

osman
Cisco Employee
Cisco Employee

Hi Tom

 

Can you please share the complete syslog for this event? I can review to see if there might be any correlation. Best bet is to attach the info to a TAC SR.

Hi,osman,

    Thank you!

    I check show controllers tenGigE 1/6/3/0 phy

   it shows Link Stage down, but i can't understand output like below?

   I attach log file ,

    thank you!

Tom

Link flapped at Mon Aug 24 11:00:52 2015
 
 Link State changed to down 
 1.1 FPGA Address : 0x60a84204    Value : 0x12070000
 1.2 FPGA Address : 0x60a84204    Value : 0x12070000
 2.1 FPGA Address : 0x60a84200    Value : 0xe
 2.2 FPGA Address : 0x60a84200    Value : 0x8
 3.1. Serdes PCS status Register1 (0x230001): 0x      82
 3.2. Serdes PCS status Register1 (0x230001): 0x      82
 4.1. Serdes PCS status Register2 (0x230008): 0x    8405
 4.2. Serdes PCS status Register2 (0x230008): 0x    8405
 5.1.Frequency status status Register (0x238126): 0x    a002 
 5.2.Frequency status status Register (0x238126): 0x    a002 
 6.1.10G Serdes status Register (0x238127): 0x      20
 6.2.10G Serdes status Register (0x238127): 0x       0
 7.1. Reg (0x238300): 0x    2301
 7.2. Reg (0x238300): 0x    2301
 8.1. Reg (0x23fe26): 0x    99d6
 8.2. Reg (0x23fe26): 0x    99d6
 9.1. PHY XGXS status 1 reg(0x240001) : 0x      86
 9.2. PHY XGXS status 1 reg(0x240001) : 0x      86
 10.1. PHY XGXS status 2 reg (0x240008): 0x    8400
 10.2. PHY XGXS status 2 reg (0x240008): 0x    8400
 11.1.PHY XGXS Lane status reg (0x240018) : 0x    1c0f 
 11.2.PHY XGXS Lane status reg (0x240018) : 0x    1c0f 
 12.1. reg (0x248009) : 0x     400 
 12.2. reg (0x248009) : 0x     400 
 13.1. reg (0x230020) : 0x       4 
 13.2. reg (0x230020) : 0x       4 
 14.1. reg (0x230021) : 0x    4000
 14.2. reg (0x230021) : 0x       0
 15.1. reg (0x23820A) : 0x     400
 15.2. reg (0x23820A) : 0x     400

 Link flapped at Mon Aug 24 11:00:52 2015

this problem not happened once, It happened for serval months, sometimes few interface were flapping , but this time many interfaces flapping at same time. including different slot, and some interface are connected directly by fiber, some inefaces are connected by DWDM

it confuse me.

Hi Osman

   we check line in customer site, found all optical line connected to a device, and found problem happened on this device.

    thank you!

   but I have a question, from XFP mas specification, Tx fault Rx fault signal come from transceiver, IN nr there 3 signal ,  laser fault, Rx lock , TX lock,  I don't understand what is Tx fault means, why outside optical line quality can make crs-1 transceiver generate Tx fault

   Thank you

Tom

osman
Cisco Employee
Cisco Employee

Hi Tom

We expect to see a remote fault in case of issues on the remote end. There is an option to print "Tx Fault" instead of "Detected Remote Fault" in the syslog but it has to be configured using a CLI using hw-module logging CLI.

Detected Remote Fault is the correct behaviour in this case.

 

 

Thank you!

tomorrow I will check other side routers, because there many routers , and different vendor. If these routers report similar log,  I think it was transmission problem. but transmission engineer said their device was ok.

  thank you for your help!

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: