failed loopback test - is this a hardware problem?

Unanswered Question
Jul 13th, 2010

I have 2 6509 running as VSS.  It a new system installed a month or so ago.  When I try configure and connect the 10G connections on a WS-X6716-10GE (16 port 10G module), one of the port "error disabled".  I reseated the card and the error came back (see syslog  below):

Jul 13 16:54:29 MDT: %OIR-SW1_SP-6-REMCARD: Card removed from slot 1, interfaces disabled
Jul 13 16:54:29 MDT: SW1_SP:  Switch 1 Physical Slot 1 - Module Type LINE_CARD  removed
Jul 13 16:54:44 MDT: SW1_SP:  Switch 1 Physical Slot 1 - Module Type LINE_CARD  inserted
Jul 13 16:56:45 MDT: %DIAG-SW1_SP-6-RUN_MINIMUM: Switch 1 Module 1: Running Minimal Diagnostics...
Jul 13 22:56:43.181: DFC1: Switch mode info received by DFC..!
Jul 13 22:56:44.741: %SYS-DFC1-5-RESTART: System restarted --
Jul 13 22:56:44.749: DFC1: Currently running ROMMON from S (Gold) region
Jul 13 16:57:04 MDT: %DIAG-SW1_SP-6-DIAG_OK: Switch 1 Module 1: Passed Online Diagnostics
Jul 13 16:57:04 MDT: %OIR-SW1_SP-6-INSCARD: Card inserted in slot 1, interfaces are now online
Jul 13 16:57:08 MDT: %C6K_PLATFORM-SW1_SP-4-OVERSUB_MODECHANGE: WARNING: Switch to

TRANSPARENT mode on module 17 port-group 3
Jul 13 16:57:11 MDT: %C6K_PLATFORM-SW1_SP-4-OVERSUB_MODECHANGE: WARNING: Switch to

TRANSPARENT mode on module 17 port-group 4
Jul 13 16:57:16 MDT: SW1_SP:  Card inserted in Switch_number = 1 ,
Jul 13 17:00:04 MDT: %HA_EM-6-LOG: Mandatory.go_nondislp.tcl: GOLD EEM TCL policy for

TestNonDisruptiveLoopback
Jul 13 17:00:03 MDT: %CONST_DIAG-SW1_SP-3-HM_PORT_TEST_FAIL: Switch 1 Module 1

TestNonDisruptiveLoopback Port(s)[13] failed. System operation continues.
Jul 13 17:00:04 MDT: %PM-SW1_SP-4-ERR_DISABLE: diagnostics error detected on Te1/1/13,

putting Te1/1/13 in err-disable state
Jul 13 17:00:04 MDT: %PM-SW2_SPSTBY-4-ERR_DISABLE: diagnostics error detected on

Te1/1/13, putting Te1/1/13 in err-disable state
Jul 13 17:09:32 MDT: %HA_EM-6-LOG: Mandatory.go_unusedportlpbk.tcl: GOLD EEM TCL policy

for TestUnusedPortLoopback
Jul 13 17:09:31 MDT: %CONST_DIAG-SW1_SP-3-HM_PORT_TEST_FAIL: Switch 1 Module 1

TestUnusedPortLoopback Port(s)[12,16] failed. System operation continues.

sa-vss>sh version
Cisco IOS Software, s72033_rp Software (s72033_rp-ADVIPSERVICESK9_WAN-VM), Version 12.2(33)SXI3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Tue 27-Oct-09 06:31 by prod_rel_team

ROM: System Bootstrap, Version 12.2(17r)SX6, RELEASE SOFTWARE (fc1)

sa-vss uptime is 1 week, 6 days, 22 hours, 41 minutes
Uptime for this control processor is 1 week, 6 days, 22 hours, 40 minutes
Time since sa-vss switched to active is 1 week, 6 days, 22 hours, 40 minutes
System returned to ROM by reload at 18:55:06 MDT Tue Jun 29 2010 (SP by reload)
System restarted at 18:58:43 MDT Tue Jun 29 2010
System image file is "sup-bootdisk:s72033-advipservicesk9_wan-vz.122-33.SXI3.bin"
Last reload reason: Reload Command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
[email protected].

cisco WS-C6509-E (R7000) processor (revision 1.5) with 1040384K/8192K bytes of memory.
Processor board ID SMC14040025
SR71000 CPU at 600Mhz, Implementation 1284, Rev 1.2, 512KB L2 Cache
Last reset from s/w reset
2 Virtual Ethernet interfaces
150 Gigabit Ethernet interfaces
36 Ten Gigabit Ethernet interfaces
1917K bytes of non-volatile configuration memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4 (1 ratings)
Loading.
Reza Sharifi Tue, 07/13/2010 - 17:22

Are these ports being used for VSL connection between the 2 6500s? If yes use port 1 for one of the connection and port 5 for the other.  Do not use ports 1 and 2, 3 ,etc.

HTH

Reza

kduran Tue, 07/13/2010 - 17:28

Hi,


You have more than 1 issue here.  What does the output of 'show module' look like?

syip Wed, 07/14/2010 - 07:12

I rebooted the whole system and the port (1/1/13) came up now.  Should I still be concerned with the previous errors?  By the way, here are the show module output for the two switches:

sa-vss#show module sw 1
Switch Number:     1   Role:   Virtual Switch Active
----------------------  -----------------------------
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  1   16  CEF720 16 port 10GE                    WS-X6716-10GE      SAL1413E9K0
  2   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP       SAL1412DBPP
  3   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAL1413E26Z
  5    5  Supervisor Engine 720 10GE (Active)    VS-S720-10G        SAL1413E2U0

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  1  001f.6ca4.51b0 to 001f.6ca4.51bf   1.1   12.2(18r)S1  12.2(33)SXI3 Ok
  2  68ef.bd16.6a3c to 68ef.bd16.6a53   4.3   12.2(18r)S1  12.2(33)SXI3 Ok
  3  c47d.4f8b.0d94 to c47d.4f8b.0dc3   3.4   12.2(18r)S1  12.2(33)SXI3 Ok
  5  0026.cb61.07d0 to 0026.cb61.07d7   3.2   8.5(3)       12.2(33)SXI3 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
  1  Distributed Forwarding Card WS-F6700-DFC3C     SAL1414ED4F  1.4    Ok
  2  Centralized Forwarding Card WS-F6700-CFC       SAL1413DVSU  4.1    Ok
  3  Centralized Forwarding Card WS-F6700-CFC       SAL1413DWM8  4.1    Ok
  5  Policy Feature Card 3       VS-F6K-PFC3C       SAL1413E0HG  1.1    Ok
  5  MSFC3 Daughterboard         VS-F6K-MSFC3       SAL1413DZPB  2.1    Ok

Mod  Online Diag Status
---- -------------------
  1  Pass
  2  Pass
  3  Pass
  5  Pass

sa-vss#show module sw 2
Switch Number:     2   Role:  Virtual Switch Standby
----------------------  -----------------------------
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  1   16  CEF720 16 port 10GE                    WS-X6716-10GE      SAL1413E9HV
  2   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP       SAL1412DBNZ
  3   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAL1412DNZA
  5    5  Supervisor Engine 720 10GE (Hot)       VS-S720-10G        SAL1413E2VE

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  1  c47d.4f8f.f100 to c47d.4f8f.f10f   1.1   12.2(18r)S1  12.2(33)SXI3 Ok
  2  68ef.bd77.c6f4 to 68ef.bd77.c70b   4.3   12.2(18r)S1  12.2(33)SXI3 Ok
  3  68ef.bd34.8b00 to 68ef.bd34.8b2f   3.4   12.2(18r)S1  12.2(33)SXI3 Ok
  5  0026.cb61.0760 to 0026.cb61.0767   3.2   8.5(3)       12.2(33)SXI3 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
  1  Distributed Forwarding Card WS-F6700-DFC3C     SAL1414ED2N  1.4    Ok
  2  Centralized Forwarding Card WS-F6700-CFC       SAL1413DWGX  4.1    Ok
  3  Centralized Forwarding Card WS-F6700-CFC       SAL1413DWAZ  4.1    Ok
  5  Policy Feature Card 3       VS-F6K-PFC3C       SAL1413E0G4  1.1    Ok
  5  MSFC3 Daughterboard         VS-F6K-MSFC3       SAL1413DZLP  2.1    Ok

Mod  Online Diag Status
---- -------------------
  1  Pass
  2  Pass
  3  Pass
  5  Pass

VSL uses the 10G ports on the two supervisors:

sa-vss#sh int status | in connected
Te1/1/5      sa-3k1 te1/0/1     connected    trunk        full    10G 10Gbase-CU5M
Te1/1/13     sa-5k1 eth1/17     connected    trunk        full    10G 10Gbase-SR
Gi1/2/24     dual-active hello  connected    routed       full   1000 1000BaseT
Gi1/3/48     dual-active hello  connected    routed     a-full a-1000 10/100/1000BaseT
Gi1/5/3                         connected    5          a-full a-1000 10/100/1000BaseT
Te1/5/4      Virtual Sw Link    connected    routed       full    10G 10Gbase-CU5M
Te1/5/5      Virtual Sw Link    connected    routed       full    10G 10Gbase-CU5M
Po1          svr uplink         connected    trunk      a-full    10G
Po5          sa-3k1 uplink      connected    trunk      a-full    10G
Po13         sa-5k uplink       connected    trunk      a-full    10G
Po54         sw1 VSL            connected    routed     a-full    10G
Po55         sw2 VSL            connected    routed     a-full    10G
Te2/1/1      svr te4/0/1        connected    trunk        full    10G 10Gbase-SR
Te2/1/13     sa-5k2 eth1/17     connected    trunk        full    10G 10Gbase-SR
Gi2/2/24     dual-active hello  connected    routed       full   1000 1000BaseT
Gi2/3/48     dual-active hello  connected    routed     a-full a-1000 10/100/1000BaseT
Gi2/5/3                         connected    5          a-full a-1000 10/100/1000BaseT
Te2/5/4      Virtual Sw Link    connected    routed       full    10G 10Gbase-CU5M
Te2/5/5      Virtual Sw Link    connected    routed       full    10G 10Gbase-CU5M
sa-vss#

Kanes Ramasamy Sun, 08/28/2011 - 08:28

Hi Guys,

I am having the same exact error as the above and curently using the same IOS as the above.

The only difference is that the one of the module is showing error minor.

Could you please assist on this?

Thanks.

Somasundaram Ja... Sun, 08/28/2011 - 09:23

Hi,

Diagnostic tests usually meant for checking the hardware .

Have you reseated the line card after it shown up minor error?

Before reseating the line card, set the diagnostic level to complete.

conf t

diagnostic bootup level complete

exit

If you still see the minor error in show module output, then it is an indication of hardware failure.

Hope this helps

Cheers

Somu

Rate helpful posts

Kanes Ramasamy Sun, 08/28/2011 - 09:26

Hi Somu,

Thanks for this and I will try to do the said.

WIll keep you updated.

Thanks.

cieftelecom Fri, 05/11/2012 - 02:30

Hi,

I have exactly the same message in our log with the same IOS. However the difference is the faulty port is actually not used and is disabled...

Could be a bug on this version? I have many other catalyst 6500 with many 10G modules and different IOS but I have only one catalyst 6500 with this IOS, two 10G modules and configured in VSS.

Thanks

Actions

This Discussion

Related Content