C3850-NM-4-10G went to err-disable when using 1GE SFP's

~chris · ‎11-14-2013

Hi together,

I have a problem with C3850-NM-4-10G module on Catalyst 3850 with 48 Ports. At the moment the switch is

running with IOS version 3.2.3.

I know that this module can pick up 1GE and 10GE SFP(+) optics in several combinations.

So that's not the problem.

If I insert any 1GE SFP to the physical ports I receive the following err-disable message:

*Nov 13 09:50:42.539: %PLATFORM_PM-6-MODULE_ERRDISABLE: The inserted SFP module with interface name Te1/1/1 is not supported

*Nov 13 09:50:42.539: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/1, putting Te1/1/1 in err-disable state

A "shut" and "no shut" helps to get the interfaces UP/UP. After the interfaces are in state UP/UP,

I see the correct SFP-type in the output of "show interface status".

IMHO it is not correct that the Ports went into err-disable state at the first time.

Right now, I am not sure, what will happen after a reboot.

Does anybody has an idea how to solve it?

regards

./chris

~chris · ‎11-14-2013

Here some additional informations and outputs from the switch:

# No 1GE SFP insert:

Switch#show int status | inc Te1

Te1/1/1 notconnect 1 auto auto unknown

Te1/1/2 notconnect 1 auto auto unknown

Te1/1/3 notconnect 1 auto auto unknown

Te1/1/4 notconnect 1 auto auto

10:02

Switch#

# Put SFP into the slots

*Nov 14 01:58:22.118: %PLATFORM_PM-6-MODULE_ERRDISABLE: The inserted SFP module with interface name Te1d

*Nov 14 01:58:22.118: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/1, putting Te1/1/1 in erre

Switch#

*Nov 14 01:58:41.935: %PLATFORM_PM-6-MODULE_ERRDISABLE: The inserted SFP module with interface name Te1d

*Nov 14 01:58:41.935: %PM-4-ERR_DISABLE: gbic-invalid error detected on Te1/1/2, putting Te1/1/2 in erre

Switch#

10:02

# Check the interfaces status

Switch#show int status | inc Te1

Te1/1/1 err-disabled 1 auto auto unknown

Te1/1/2 err-disabled 1 auto auto unknown

Te1/1/3 notconnect 1 auto auto unknown

Te1/1/4 notconnect 1 auto auto

10:02

# Perform a "shut" and "no shut"

Switch(config)#int ten1/1/1

Switch(config-if)#no shut

Switch(config-if)#int ten1/1/2

Switch(config-if)#

# Interfaces change do "normal status" UP but notconnected

*Nov 14 01:59:33.126: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1sht

*Nov 14 01:59:33.976: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/1/1, changed state to do

10:03

# Check interface status again.... hmm working as aspected

Switch#show int status

*Nov 14 01:59:54.975: %SYS-5-CONFIG_I: Configured from console by console| inc Te1

Switch#show int status | inc Te1

Te1/1/1 notconnect 1 full 1000 1000BaseSX SFP

Te1/1/2 notconnect 1 full 1000 1000BaseSX SFP

Te1/1/3 notconnect 1 auto auto unknown

Te1/1/4 notconnect 1 auto auto

Doing a "write memory" followed by reload

# After the reload and connectting a fibercable to see if the interfaces are forwared traffic and STP

is blocking as aspected.

Switch#show int status | inc Te1

Te1/1/1 connected 1 full 1000 1000BaseSX SFP

Te1/1/2 connected 1 full 1000 1000BaseSX SFP

Te1/1/3 notconnect 1 auto auto unknown

Te1/1/4 notconnect 1 auto auto unknown 10:10

VLAN0001

Spanning tree enabled protocol ieee

Root ID Priority 32769

Address c025.5c84.cf80

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32769 (priority 32768 sys-id-ext 1)

Address c025.5c84.cf80

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Te1/1/1 Desg FWD 4 128.53 P2p

Te1/1/2 Back BLK 4 128.54 P2p

Looks quite good after re reload, but the err-disable messages at the beginning still stay spooky to me... or?

regards

./chris

Leo Laohoo · ‎11-14-2013

What is the exact model of the SFP?

~chris · ‎11-14-2013

GLC-SX-MMD

C-Spilker · ‎11-14-2013

HI!

I am Just doing something similar and have exactly the same issue.

No matter wich port and combination I use, all GLC-SX-MMD= go into err-disable after insertion on any port (using the NM-2-10G).

A reboot seems to work without problems, all Modules inserted before the reboot come up successfull.

I use IOSX version 3.3.0

My secound deployment of 3850 and again bugs bugs bugs

Leo Laohoo · ‎11-14-2013

Create a TAC Case.

This could be an IOS bug.

Leo Laohoo · ‎11-14-2013

GLC-SX-MMD

Hmmmmm ... Stupid question: Have you tried GLC-SX-MM (non-DOM)?

~chris · ‎11-15-2013

It's not a stupid question :-)

But yes I did. I have tested with GLC-SX-MM and GLC-T.... always the same issue :-(

In the meantime I really think it is a bug. I also did some research in bug search database but did not find any hint.

So I think I will open a TAC case for it.

regards

./chris

Leo Laohoo · ‎11-15-2013

Thanks, Chris.

Do let us know if and when you get the Bug ID.

C-Spilker · ‎11-15-2013

Hi!

As I have a similar issue at the same time I tested with a pretty old GLC-SX-MM= and it worked for me. but thats not an option as I prepare customer equipment....

As a Quick fix i enabled autorecovery:

errdisable recovery cause gbic-invalid

errdisable recovery interval 30

This way my GLC-SX-MMD= starts working properly with a delay of 30 seconds.

I also did several reboots without autorecovery and everytime all SFPs came up successfull.

So maybe this helps to get your devices runing too.

Christian

~chris · ‎11-15-2013

Sure I know this commands, but IMO it is only a workaround to get the SFPs working during implementation at customer site. This is not the answer why 1GE SFPs went to err-disable.

But it is interessting that a very old GLC-SX-MM w/o DOM is working, because I also tested with the old SFPs. Result: still err-disable.

Actually we are discussing if we open a TAC case. If we will do so, I will post the result here

regards

./chris

C-Spilker · ‎11-15-2013

No, it can't be a soultion, it's nothing more than a hotfix.

I didn't want to blame you with the commands but wanted to document a way to fix for other guys finding this thread

I am in the same situation as you are and actually prepare customer equipment.

While browsing the releasenotes to verify my 3850-NM-2-10G config with with 2x GLC + 2x SFP+, I just found a bug that could match to our issue:

CSCuj31712

Removing and reinserting SFP modules causes the port to go into an error-disabled state.

The workaround is to use the shutdown/no shutdown commands on the port.

My TAC case is already opened...

Christian

~chris · ‎11-15-2013

np :-)

The bug ID is quite interessting, I was only searching for Catalyst 3850, NIM, err-disable etc. but did not find that bug.

Hmm... if you have already opened a TAC case, I do not need to open once more for the same issue. Perhaps you can post the findings of the TAC here in die CSC thread.

thx

./chris

Leo Laohoo · ‎11-15-2013

Hmm... if you have already opened a TAC case, I do not need to open once more for the same issue.

I don't have a 3650/3850 but I would like to offer my opinion. I disagree with you. Even though Christian has created a TAC Case, I would also recommend you create one as well. I believe people inside TAC will be able to determine how serious this bug is if more and more people create TAC Cases for similar cases. Call it "measuring metrics".

Richard Primm · ‎11-15-2013

Hi Guys,

I'm more than happy to help you with this and get a bug filed if need be. I'm not aware of this particular issue, but I have all of the above components and can cycle through them in my lab early next week (out of the office for the weekend). Once you get a tac case open, post or send me the SR number and I will work on it for you.

thanks

-Luke