Solved: C3750 Stack - %LINEPROTO-5-UPDOWN all three seconds

Erich Schommarz · ‎04-28-2010

Hello Together

We have a stack of 3750 Switches which logs funny errors in the log. I was able to find similar cases in the forum, but none with a resolution to it. So I try it once again and hope for good input and advise or tips.

The stack contains three switches:

*    1 12    WS-C3750G-12S      12.2(44)SE2           C3750-IPBASEK9-M
     2 28    WS-C3750G-24TS     12.2(44)SE2           C3750-IPBASEK9-M
     3 28    WS-C3750G-24TS     12.2(44)SE2           C3750-IPBASEK9-M

In the log we receive a lot of Interface down and up messages with a difference of three seconds. See a example further below. What we where able to find out so far:

- Problem only happens on Switch 1 of the stack

- All Interfaces are Fiber SFP GLX-SX Uplinks port to Cisco Access Switches

- Not all Uplinks are affected of this problem

- Reboot of switch member 1 didn't solve the problem, reboot of whole stack we weren't able to do so far

- If we verify the counter part -> User Access Switch of a Intf. which goes down on the core, we see no messages in the log (Down/Up)

- here e example of the config of the uplink port

          interface GigabitEthernet1/0/1
           description Trunk_Port_to_3560G_A31_01
           switchport trunk encapsulation dot1q
           switchport trunk native vlan 200
           switchport trunk allowed vlan 5,10,30,52,53,90,94,95,190,192,194,195,198,200
           switchport trunk allowed vlan add 294,295,900
           switchport mode trunk
           srr-queue bandwidth share 10 10 60 20
           srr-queue bandwidth shape 10 0 0 0
           udld port aggressive
           mls qos trust cos
           auto qos voip trust
           spanning-tree guard root

Attached the log filtered with the Down/Up messages.

dmitry_pimkin · ‎06-28-2010

Has faced such problem on C3750G-12S in a stack.
The problem was in failure of electrolytic capacitors in stabilizers of power supply of controllers.

View solution in original post

Ganesh Hariharan · ‎04-28-2010

Hello Together

We have a stack of 3750 Switches which logs funny errors in the log. I was able to find similar cases in the forum, but none with a resolution to it. So I try it once again and hope for good input and advise or tips.

The stack contains three switches:

*    1 12    WS-C3750G-12S      12.2(44)SE2           C3750-IPBASEK9-M
     2 28    WS-C3750G-24TS     12.2(44)SE2           C3750-IPBASEK9-M
     3 28    WS-C3750G-24TS     12.2(44)SE2           C3750-IPBASEK9-M

In the log we receive a lot of Interface down and up messages with a difference of three seconds. See a example further below. What we where able to find out so far:

- Problem only happens on Switch 1 of the stack

- All Interfaces are Fiber SFP GLX-SX Uplinks port to Cisco Access Switches

- Not all Uplinks are affected of this problem

- Reboot of switch member 1 didn't solve the problem, reboot of whole stack we weren't able to do so far

- If we verify the counter part -> User Access Switch of a Intf. which goes down on the core, we see no messages in the log (Down/Up)

- here e example of the config of the uplink port

interface GigabitEthernet1/0/1

description Trunk_Port_to_3560G_A31_01

switchport trunk encapsulation dot1q

switchport trunk native vlan 200

switchport trunk allowed vlan 5,10,30,52,53,90,94,95,190,192,194,195,198,200

switchport trunk allowed vlan add 294,295,900

switchport mode trunk

srr-queue bandwidth share 10 10 60 20

srr-queue bandwidth shape 10 0 0 0

udld port aggressive

mls qos trust cos

Hi,

As per the logs The data link level line protocol changed state,If the state change was unexpected, confirm the configuration settings for the interface and line protocol.Genrally these message is informational and usually requires no action.If the state change was unexpected, confirm the interface was not administratively shutdown. If so, Issue a 'no shutdown' on this interface to enable it. Also, check the physical cabling. Ensure that the appropriate cables (i.e. straight through or cross-over) are being used. Moreover, verify the proper duplex and speed is configured. Check that the correct media-type is configured when applicable (10BASE-T or AUI). Lastly, if this interface is not in use issue a 'shutdown' to disable it.

Hope to help !!

Ganesh.H

rahurao · ‎04-30-2010

Hi Ganesh,

I agree about the link to be checked physicall as it is going up and down but also i think you have missed the root guard configuration on the interface.

As per root guard is concerned it can be an issue for the flaaping of the link asroot guard still allows the device connected on the interface to participate in STP.

Root guard allows the device to participate in STP as long as the device does not try to become the root. If root guard blocks the port, subsequent recovery is automatic. Recovery occurs as soon as the offending device ceases to send superior BPDUs.

Reference link:

http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a00800ae96b.shtml#does

Also as a suggestion as would ask you to upgrade the IOS code on the switch as it is SE2 but there have been many versions updated after the same.

I hope this helps!

Erich Schommarz · ‎04-30-2010

Hello Ganesh, Hello Rahul

First of all thanks a lot for your feedback. About the feedback of Ganesh, like mentioned in my post the affected links are all Uplink ports to other Cisco Switches which are Access Switches for Enduser. So the log entry on the 3750 Switch is unexpected and shouldn't happen. Shutdown administrative is of course als not the case. And 10-BaseT and AUI, that was used in my last life. We are speaking here about SFP GLC-SX-MM Modules Switch to Switch.

Than both of you had the idea with checking the link physical. Normally I agree with you. But what I don't understand is, that when the link goes down on the C3750 Switch, the link on the access switch doesn't change. No link down / up message on the connected Access Switch.

Root Guard I think we can also ignore, because no log message.

My next steps which I have proposed to the customer are the following:

1. Upgrade Switch and reboot whole stack

2. Change affected Links from Switch member 1 to one of the SFP Slots on Switch member 2 or 3 to see if the problem moves or disappears.

3. Open a Support Ticket. If necessary exchange switch or open a TAC case at Cisco.

As soon I have new news I will update this post. Thanks all so far for your feedback.

Erich

rahurao · ‎04-30-2010

Hi Erich,

Thanks for the reply.

I am really not sure if upgrading the IOS would fix the issue as that is not related to the IOS.

For the second step i would say that you can open up a TAC case and i will grab it for you and then work with you as i am from TAC.

HTH

Rahul

smitesh kharecha · ‎04-30-2010

There are number of possible causes for that..

Few are already been discussed.. my guess will be following:

1. Faulty switchport. try changing SPF module

2. Cables might be damanaged.

3. Try upgrading the IOS.

And if none works, its time to open a TAC case which i assume there answer will be to upgrade IOS or will ask to raise RMA...

Erich Schommarz · ‎05-12-2010

Hello Together

Just for Info. IOS Upgrade and Reboot hasn't solved the problem. Today the customer has changed one SFP Module from Port 1/0/1 to 3/0/25. Since this change was made, the error disappeard.

log before change:

*Mar 3 17:02:01.960 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/10, changed state to up
*Mar 3 17:02:03.327 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/9, changed state to up
*Mar 3 17:19:40.933 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/10, changed state to down
*Mar 3 17:19:44.129 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/10, changed state to up
*Mar 3 18:01:36.962 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
*Mar 3 18:01:40.158 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Mar 3 18:02:47.733 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
*Mar 3 18:02:50.929 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Mar 3 18:30:01.246 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
*Mar 3 18:30:04.467 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Mar 3 18:39:23.958 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
*Mar 3 18:39:27.162 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Mar 3 18:48:01.477 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/5, changed state to down
*Mar 3 18:48:04.682 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/5, changed state to up
*Mar 3 18:53:03.730 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/10, changed state to down
*Mar 3 18:53:06.935 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/10, changed state to up
*Mar 3 18:57:27.729 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
*Mar 3 18:57:30.925 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Mar 3 19:53:11.483 SWISS: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (195.48.138.16)

log after the following change:

--> Port 3/0/25 angepasst, gleich wie 1/0/1

*Mar 3 19:53:37.775 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
*Mar 3 19:53:38.790 SWISS: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
*Mar 3 19:54:10.761 SWISS: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/25, changed state to up
*Mar 3 19:54:12.800 SWISS: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/25, changed state to up

one hour later no errors anymore:

3750_AV_01#sh clock
*20:45:31.607 SWISS Wed Mar 3 1993
3750_AV_01#

Customer is reconnecting the interface now back to 1/0/1. As soon new infos I will make a new update.

Erich

dmitry_pimkin · ‎06-28-2010

Has faced such problem on C3750G-12S in a stack.
The problem was in failure of electrolytic capacitors in stabilizers of power supply of controllers.

jorge.calvo · ‎06-28-2010

Hello,

Could you please post the output of the show interface gi1/0/1 command? I am interested in the amount of broadcast packets as I have seen many times this as being the cause for the line protocol being down and up on a trunk.

Regards.

Erich Schommarz · ‎07-19-2010

Hello Together

In the meantime the problem is solved. We have exchanged the affected switch and the problem disappeared now. It seems, that the answer of Dmitry Pimkin could be correct:

Has faced such problem on C3750G-12S in a stack.
The problem was in failure of electrolytic capacitors in stabilizers of power supply of controllers.

Thanks all for there help and input.

Erich

jayms1992 · ‎10-10-2012

Hi Dmitry ,

Could you please paste the appropriate link for solution u had given for this. Thanks

dmitry_pimkin · ‎10-11-2012

jaya raman wrote:

Hi Dmitry ,

Could you please paste the appropriate link for solution u had given for this. Thanks

I just picked up a soldering iron and replaced the swollen capacitors