switchs go down completely, then come up

jjoseph01 · ‎02-17-2010

Hi all. I have a problem that I have not seen before that I need some input on. I have a company that has about 50 cisco switches, on multiple floors. They occasionally will call me and say that one floor in particular has goin down, meaning that in their view that their monitoring services lets them know that the switches cant be ping'ed. All of these switches on each floor go back to a core switch, which is vlan'ed off for each floor. When this happens, usually it only affects one floor, but it has affected more than one floor before (by the switches not being able to be reached, and about 5 to 7 switches on each floor). Usually, if Im not there, it will resolve itself in time. But what I do remember is that the very first time this happened, we found a laptop on the network with "internet connetion sharing" enabled and when we turned that off, the problem went away. However, I dont know for sure if the problem resolution was a coincedence or not. Has anyone seen anything like this before? And if so, can you give me some background as to what might be really going on (at least in your situation)? I appreciate any help.

spremkumar · ‎02-17-2010

Hi

The first thing comes to my mind is some kinda broadcast storm which might affect the reachability to the switch.

Have you checked accessing the application when the switch is not reachable. Also do try to console onto the switch and check whats exactly happening with and inside the switch.

I assume each floors belongs to its own respective vlan and also which model cisco switches you are using out over there?

regds

jjoseph01 · ‎02-17-2010

Yes, each floor is on its own vlan. These are 3560 and 2948 switches, all tied together with redundant paths back to the core switch. It will look like this:

core switch --fiber--> 3560 --fiber--> 3560 --fiber--> 3560 --fiber--> back to second fiber blade in core

So that if one switch goes down, STP will account for the lost switch and the other switches will be ok. But, when this issue happens, the computers on that floor can NOT access resources on the network (which would be to the server vlan, through the core).

Leo Laohoo · ‎02-17-2010

Sounds like a broadcast storm alright. I had this issue once and it's because someone installed a switch into a switchport with portfast. So everytime he would turn the switch on/off (inside a training room) we'd get a storm. Sounds like a broadcast storm alright. I had this issue once and it's because someone installed a switch into a switchport with portfast. So everytime he would turn the switch on/off (inside a training room) we'd get a storm.

Enable BPDUguard (spanning-tree bpduguard enable) on all of your access ports and ensure portfast on your trunks are disabled.

Kevin Dorrell · ‎02-17-2010

to add to Leo's post .... and make sure you don't have bpdufilter on the ports.

In the longer term, you should also consider implementing storm control on your switches.

Kevin Dorrell

Luxembourg

jjoseph01 · ‎02-18-2010

Thanks guys, how would I enable storm control on the switches?

Jerry Ye · ‎02-18-2010

Storm-control is an interface level command. I don't know what version you are running and I am providing the link to 12.2(46).

http://www.cisco.com/en/US/docs/switches/lan/catalyst3560/software/release/12.2_46_se/command/reference/cli3.html#wp2278213

HTH,

jerry