Broadcast Storm that won't go away.

flnetengineer · ‎08-30-2007

We have a site located across the street where we provide service. They are connected with 100 meg MM fiber link. We recently moved these folks onto our wire from a 3com network that their IT guys used to run, and when I brought them over I put them on the same VLAN as some of their users who reside on the same side of the street as us. So basically here is how they are connected:

near side users -> (6509 vlan 87) -> SM fiber (Cisco 3508) -> MM Fiber 100 Meg link -> (Remote Site Cisco 2924 vlan 87)

These people have been experiencing broadcast storms almost everyday, each time happening mid morning and generally around mid afternoon. I have traced down several PC's and shut them down to alleviate the problem, but their IT folks say they can't find anything wrong with the PC's even though once these are shut down the problem clears up. It's been different PC's each time. I implemented port-storm control with a level of 5%, but that seems like it might be to high, and I may need to lower it, but it has helped to recover from the problem faster. Unfortunately I have a few switches over there that don't support the command.

The funny thing is the broadcasts only effect people in that building, and they latency isn't experienced across the street, not even the 3508 is affected neither are the users on the same vlan across the street.

I am probably going to implement VLANS within the building as I do have a trunk into the building and vtp setup, but since it is a building wide problem I am not sure how much that's going to help. Anyone have any suggestions, or ideas? Thanks.

Francois Tallet · ‎08-30-2007

Sounds mysterious indeed;-) What has changed on the side that is experiencing the problem? Was it just a 3com replaced by a Cisco or have their been more complex re-design over there? I can imaging that some PC are bridging traffic between them, outside of the network for instance, but it's difficult to understand why it would not have affected the previous design at that stage.

Regards,

Francois

flnetengineer · ‎08-30-2007

They basically had a network with a 3com cisco mix (mostly 3com), but for the most part they had no special configs everything just out of the box and plug in. I moved all of their users and IT staff onto our wire to an all Cisco environment. But, as a result of that some of their 3coms came over onto our wire still configured with their IP's. I have no way of managing these 3Coms, and political red tape prevents me from removing them from the network at this time.

I will say what makes this real mysterious is the fact that I have isolated the issue to the 3rd and 4th floors of the building, and the 3rd floor just happens to be where their IT group sits.

I am wondering if separating the building into different VLANS by floor will help?

james.hicks · ‎08-31-2007

This may not have anything to do with your problem, but at one time when we used them extensively, 3Com 3300 switches out of the box had spanning-tree turned off by default. In addition after spanning tree was enabled, when a port was configured 100/full, the port cost became 18 instead of 19 per IEEE (we always manually reset cost to 19). Sniff the network/vlan to identify the traffic type. Multicast can be a major concern.

jph

glen.grant · ‎08-30-2007

How did you determine it was a broadcast problem ? Could it be someone who has multicast turned on , if not configured correctly this could bury a 2924 . I would suggest maybe download something like wireshark and mirror the vlan to a destination port when it is happening and see if you can see anything . You may need to use a analyzer to figure this out . I would also check the pc's out for viruses and anamolies like that.

flnetengineer · ‎08-31-2007

I cleared the counters on all my interfaces last week, and some of your responses got me thinking. I had considered multicast to possibly be the problem, but I hadn't fully explored that hypothesis until I read some of your responses. As I said in a previous post having isolated the cause to two floors, I focused on the 3rd floor switches and I found a port that had accumulated over 4000 collisions, and was running at 10Mb/s Half-Duplex. I also checked the uplink port on the switch and within a weeks time the switch had processed 17 million multicasts!!! Unfortunately I can't get into the building until Tuesday to tell their IT staff to remove the device, and politics wont' allow me to shut the port down.

Does this seem like the culprit? I included screenshots, I believe it is as that switch only has 8 users connected to it never any more than that.

storm