Broadcast Storm

Answered Question
Jan 8th, 2010

We appear to have a broadcast storm happening on our LAN.  The utilization of our switch ports is pretty much always under 5%.  Currently, most of the ports on our switches are around 40% utilization.  We're having major issues with connections dropping right now.  The switches themselves are HP, but I was wondering if there is anything I could log on the inside interface of the ASA 5510 that would help to isolate the source of this undesired traffic.  The HP switches don't seem to offer much help in determining the source.  Is there anything that can be done short of shutting down links between switches one-at-a-time to isolate the problem?  Thanks!

I have this problem too.
0 votes
Correct Answer by sachinraja about 4 years 3 months ago

Hi Scnitzer

Is your ASA the default gateway for the LAN network ? Are the HP switches layer 3 and terminate the broadcast domain ? Incase HP switches are layer 3, I think thats the place where you would look at.. not sure if there will be useful messages on the ASA if the broadcast domain terminates on the switch. You can anyway see the utilization of the port connecting to ASA and see if there are any alarms there (increased traffic etc).. practically there could be a loop on the LAN network or a PC sending unncessary broadcasts (virus, worms etc), and isolating this is challenging.. You need to track suspicious mac addresses end to end and see where it leads to.. if you see huge traffic on PC ports, you need to look at that... lots to look at, but ONLY when the problem occurs ....

Regards

Raj

  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 5 (1 ratings)
Correct Answer
sachinraja Fri, 01/08/2010 - 11:05

Hi Scnitzer

Is your ASA the default gateway for the LAN network ? Are the HP switches layer 3 and terminate the broadcast domain ? Incase HP switches are layer 3, I think thats the place where you would look at.. not sure if there will be useful messages on the ASA if the broadcast domain terminates on the switch. You can anyway see the utilization of the port connecting to ASA and see if there are any alarms there (increased traffic etc).. practically there could be a loop on the LAN network or a PC sending unncessary broadcasts (virus, worms etc), and isolating this is challenging.. You need to track suspicious mac addresses end to end and see where it leads to.. if you see huge traffic on PC ports, you need to look at that... lots to look at, but ONLY when the problem occurs ....

Regards

Raj

r.d.schnitzer Fri, 01/08/2010 - 11:33

The HP switches are actually only operating at layer 2, and the ASA is the default gateway.

How would I go about finding suspicious MAC addresses, and then track them end to end?

Thanks,

Ryan

sachinraja Fri, 01/08/2010 - 11:37

Hello Ryan

How did you find that there is a broadcast storm ? Is the connectivity to ASA inside interface dropping a lot of packets ? ping to default gateway from the PC's giving packet loss ? or did you notice extremely high traffic on some ports of the HP switch ?

with ASA you can look at the "show log" and see if it shows any dropped connection.. i guess for this to show , you might need to have logging levels defined with debugging option.. not sure if access-list misses are shown in notification level... we need to have see some kinda suspicious mac address or ip address visible on the layer 3 devices to track them down... did this happen after you added any new component on the network or due to a recent change ?

Raj

r.d.schnitzer Fri, 01/08/2010 - 12:23

We first noticed the problem because our remote sites were having trouble maintaining their connections to our servers.  We then found high packet loss when pinging the servers the remote users were accessing.  When checking our switches we saw that most of the switches had abnormally high usage on many of the ports.

It would be easier if we had added a new device to our network or made a change, but nothing has been done recently.

I increased buffered logging to debugging for the time being, but it hasn't shown anything interesting so far.

And now we think we may have located the source.  Right when we figured out which server it was, it seems to have stopped whatever process was bogging down the network with traffic.  The server was sending out 80 MB/s of traffic.  We're going to do some malware scans.  Hopefully this isn't a false positive.  Thanks for your help.

Ryan

sachinraja Fri, 01/08/2010 - 12:51

Ryan

Thats great.. But im not sure why you should have your servers, desktops, and other componenets on the same broadcast domain.. You can possibly isolate them and have them protected ! especially the servers ! Since you dont have a layer 3 switch on your network, the only way to isolate them (if you think of) is to use a dmz interface or create sub-interfaces on ASA and bifurcate the network... but all these involve a lot of planning, re-addressing etc, which might take sometime.. Infact my best advice for you is to procure a layer 3 switch and have different VLAN SVIs configured on it, to reduce the broadcast domain.. based on your user count, you can choose from a variety of products doing Layer 3 on your LAN.

Hope this helps.. All the best..

Raj

Actions

Login or Register to take actions

This Discussion

Posted January 8, 2010 at 10:55 AM
Stats:
Replies:5 Avg. Rating:5
Views:7694 Votes:0
Shares:0

Related Content

Discussions Leaderboard

Rank Username Points
1 14,997
2 8,150
3 7,720
4 7,078
5 6,723
Rank Username Points
175
80
60
59
55