I have been working on a serious problem for over a week but so far have not reached a solution. We are experiencing severe network flooding on all switchports which is causing heavy packet loss across servers. To start with I thought it was a broadcast storm but when I connected a laptop to the Server VLAN and used Wireshark I was seeing a large number of unicast packets not relating to the laptop but communication between servers. To start with this didn't make a lot of sense as it was looking like our switches were behaving more like layer 1 hubs.
To describe our infrastructure we have 2 x 6513 core switches, each of these has a 4948 switch connected to it, and connected to these are 3020 switches in HP blade enclosure chassis. There are 4 x 3020 switches in a chassis with 2 of them connected to one 4948 and 2 to the other. There are port-channels connecting between the 6513s, 4948s and 3020s.
The server blades within the enclosures have 4 NIC ports with one connecting to each 3020 switch. The four NIC ports are bonded together in a mixture of Transmit-Load-Balance and Aggregate-Load-Balance.
Now this is where one of my theories to the source of the problem as TLB bonding across switches may cause the mac-address-table to miss entries and therefore flood out of all ports. I'm not sure if ALB has the same outcome.
Something else I've noticed on our network is the port-channels are not consistant. The 6513 and 4948 port-channels are set to src-dst-ip load balancing but the 3020 switches are set to src-mac load balancing. This one may be unrelated as it would only affect the load balancing technique but it's something I thought I should mention.
Please let me know if anyone has come across a similar issue in the past and what you were able to do to remedy it.
Thank you in advance!
There might be a clue in ". . . not sure if bonded across 2 or 4 switches is such a good idea. "
If L2 flows can happen such that switch doesn't see flow in both directions, switch will unicast flood.
One possible solution, might be to bond to the same physical or logical switch but on different hardware ports. I.e., different cards on a chassis switch or different switch members on a stackable switch (e.g. 3750, 2975).