Traffic Everywhere

scottyd · ‎04-29-2009

Hi,

In our network we have about 90 switches. All core and distribution layer switches have redundant links. Spanning tree is configured and seems to be working, in that is is blocking appropriate ports. Cisco Network assistant is an easy way to see this.

However when I sniff traffic on an ordinary port on any switch I can see traffic that should not be there. The port is not spanned or anything. But I can see some traffic between hosts that is not broadcast traffic and none of which should be on that port.

The thing is that the more loops that I put in the network the worse it gets, even though STP takes care of it. As a result it seems that during high traffic usage there can be times that uplinks to dist switches are overloaded and discard packets due to no buffers.

Most switches have uplink ports set to priority 16 or 32 to determine which is primary etc. But not all, is this a problem?

There is a root set. There a many VLANs on the network.

I can only think that there is a spanning tree problem with my config.

Where do I start looking?

Thanks

chrisayres · ‎04-29-2009

Hi,

You have multiple routed connections into the switch network. So I would check for asymmetric routing. If this is happening switches will not see return traffic and will therefore not be able to build complete CAM tables causing unicast traffic to be broadcasted around the entire VLAN / switched network.

Joseph W. Doherty · ‎04-29-2009

Agree with Chris as a possible cause.

More info on this issue can be found here: http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801d0808.shtml

scottyd · ‎04-30-2009

Thanks for the replies guys.

I have had a close look at the documentation you linked to. I still have no answer but some more information that may help you help me.

I have run some "traceroute mac" commands with some of the IPs that I see traffic from, there seems to be no problem with them and the path they take. Besides some of the traffic that is being unicast is on the same subnet so layer three is not part of the problem.

There seems to not be STP Topoligy Changes either. The unicast traffic is pretty consistant and I cannot see any changes happening.

I do know that not all switches have a weighting on their uplink interfaces. But they do block one of the ports and they are not changing either.

I have seen somewhere that you are not suppossed to use UDLD in a multiple VLAN spanning tree environment. But I have seem some UDLD commands around.

Thanks for you advise.

chrisayres · ‎04-30-2009

Scott,

It doesn't sound like a STP problem to me, I would expect more devistating problems if it was. Do you know if all traffic is being flooded or just certain MAC's. Have a look at a destination MAC that is being flooded and check that the switches have CAM entries for this MAC. Are you doing any NIC teaming on the end hosts, the only time I have seen intravlan unicast traffic being flooded is when a mainframe that had dual NIC's had been configured to Rx on one NIC and Tx on the other so the destination MAC was never seen by the switches.

scottyd · ‎04-30-2009

Hi Chris,

Yes we do alot of teaming and VMWare. But I would think that they all TX and RX. But I will check some examples out.

How do I easily check the CAM entries? We have 3750, 2960 and 2950 switches.

Thanks

Scott

chrisayres · ‎04-30-2009

Scott,

use "sho mac-address-table" to view the entire table

or "sho mac-address-table address " to look for a specific MAC

bretjaquish · ‎04-30-2009

Are you seeing one way or two way unicast traffic?

One way unicast streams could be because of CAM timers and ARP timing not configured the same.

scottyd · ‎05-06-2009

Hi,

Yes I think they are one way. I found this article.

http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a00807347ab.shtml#broadcast

Does this mean that I should increase the CAM to four hours to match the ARP? Do you know of any side effects of this? We are running HSRP also, as the article suggests. But not all the traffic I see is being routed, some is on the same subnet.

Thanks all for your comments, I really need to get this sorted.