Strange Trunk utilization

[mlipsey]

I've inherited a poorly designed network that is experiencing some problems.

It has 2 6509 core switches and then extending out from the core are individual 3750s connected with a single trunk. one 6509 is side "A" and the other is side "B".

Our backup server is directly connected to side "B", whenever it kicks off I see a serious spike in the utilization of the trunks between side "A's" 3750s and side "A's" 6509, this spike is there even though there are no target hosts being backed up on those 3750s.

I can't explain it so I can't fix it.

Any insight would be helpful, we have of course vlan interfaces running HSRP on the 6509s and multiple vlans. No pruning is occuring either via VTP or allowed vlans. BAsically all vlans are allowed on all trunks.

I'm going to try to hit a trunk with a sniffer tonight I think.

Thanks

[johgill]

Hello,

Are you saying the backups are not warranting the high trunk utilization? Do you know what rate the backups are running vs how much the trunks spike?

It sounds like you may be running into unicast flooding, you can test this by sniffing on a member of the same vlan as the backup server. Do not use a span session and lets see if we can see traffic that belongs to the backup server.

The solution to this issue is usually to match cam and ARP timers, but we can take a look as we go. Leet us know the outcome of the sniffing.

("show interface" includes the rate in and out of a port - conf t, int x, load-interval 30 helps the granularity of this rate)

[mlipsey]

I'll do a sniff of that tomorrow if I can. I did one today using span; it wasn't very helpful...

[mlipsey]

Well, I didn't do a sniff of just the vlan yet but I did input a static cam entry on the switch for the one backup server and my blasts of traffic to those other trunks stopped.

Question is, why isn't a cam entry being built for this server. It seems like there never is one built which causes the unicast broadcast problem...

[johgill]

Is the port the directly connected port where the server is connected? I assume it is actually the port across from this switch that lost the entry. This is typical in backup situations where the server talks to one host for an extended period of time or is quiet for a long time.

The cam entries age every 5 minutes by default and the arp entry every 4 hours. There is a big overlap there in terms of when we will send an ARP and thus learn a new entry. The issue is typically that you have vlans active on different switches and depending on who you are talking to, you may be routing through one switch and coming back in through another. At layer 2, that traffic never hits your cross-connect between the switches and thus the entry is not updated.

Here is a document that explains this issue:

http://www.cisco.com/en/US/tech/tk648/tk362/technologies_tech_note09186a0080094afd.shtml#t8

Regards