Solved: Mac address tables filling up on end switches

AndiMorris · ‎01-14-2014

We have an odd problem on our network where our mac-address tables on our endpoint switches are filling up very quickly. At first I thought it could be mac-flooding, and started to investigate which part of the network this could be coming from, however when looking at the mac-address-table count for each vlan I'm seeing quite an even spread of macs in each vlan, no obvious spiked anywhere. Another curious thing is that vlans that have only ever had 3 devices (2 servers, and the vlan interface) in are showing a much higher mac-address count than they should.

Is it possible that this is something is looping somewhere?

This is an example snippet from one of the endpoint 2960 switches. We have a mixture of 2950 and 2960s, and a 6509 core switch.

2960switch#sh mac-address-table count

Mac Entries for Vlan 1:

---------------------------

Dynamic Address Count : 271

Static Address Count : 0

Total Mac Addresses : 271

Mac Entries for Vlan 700:

---------------------------

Dynamic Address Count : 262

Static Address Count : 0

Total Mac Addresses : 262

Mac Entries for Vlan 703:

---------------------------

Dynamic Address Count : 264

Static Address Count : 0

Total Mac Addresses : 264

Mac Entries for Vlan 704:

---------------------------

Dynamic Address Count : 265

Static Address Count : 0

Total Mac Addresses : 265

Mac Entries for Vlan 705:

---------------------------

Dynamic Address Count : 260

Static Address Count : 0

Total Mac Addresses : 260

This continues across all the vlans, of which there are 50. Vlan705 for example is one where there have never been more than 3 mac addresses on that segment.

I can provide configs from the core.

Jon Marshall · ‎01-14-2014

Andi

How exactly have you configured the bridge group ?

It sounds like what you have done is bridged all the vlans together. If you have then that is why you are seeing all the mac addresses in all vlans. I'm not familiar with the WLC but i wouldn't have thought that is what you want to do. If you have redundant connections in your network this could create potential STP problems.

Jon

View solution in original post

Jon Marshall · ‎01-14-2014

Andi

What are those mac addresses ie. have you tried to identify if they are valid on your network or not ?

You mentioned STP, do you have redundant uplink to a pair of switches doing the inter vlan routing ?

It is unlikely to be a L2 loop as generally these tend to take down networks quite quickly.

Jon

devils_advocate · ‎01-14-2014

Have you tried to trace any of the addresses?

Pick a MAC address on Vlan705 that is not one of the 3 known ones and see which port the CAM table shows for it. I suspect it will be an uplink/downlink port to another switch so login to the next switch in the chain and do the same thing until you find the port the MAC address is actually plugged into.

AndiMorris · ‎01-14-2014

Interesting, thanks both.

The mac addresses seem to be genuine devices on the network, but not ones that are actually in that vlan, so it's almost like a leak.

Tracing this up the chain I get right back to the core, where I see:

cyhr1#sh mac-address-table vl 705

Codes: * - primary entry

vlan mac address type learn qos ports

------+----------------+--------+-----+---+--------------------------

* 705 000e.d612.3456 static No -- Router

I'm curious by the learn=no parameter here.

paul driver · ‎01-14-2014

Hello

Is this the count history after you have cleared the cam table?

Do you have any port-security configured?

If not, try enabling port sec on just one switch for testing with say a max default of 10 mac entries per port

int ran xxx

switchport port-security maximum 10

switchport port-security violation restrict

switchport port-security

or enable DAI

res

Paul

Please don't forget to rate any posts that have been helpful.

Thanks.

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

AndiMorris · ‎01-14-2014

Hi,

that is the count history not long after clearing. I'm finding the address table fills up in a matter of minutes, and is spread across all the vlans.

We don't have any port security configured, and this is something we will definitely look at doing from now on to stop this affecting us like this in the future, however for this purpose it's treating the symptom rather than the cause, and I'd really like to find out what is wrong here.

devils_advocate · ‎01-14-2014

That MAC address was manufatured by Cisco Systems so I suspect its a built in address for a Router/Switch.

If you do a #show interface vlan705, does the MAC address match the entry listed for this SVI?

AndiMorris · ‎01-14-2014

Yes, it is the (obfuscated) mac address of our core switch. Running a show interface vlan 705 as you suggest does show the same mac address.

devils_advocate · ‎01-14-2014

Ahhh I would expect to see that in the CAM table of any switch which has Vlan705 on it but it doesn't explain the other 200+ addresses.

A CAM table for a Vlan with only a few devices in will likely have more than those devices to account for built in addresses etc but it shouldn't be that many.

Try tracing another one?

Thanks

AndiMorris · ‎01-14-2014

Apologies, I misunderstood you.

Tracing another valid mac address up the chain, I again get to the core, however this time I see:

cyhr1#sh mac-address-table | inc 0024.d675.cca6

* 704 0024.d675.cca6 dynamic Yes -- Bridge

* 710 0024.d675.cca6 dynamic Yes -- Po10

* 729 0024.d675.cca6 dynamic Yes -- Bridge

* 728 0024.d675.cca6 dynamic Yes -- Bridge

* 731 0024.d675.cca6 dynamic Yes -- Bridge

* 721 0024.d675.cca6 dynamic Yes -- Bridge

* 727 0024.d675.cca6 dynamic Yes -- Bridge

* 746 0024.d675.cca6 dynamic Yes -- Bridge

* 751 0024.d675.cca6 dynamic Yes -- Bridge

* 742 0024.d675.cca6 dynamic Yes -- Bridge

* 743 0024.d675.cca6 dynamic Yes -- Bridge

Vlan 710 is the wireless client vlan, and is where I would expect to find this mac address. The Po10 does go to the WLC controller.

I can't explain the Bridges though.

paul driver · ‎01-14-2014

Hello

Can you post the out from sh sdm prefer this will show the TCAM resource

res

Pau

Please don't forget to rate any posts that have been helpful.

Thanks.

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

AndiMorris · ‎01-14-2014

This is the output of that command from my endpoint 2960:

sh sdm prefer

The current template is "default" template.

The selected template optimizes the resources in

the switch to support this level of features for

0 routed interfaces and 255 VLANs.

number of unicast mac addresses: 8K

number of IPv4 IGMP groups: 0.25K

number of IPv4/MAC qos aces: 0

number of IPv4/MAC security aces: 0.25K

My 6509 core does not know the command.

devils_advocate · ‎01-14-2014

I have not seen an interface listed as 'Bridge' before so this is where my knowledge ends I am afraid.

Perhaps one of the more experienced guys on here can help with this one?

AndiMorris · ‎01-14-2014

I think you may have pointed me in the right direction actually. Removing my wireless vlan out of the bridge-group 1 has dramatically decreased the amount of entries in my end user switch TCAM.

I think tomorrow I will remove all vlans from the bridge group. I don't really know why we have it setup anyway.

devils_advocate · ‎01-14-2014

Does your Wireless use a controller or are your Access Points autonomous?

Thanks