01-14-2014 02:17 AM - edited 03-07-2019 05:33 PM
We have an odd problem on our network where our mac-address tables on our endpoint switches are filling up very quickly. At first I thought it could be mac-flooding, and started to investigate which part of the network this could be coming from, however when looking at the mac-address-table count for each vlan I'm seeing quite an even spread of macs in each vlan, no obvious spiked anywhere. Another curious thing is that vlans that have only ever had 3 devices (2 servers, and the vlan interface) in are showing a much higher mac-address count than they should.
Is it possible that this is something is looping somewhere?
This is an example snippet from one of the endpoint 2960 switches. We have a mixture of 2950 and 2960s, and a 6509 core switch.
2960switch#sh mac-address-table count
Mac Entries for Vlan 1:
---------------------------
Dynamic Address Count : 271
Static Address Count : 0
Total Mac Addresses : 271
Mac Entries for Vlan 700:
---------------------------
Dynamic Address Count : 262
Static Address Count : 0
Total Mac Addresses : 262
Mac Entries for Vlan 703:
---------------------------
Dynamic Address Count : 264
Static Address Count : 0
Total Mac Addresses : 264
Mac Entries for Vlan 704:
---------------------------
Dynamic Address Count : 265
Static Address Count : 0
Total Mac Addresses : 265
Mac Entries for Vlan 705:
---------------------------
Dynamic Address Count : 260
Static Address Count : 0
Total Mac Addresses : 260
This continues across all the vlans, of which there are 50. Vlan705 for example is one where there have never been more than 3 mac addresses on that segment.
I can provide configs from the core.
Solved! Go to Solution.
01-14-2014 11:21 AM
Andi
How exactly have you configured the bridge group ?
It sounds like what you have done is bridged all the vlans together. If you have then that is why you are seeing all the mac addresses in all vlans. I'm not familiar with the WLC but i wouldn't have thought that is what you want to do. If you have redundant connections in your network this could create potential STP problems.
Jon
01-14-2014 04:33 AM
Andi
What are those mac addresses ie. have you tried to identify if they are valid on your network or not ?
You mentioned STP, do you have redundant uplink to a pair of switches doing the inter vlan routing ?
It is unlikely to be a L2 loop as generally these tend to take down networks quite quickly.
Jon
01-14-2014 07:10 AM
Have you tried to trace any of the addresses?
Pick a MAC address on Vlan705 that is not one of the 3 known ones and see which port the CAM table shows for it. I suspect it will be an uplink/downlink port to another switch so login to the next switch in the chain and do the same thing until you find the port the MAC address is actually plugged into.
01-14-2014 07:34 AM
Interesting, thanks both.
The mac addresses seem to be genuine devices on the network, but not ones that are actually in that vlan, so it's almost like a leak.
Tracing this up the chain I get right back to the core, where I see:
cyhr1#sh mac-address-table vl 705
Codes: * - primary entry
vlan mac address type learn qos ports
------+----------------+--------+-----+---+--------------------------
* 705 000e.d612.3456 static No -- Router
I'm curious by the learn=no parameter here.
01-14-2014 07:44 AM
Hello
Is this the count history after you have cleared the cam table?
Do you have any port-security configured?
If not, try enabling port sec on just one switch for testing with say a max default of 10 mac entries per port
int ran xxx
switchport port-security maximum 10
switchport port-security violation restrict
switchport port-security
or enable DAI
res
Paul
Please don't forget to rate any posts that have been helpful.
Thanks.
01-14-2014 07:49 AM
Hi,
that is the count history not long after clearing. I'm finding the address table fills up in a matter of minutes, and is spread across all the vlans.
We don't have any port security configured, and this is something we will definitely look at doing from now on to stop this affecting us like this in the future, however for this purpose it's treating the symptom rather than the cause, and I'd really like to find out what is wrong here.
01-14-2014 07:47 AM
That MAC address was manufatured by Cisco Systems so I suspect its a built in address for a Router/Switch.
If you do a #show interface vlan705, does the MAC address match the entry listed for this SVI?
01-14-2014 07:51 AM
Yes, it is the (obfuscated) mac address of our core switch. Running a show interface vlan 705 as you suggest does show the same mac address.
01-14-2014 07:59 AM
Ahhh I would expect to see that in the CAM table of any switch which has Vlan705 on it but it doesn't explain the other 200+ addresses.
A CAM table for a Vlan with only a few devices in will likely have more than those devices to account for built in addresses etc but it shouldn't be that many.
Try tracing another one?
Thanks
01-14-2014 08:10 AM
Apologies, I misunderstood you.
Tracing another valid mac address up the chain, I again get to the core, however this time I see:
cyhr1#sh mac-address-table | inc 0024.d675.cca6
* 704 0024.d675.cca6 dynamic Yes -- Bridge
* 710 0024.d675.cca6 dynamic Yes -- Po10
* 729 0024.d675.cca6 dynamic Yes -- Bridge
* 728 0024.d675.cca6 dynamic Yes -- Bridge
* 731 0024.d675.cca6 dynamic Yes -- Bridge
* 721 0024.d675.cca6 dynamic Yes -- Bridge
* 727 0024.d675.cca6 dynamic Yes -- Bridge
* 746 0024.d675.cca6 dynamic Yes -- Bridge
* 751 0024.d675.cca6 dynamic Yes -- Bridge
* 742 0024.d675.cca6 dynamic Yes -- Bridge
* 743 0024.d675.cca6 dynamic Yes -- Bridge
Vlan 710 is the wireless client vlan, and is where I would expect to find this mac address. The Po10 does go to the WLC controller.
I can't explain the Bridges though.
01-14-2014 08:13 AM
Hello
Can you post the out from sh sdm prefer this will show the TCAM resource
res
Pau
Please don't forget to rate any posts that have been helpful.
Thanks.
01-14-2014 08:17 AM
This is the output of that command from my endpoint 2960:
sh sdm prefer
The current template is "default" template.
The selected template optimizes the resources in
the switch to support this level of features for
0 routed interfaces and 255 VLANs.
number of unicast mac addresses: 8K
number of IPv4 IGMP groups: 0.25K
number of IPv4/MAC qos aces: 0
number of IPv4/MAC security aces: 0.25K
My 6509 core does not know the command.
01-14-2014 08:17 AM
I have not seen an interface listed as 'Bridge' before so this is where my knowledge ends I am afraid.
Perhaps one of the more experienced guys on here can help with this one?
01-14-2014 08:46 AM
I think you may have pointed me in the right direction actually. Removing my wireless vlan out of the bridge-group 1 has dramatically decreased the amount of entries in my end user switch TCAM.
I think tomorrow I will remove all vlans from the bridge group. I don't really know why we have it setup anyway.
01-14-2014 08:55 AM
Does your Wireless use a controller or are your Access Points autonomous?
Thanks
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: