Help with VLAN1 Broadcast

anthony.baker · ‎12-12-2007

Hey guys,

I wonder if you can help me....

I have two 6513's running IOS with FWSM's. Everything was fine for a long time until one switch decided to reboot.

I got it running again and I now have a problem that applies only to the switch that rebooted (even tho both are almost identical in config).

What's happening it seems is that any traffic on VLAN1 of that switch is being flooded out of all VLAN1 interfaces, even tho the MAC-address-table knows where everything is. The other switch doesn't have the problem and is working as expected.

I do see something strange in the mac-table -- it's a mac 3333.0000.000d that says it's static that has assigned itself to every port and every VLAN. However this is seen on the switch that isn't flooding as well so it may not be the problem.

Does anyone have any ideas as to why this could be happening - it's really slowing things down for certain critical machines here and I'm a bit stuck for ideas.

Please ask for any other info you may require to help.

Thanks in advance

Anthony

anthony.baker · ‎12-12-2007

Just some more info...

It's not limited to VLAN1 as I first thought but any attached VLAN to that switch.

The ARP and MAC tables are really big, holding info for things that the other switch should only know about.

Any ideas greatly appreciated!

Thanks

Anthony

Kevin Dorrell · ‎12-12-2007

Does the MAC table hold entries for stuff you have never heard of before? When the forwarding table gets full, the switch floods any other unknown MAC destinations. I wonder if that is what is happening.

There is a hacker tool called MAC-OFF that provokes this behavior deliberately so that traffic can be illicitly monitored. It can generate packets from 10000 MAC different addresses per second, filling the switch in a few seconds. Of course, the remedy for that is port-security on all access ports.

(Before any shouts "foul", that information is announced by Cisco in their layer-2 security Networkers session.)

Kevin Dorrell

Luxembourg

anthony.baker · ‎12-12-2007

Hey mate - thanks for the response...

Well, there's a couple that I've never seen before, namely:

3333.0000.0016 static Yes - Switch,Stby-Switch

3333.0000.0001 static Yes - Switch,Stby-Switch

3333.0000.000d static Yes

The last of which is most worrying as it's assigned itself to every port and then every VLAN so it's making the MAC table pretty big (There are probably 20 or so VLAN's).

Apart from that the MAC's look real enough, although strangly it's keeping MAC's of stuff that's on it's neighbour switch?? Is that normal?

Cheers

Anthony

Kevin Dorrell · ‎12-12-2007

Anthony,

To answer your last question first: that is quite normal, so long as the entries point to the trunk that links the two switches. After all, if the end device does a broadcast, both switches will see it, and both will make a note of where it thinks that MAC address can be found.

Now, to the addresses starting 3333 ... I have seen similar addresses in connection with IPv6 multicast. Has someone enabled IPv6 multicast routing on the switch? For example, I think 3333.0000.0001 corresponds to "all hosts", 3333.0000.000d to "all PIM multicast routers", and 3333.0000.0016 to IGMP.

Kevin Dorrell

Luxembourg

anthony.baker · ‎12-12-2007

Ok, yeah the MAC's are pointing at the etherchannel link so that's cool.

So you reckon IPv6 has got turned on somehow, from the reboot I guess. Do I need to turn just IPv6 off somehow or IGMP too?

I'm playing with my test switch now, trying to turn everything off but your continued help is much appreciated!

Thanks

Kevin Dorrell · ‎12-12-2007

If somehow IPv6 has got enabled by mistake and you don't use it, then disable it.

On the other hand, don't touch IGMP, and especially don't disable IGMP snooping, otherwise you will find that all your multicasts are flooded to all ports, and not just where they are required.

Good luck.

Kevin Dorrell

Luxembourg

anthony.baker · ‎12-12-2007

Thanks Kevin,

We don't use IPv6.

Is it as simple as 'turning it off' - I can't find a command that will do just that. (I'm using 12.2(18).

And you reckon that it's that (becuase it's filling up the mac-table) that's causing the problem?

I need all the luck I can get!

Anthony

Kevin Dorrell · ‎12-12-2007

Anthony,

I'm really not sure yet.

To fill the forwarding table, you need 100s of 1000s of entries. (Or about 20 seconds of MAC-OFF) Do you have any idea how many entries there are now? Could there be that order of magnitude?

The flooded traffic ... is it really unicasts? (That is, NOT packets with MAC addresses starting FF or 33 or 01)

Do you have IGMP snooping enabled? You should have. show ip igmp snooping

Can you see any ipv6 in the config? Could you post the config, with any sensitive stuff removed?

Kevin Dorrell

Luxmbourg

allan.thomas · ‎12-12-2007

In addition to the valuable information provided by Kevin, the address 3333.0000.0000.000x are specifically multicast mac address as Kevin had pointed out.

The reason why the appear in the table would be simply because IGMPv3 explicit host tracking has been enabled.

In Supervisor Engine 720 with Cisco IOS Software Release 12.2(14)SX and later, explicit host tracking is enabled for IGMPv3 hosts.

The switch software automatically programs the hardware for these multicast MAC addresses [3333.0000.0000.000x].

Refer to RFC 2464 for more information on these multicast MAC addresses.

If IGMP is not used in the network and if it is needed in order to eliminate the multicast MAC address entries for the MAC address table, issue these commands under the VLAN interface configuration:

no ip igmp snooping explicit-tracking

no ipv6 mld snooping explicit-tracking

no ipv6 mld snooping

These IGMP commands are automatically enabled when the Cisco IOS software is upgraded. If this is happens, then these IGMP commands would have to be disabled again.

What is Explicit Host Tracking of IGMPv3 Hosts?

Explicit host tracking is supported only with IGMPv3 hosts.

When you enable explicit host tracking and the Catalyst 6500 Series Switch works in proxy-reporting mode, the router might not be able to track all the hosts that are behind a VLAN interface. In proxy-reporting mode, the Catalyst 6500 Series Switch forwards only the first report for a channel to the router and suppresses all other reports for the same channel.

With IGMPv3 proxy reporting, the Catalyst 6500 Series Switch does proxy reporting for unsolicited reports and reports that are received in the general query interval.

Proxy reporting is turned on by default. When you disable proxy reporting, the Catalyst 6500 Series Switch works in transparent mode and updates the IGMP snooping database as it receives reports and forwards this information to the upstream router. The router can then explicitly track all reporting hosts.

When you disable explicit tracking, it disables fast-leave processing and proxy reporting.

IGMPv3 supports explicit host tracking of membership information on any port. The explicit host-tracking database is used for fast-leave processing for IGMPv3 hosts, proxy reporting, and statistics collection. When you enable explicit host tracking on a VLAN, the IGMP snooping software processes the IGMPv3 report that it receives from a host and builds an explicit host-tracking database that contains this information:

The port that is connected to the host

The channels that are reported by the host

The filter mode for each group that is reported by the host

The list of sources for each group that is reported by the hosts

The router filter mode of each group

The list of hosts that request the source for each group.

When you have to opportunity review the above information in the following link:-

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a008015504b.shtml#multicast720

Regards

Allan.

anthony.baker · ‎12-12-2007

Cheers guys,

1. The mac-table is not thousands and thousands big, it's about 10 hits on the space bar/10 pages or so. It looks big because it's like the below for every VLAN that's configured on the switch....

* 1 3333.0000.000d static Yes - Gi1/1,Gi1/2,Gi1/3,Gi1/5

Gi1/7,Gi1/11,Gi1/13,Gi1/33

Gi1/41,Gi1/42,Gi3/1,Gi3/3

Gi3/7,Gi3/18,Gi3/34,Gi3/37

Gi3/47,Gi4/11,Gi4/12,Gi4/14

Gi4/30,Gi4/31,Gi4/32,Gi4/35

Gi4/36,Gi4/40,Gi4/46,Gi4/48

Gi5/1,Gi5/2,Gi5/3,Gi5/4

Gi5/5,Gi5/6,Gi5/7,Gi5/8

Gi5/17,Gi5/18,Gi5/19,Gi5/20

Gi5/21,Gi5/22,Gi5/23,Gi5/24

Gi5/25,Gi5/26,Gi5/27,Gi5/28

Gi5/29,Gi5/30,Gi5/31,Gi5/32

Gi5/43,Gi5/44,Gi5/45,Gi5/46

Gi5/48,Gi6/2,Gi6/4,Gi6/6

Gi6/10,Gi6/12,Gi6/14,Gi6/21

Gi6/41,Gi6/46,Gi13/1,Gi13/2

Gi13/3,Gi13/4,Gi13/5,Gi13/6

Po5,Po6,Po10,Po282,Router

Switch

If it's not the mac-table being too big then what else would cause the traffic to be flooded?

I'm not too bothered about having that stuff in the table if everything's working - I just thought it was part of the problem.

I've attached the basic config (less sensitive bits and most of the ports) for you to have a look at but it's been running like that for about 2 years without any problems up till now. I don't understand what's changed to cause the problem.

I guess the switch could just have it's knickers in a twist and need a reload but I don't really want to do that as a) it's got live services running on it and b) it's admitting defeat!

Thanks again for the continued help...

Anthony

Kevin Dorrell · ‎12-13-2007

I'm still looking at the config, but there was one curious line that caught my eye:

arp 10.1.223.200 03bf.0a01.dfc8 ARPA

It caught my eye because that MAC address is a multicast, but not an IP multicast. It is therefore not subject to the normal filtering you would get from IGMP snooping, and so would be flooded to all ports. Now, if 10.1.223.200 was an important gateway, then I guess you would get a lot of unicast packets flooded on all ports.

That might not be the cause of your problem, but it is worth following up.

Kevin Dorrell

Luxembourg

anthony.baker · ‎12-13-2007

Hey Kevin,

That address is an entry for a windows web cluster that we have. It's a virtual MAC that windows comes up with and 'floats' between the servers. I can actually add the ports to that command and see if that helps -- but it's not a busy cluster and again it's been like that for yonks now (and is like that on the working of the two switches).

Do you think that it could be a bug and somehow I've just hit on it randomly!?

Cheers mate

alsayed · ‎12-13-2007

hello alan

i dont have this :explicit-tracking

DS2(config-if)#no ip igmp snooping ?

fast-leave Enable IGMP fast leave processing

last-member-query-interval Configure IGMP leave query timeout

mrouter Configure an L2 port as a multicast router port

querier Enable IGMP querier processing

ssm-safe-reporting Enable SSM Safe Reporting

static Configure an L2 port as a member of a group

i dont have this :explicit-tracking::: on my 720:s72033-ipservicesk9-mz.122-18.SXE6a.bin

anthony.baker · ‎12-13-2007

Tried deleting that static MAC and made no difference. Also tried adding my machine's statically but I'm still receiving flooded stuff I shouldn't be.

I've attached a packet that came to me that shouldn't -- I can't see why I'd be getting it (my ip is 10.1.230.250).

What a pain in the ass!

Ant