Its not just VLAN 303 though :( It seems ARP traffic to different VLANs gets trough to 101 too some times. Also if it makes any difference (I didn't think it did) - 6509s share the same MAC for each VLAN.

A bit of an update on this one.

One (and only one) of my core switches seems to be generating (or passing through?) frames with 0000.0000.0000 MAC addresses. These very frames then are seen in the wrong VLAN on all other switches.

SW1 -Apr 1 14:17:13.988 gmt: IP ARP: sent req src 80.68.43.2 001b.0dec.59c0, dst 80.68.43.81 0000.0000.0000 Vlan303

Apr 1 14:17:13.992 gmt: IP ARP: sent req src 80.68.48.34 001b.0dec.59c0,

dst 80.68.48.48 0000.0000.0000 Vlan202

SW2 -Apr 1 14:17:16.556 gmt: IP ARP req filtered src 80.68.43.2 001b.0dec.59c0, dst 80.68.43.81 0000.0000.0000 wrong cable, interface Vlan101

Apr 1 14:17:16.556 gmt: IP ARP req filtered src 80.68.48.2 001b.0dec.59c0, dst 80.68.48.26 0000.0000.0000 wrong cable, interface Vlan101

Plus I get a lot of unicats ARP flooding in the VLAN 303 - I have set CAM and ARP timers to the same value.

Any idea?

Hello Victor,

the issue is more complex then what I thought of at the beginning.

A basic note: an ARP request should be sent out with a broadcast address of ffff.ffff.ffff I'm concerned with these frames having an all zeroes destination MAC address.

I would do the following:

I would check if the source ip addresses/ source mac addresses are legitimate users of the network.

For example I notice:

all frames have the same source MAC:

src 80.68.43.2 001b.0dec.59c0

src 80.68.48.2 001b.0dec.59c0

and OUI 001b0d is Cisco systems.

probably it is a MAC address in your core switch and it is misbehaving.

one of the two devices is not working correctly.

Hope to help

Giuseppe

Giuseppe

80.68.43.2 and 80.68.48.2 are the IP addresses on one of two core switches, they belong to interfaces VLANs 303 and 301 respectively.

Each of those VLANs has an HSRP group configured (with different IDs) on it as well with IPs of 80.68.43.1 and 80.68.48.1

The IPs are legitimate.

I am quite sure that I do not have any miss-configured access ports, or improperly formed dot1q trunks.

The other problem is that the core switch in question seems to flood ARP requests to this IP ranges regularly. I have already changed MAC and CAM tables time outs to 14400, but it did not help. So at the moment every device that has access to the management VLAN 101 gets hit by this traffic.

Hello Victor,

1 VIP, 2 and 3 on the core switches is really classic we do the same.

I would examine L2 topology with

sh spanning-tree vlan 101

sh spanning-tree vlan 303

sh spanning-tree vlan 301

looking for any strange info.

if this confirms no problem at layer 2 I would consider:

reloading the misbehaving switch

looking for ARP related bugs that can apply to your device.

if also a reload doesn't fix and you don't find known bugs that match what you see I would open a TAC service request.

hint:

have you enabled or disabled ip proxy-arp on the involved SVIs ?

you use different HSRP ids : how many groups are you using ?

C6500 should be fine also with one hundred of different groups but this is not true for less powerful platforms.

Hope to help

Giuseppe

Giuseppe

Thank you.

I will go through L2 again in case I did miss something.

no ip proxy-arp is set on all routed interfaces.

There are currently 7 HSRP groups, so it is way below the limit. In any case I would expect the 6509E with a Sup7203BXL and 1Gb of RAM to be able to cope with it.

Any idea why it would generate so many ARP retransmissions in a particular IP range?

Regards,

Hello Victor,

>> Any idea why it would generate so many ARP retransmissions in a particular IP range?

No, it looks like wrong.

another aspect to verify is the service modules that are on the chassis.

We had a problem with a CSM pair that under heavy traffic load conditions was leaking random mac addresses causing a mac attack to access-layer switches.

We opened a case it was a bug and we solved with a CSM software upgrade.

So the problem can be also in a service module if you have any FWSM, ACE or CSM or others.

Hope to help

Giuseppe

we have ACE appliances connected to both core switches, but I have shut the interfaces to them down for now.

I have noticed that my 6509s share one MAC address among all the VLANs. While it is normal, as far as I understand, do you think it may confuse the switches downstream? I have Cat3560s 24G-TS there.

Hello Victor,

the usage of the same MAC address on all vlans is not a problem for other devices.

the messages are sourced by ip addresses of one switch as we noted in previous posts.

I guess you have external stand-alone ACE appliances.

Hope to help

Giuseppe

Leaky vlans?

I hate when that happens.

You use ACE?

Great, ACE Hardware makes a great bucket. Rubberized and rugged to hold all that yukiness from the leaking vlans.

Iamav

That sounds helpful, but I can't quite figure out what to do with your advice :)

I have disconnected both ACE 4710s by the way, so they are not on the network at the moment.

Regards,

Can HSRP be responsible for cross-vlan talk?

Hello Victor,

>> Can HSRP be responsible for cross-vlan talk?

No, it should stay confined in the origin vlan.

HSRP messages are sent as UDP on multicast destination 224.0.0.2 all routers on subnet with TTL 1.

So a correct implementation shouldn't be able to cross-talk between different Vlans.

One thing you could try is to open the network leaving only one core router connected to access layer switches and to see if anything changes

Hope to help

Giuseppe

Hope to help

Giuseppe