CAT 4006 quuestion - invalid source address issue

brenteverett · ‎09-10-2008

Hello All,

We have a CAT 4006 that is experiencing multiple instances of this error in the log:

Sep 10 10:55:51: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 238722 times)Packet received with invalid source MAC address (00:00:00:00:00:00) on port Gi5/24 in vlan 1

We started receiving this error after our conversion from a Sup II engine (CATOS) to a Sup IV engine (IOS). I only receive the message on that one particular interface which happens to be connected to our firewall (ASA5510 on the inside interface). Any thoughts on how I should go about troubleshooting this?

bmcginn · ‎09-10-2008

Hi there,

Have you tried a monitor session so you can capture all packets on that link.. Something like ethereal/wireshark will get a good enough capture for you too look at.

If you do that you ought to be able to see the inside of the offending frame, maybe even an IP or multiple IP addresses??

On the cisco error message decoder it firmly points the blame at the thing on the other end of the link.. aka your ASA. If thats the case, there is a capture packet function on the ASA which can also perhaps shed some light.

Good luck!

Brad

brenteverett · ‎09-11-2008

I seem to remember seeing some instructions on how to do this but that seems like a good idea. I'll have to look at that again, also at one point I did stick a HUB in between the firewall and our ISP's switch and ran a filtered capture with wireshark for an all 0 mac address however now that I think about I had the hub in the wrong place!

I'll look at setting up monitoring on this port. Thanks!

bmcginn · ‎09-11-2008

The command is:

monitor session 1 source interface [interface to mirror]

monitor session1 destination interface [interface to mirror to]

I think the asa command is:

capture [capt_name] interface [inside|outside]

Giuseppe Larosa · ‎09-10-2008

Hello Allen,

we receive this kind of messages sometimes both on IOS based or CatOS based switches.

One possible source of frames with all zeros source MAC are just initiliazed unconfigured VMware instances.

In your case the port connects to a firewall ASA, how is it configured is it performing any bridging ?

the number of events is quite high.

this is what says the error message decoder tool:

1. %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid source MAC address ( [mac-addr] ) on port [char] in vlan [dec]

A packet was received with an all zero or a multicast source address. The packet is treated as invalid and no learning is done. Excessive flow of such packets can waste CPU cycles. This message is rate-limited and is displayed only for the first such packet received on any interface or VLAN. Subsequent messages will display cumulative count of all such packets received in given interval on all interfaces.

Recommended Action: Check the switch configuration file to find the source of these packets on the specified port and take corrective action to fix them at the source end. You can also enable port security on that interface to shutdown the port if the incoming rate of packets with invalid source mac address is too high by issuing the switchport port-security limit rate invalid-source-mac command.

Related documents- No specific documents apply to this error message.

Hope to help

Giuseppe

brenteverett · ‎09-11-2008

We do have a VMware virtual environment (3 ESX Host servers, each with 2 trunked NIC's attached to our 4006 switch for the actual virtual machines) and I remember seeing some post in the VMware forums about this a while back. I'll dig them up and see what I find. Thanks for the hint :) However none of the VMware servers are connected to the offending port (just our firewall).

A couple of other things that I've found "may" be contributing to this. I ran across a post on another forum that was pointing to newer Mac's and Vista machines that have IPv6 enabled but not used. I went around yesterday and disabled this on the few mac's and 1 vista machine we have. I'll wait to see if that makes a difference.

Also, I noticed a route statement that directs traffic to the firewall on our router:

ip route 0.0.0.0 0.0.0.0 10.1.1.120

I know that this is necessary since the next hop from the router to the internet is our firewall however I'm guessing if the router doesn't know what to do with that suspect traffic it's forwarding it to the firewall.

Giuseppe Larosa · ‎09-11-2008

Hello Allen,

the error message is related to received frames and they are offending at OSI layer 2.

the default static route is necessary and influences outgoing traffic on the interface.

you could try to SPAN the traffic on the port and to send it to a protocol analyzer connected to another port: in this way you can determine if the frames really exist and if they have an upper layer info inside that can help to find their source.

http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008015c612.shtml

Hope to help

Giuseppe

brenteverett · ‎09-12-2008

I'm going to attemp setting up the SPAN this weekend and see what happens. However since I've disabled IPv6 on our Mac's with OS X and our Vista PC I'm getting significantly less of these error messages. Here's the log from this morning:

*Sep 11 07:57:54: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 4 times)Packet received with invalid source MAC address (00:00:00:00:00:00) on port Gi5/24 in vlan 1

*Sep 11 11:16:30: %C4K_EBM-4-HOSTFLAPPING: Host 00:0F:B3:F5:34:D0 in vlan 1 is flapping between port Gi4/17 and port Po3

*Sep 11 11:18:30: %C4K_EBM-4-HOSTFLAPPING: Host 00:0F:B3:F5:07:CC in vlan 1 is flapping between port Po3 and port Gi4/17

*Sep 11 13:58:26: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 12 times)Packet received with invalid source MAC address (00:00:00:00:00:00) on port Gi5/24 in vlan 1

*Sep 11 15:01:33: %C4K_EBM-4-HOSTFLAPPING: Host 00:0F:B3:F5:0D:70 in vlan 1 is flapping between port Gi4/17 and port Po3

*Sep 12 04:59:35: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 1 times)Packet received with invalid source MAC address (00:00:00:00:00:00) on port Gi5/24 in vlan 1

I still want to find out where it's coming from but it would seem the volume of errors I was getting was related to those suspect workstations.

brenteverett · ‎09-15-2008

Ok, I'm still getting the message however the frequency has been reduced tremendously. here's the log from this morning:

*Sep 11 07:57:54: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 4 times)Packet received with invalid source MAC address (00:00:00:00:00:00) on port Gi5/24 in vlan 1

*Sep 11 11:16:30: %C4K_EBM-4-HOSTFLAPPING: Host 00:0F:B3:F5:34:D0 in vlan 1 is flapping between port Gi4/17 and port Po3

*Sep 11 11:18:30: %C4K_EBM-4-HOSTFLAPPING: Host 00:0F:B3:F5:07:CC in vlan 1 is flapping between port Po3 and port Gi4/17

*Sep 11 13:58:26: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 12 times)Packet received with invalid source MAC address (00:00:00:00:00:00) on port Gi5/24 in vlan 1

*Sep 11 15:01:33: %C4K_EBM-4-HOSTFLAPPING: Host 00:0F:B3:F5:0D:70 in vlan 1 is flapping between port Gi4/17 and port Po3

*Sep 12 04:59:35: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 1 times)Packet received with invalid source MAC address (00:00:00:00:00:00) on port Gi5/24 in vlan 1

I'm still looking into SPANing that port but at least I know that IPv6 is partly the culprit.

brenteverett · ‎09-25-2008

Ok, I setup a SPAN on the port in question and ran a packet capture via whireshark with an all 0 mac address filter in place for a few days. I did get some content from that however I'm not sure how to interpret it. All of the results are from one server which would happen to be our Exchange server which is also a virtual server (VMware ESX 3.5). Any thoughts?

Giuseppe Larosa · ‎09-25-2008

Hello Allen,

you need to check the configuration of VMware according to other posts a vmware instance just initialized and not configured uses all 0s source MAC.

Verify or have someone verify the vmware looking for a pending VM not used

Hope to help

Giuseppe