6500 CAM Problem

[chrisayres]

I am having problems where it appears that the 6500 is not learning MAC addresses correctly. If I do a ping from the MSFC on an 6500 to a device connected to another 6500 I would expect the MAC address to go into the CAM as soon as I do this but sometimes it doesn't. I am also seeing this traffic being flooded to all ports in the VLAN.

see below

HAVC6501> (enable) sho cam 00-09-6b-1a-b3-90

* = Static Entry. + = Permanent Entry. # = System Entry. R = Router Entry.

X = Port Security Entry $ = Dot1x Security Entry M = Mac-Auth-Bypass Entry

VLAN Dest MAC/Route Des [CoS] Destination Ports or VCs / [Protocol Type]

---- ------------------ ----- -------------------------------------------

Total Matching CAM Entries Displayed = 0

HAVC6501> (enable) sess 15

Trying Router-15...

Connected to Router-15.

HAVR6501>ping 10.4.123.142

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.4.123.142, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

HAVR6501>sho ip arp 10.4.123.142

Protocol Address Age (min) Hardware Addr Type Interface

Internet 10.4.123.142 86 0009.6b1a.b390 ARPA Vlan102

HAVR6501>exit

HAVC6501> (enable) sho cam 00-09-6b-1a-b3-90

* = Static Entry. + = Permanent Entry. # = System Entry. R = Router Entry.

X = Port Security Entry $ = Dot1x Security Entry M = Mac-Auth-Bypass Entry

VLAN Dest MAC/Route Des [CoS] Destination Ports or VCs / [Protocol Type]

---- ------------------ ----- -------------------------------------------

Total Matching CAM Entries Displayed = 0

HAVC6501> (enable)

This is happening on more than 1 IP address and on more than 1 6500. The only 6500 which consistently has the correct info is the only that has the device directly connected.

Any Ideas

[ankbhasi]

Hi Friend,

Do you have a trunk between 2 cat6500 switches. I mean from the switch where you initiate ping and the switch on which you have device directly plugged into?

Regards,

Ankur

[chrisayres]

Ankur

Yes, It is a .1q trunk allowing all VLANs over an lacp-channel

[ankbhasi]

Hi Chris,

Can you issue a command

"sh mac-address-table interface port-channel " and see if you get the mac address of the devices learned from remote cat6500 after you initiate ping from your cat6500.

Ankur

[chrisayres]

Ankur,

I cannot find a command like this or any other command that would show this info. We are running CatOS

Mod Port Model Serial # Versions

--- ---- ------------------- ----------- ------------------

1 2 WS-X6K-SUP2-2GE SAD060300GH Hw : 3.15

Fw : 7.1(1)

Fw1: 6.1(3)

Sw : 8.5(2)

Sw1: 8.5(2)

[ankbhasi]

Hi Chris,

Can you issue "sh cam dynamic " where mod/port will be the trunk port between the switches?

Ankur

[chrisayres]

Ankur,

See below, the MAC does appear in the mod/port

but not in sho cam. But even though the MAC is on the port the switch is still flooding the traffic to this MAC rather than send it just the the 4/14, 5/14

HAVC6501> (enable) sess 15

Trying Router-15...

Connected to Router-15.

HAVR6501#ping 10.4.123.142

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.4.123.142, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms

HAVR6501#sho ip arp 10.4.123.142

Protocol Address Age (min) Hardware Addr Type Interface

Internet 10.4.123.142 78 0009.6b1a.b390 ARPA Vlan102

HAVR6501#exit

HAVC6501> (enable) sho cam dyn 4/14 ?

| Output modifiers

HAVC6501> (enable) sho cam dyn 4/14 | i b3-90

102 00-09-6b-1a-b3-90 4/14,5/14 [ALL]

HAVC6501> (enable) sho cam 00-09-6b-1a-b3-90

* = Static Entry. + = Permanent Entry. # = System Entry. R = Router Entry.

X = Port Security Entry $ = Dot1x Security Entry M = Mac-Auth-Bypass Entry

VLAN Dest MAC/Route Des [CoS] Destination Ports or VCs / [Protocol Type]

---- ------------------ ----- -------------------------------------------

Total Matching CAM Entries Displayed = 0

HAVC6501> (enable) sho cam dyn 4/14 | i b3-90

102 00-09-6b-1a-b3-90 4/14,5/14 [ALL]

HAVC6501> (enable)

[chrisayres]

Ankur,

Does the above help you any or would you like me to do some further commands

Regards

Chris

[ankbhasi]

Hi Chris,

How did you confirm that the traffic for this mac is flooded on all the ports for tha vlan? Did you sniffed the ports or did something like that?

Ankur

[chrisayres]

Ankur,

Yes I mirrored a port in the same VLAN and put a sniffer on it. I saw unicast traffic for 10.4.123.142 on a port that shouldn't see that traffic.

Also 10.4.123.142 is a tape silo and receives large amounts of backup data over night. I have Infovista monitoring the switches and when this backup traffic is happening Infovista shows an increase to traffic on other ports in the VLAN aswell.

However the Infovista graph for the tape silo shows a constant 250Mb/s of inbound traffic for the duration of the backups (about 2 hrs). But the graphs for other ports show the traffic going between normal levels then spiking up to 250Mb/s every 15 minutes or so.

[glen.grant]

Its possible you are getting some unicast flooding going on in your network , here is a link to show you how this might happen , http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801d0808.shtml

[chrisayres]

Ankur,

I have been looking at this a bit more.

I have sat on the one of the 6500's and repeatedly done

sho cam count dyn

When I started the count was at 1338 then over the next 10 mins it slowly went up to 1461 adding 1 or 2 new cam entries every time I did the command. But then it lost 138 cam entries in the time it took me the repaet the command (ie less than 1 second)

Two things I find strange about this

1. It is Sunday morning so there should few users logging on, so why is the cam table increasing (also most devices connected to the 6500 L2 network are servers which should be up all the time)

2. Why does the switch lose 138 entries at the same time, I find it unlikely that they were all learnt at the same time so why did they reach the 300 sec expiry time together.

I have done this a few times now and it seems to go through the same cycle every 7 to 10 mins of slowly learning then dropping 130 - 200 entries all at the same time.

I haven't done it on the other 3 6500's but from what I have seen before I expect they are doing the same