Solved: Re: MAC Address not captured on switch ports

esamaniego · ‎03-01-2012

In my environment we have 3750x switches running ios 15.0 (1) SE2. We have port security mac address sticky configured on all our switch ports. I noticed that we have several interfaces (on different switches) that are up but have not captured the MAC address from the workstation. Here is one example:

interface GigabitEthernet2/0/11

switchport mode access

switchport port-security

switchport port-security mac-address sticky

spanning-tree portfast

end

SWIITCH(config)#do show mac address-t int g2/0/11
Mac Address Table
-------------------------------------------

Vlan Mac Address Type Ports
---- ----------- -------- -----

SWITCH(config)#do show int g2/0/11

GigabitEthernet2/0/11 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is 70ca.9bca.760b (bia 70ca.9bca.760b)

MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input never, output 00:00:00, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 5454502

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 63000 bits/sec, 51 packets/sec

0 packets input, 0 bytes, 0 no buffer

Received 0 broadcasts (0 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

56272056 packets output, 9223565276 bytes, 0 underruns

0 output errors, 0 collisions, 2 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

Kyle McKay · ‎03-20-2012

I've recently experienced the same issue on a 3750 stack. The switch was initially installed however the first 12 ports would not capture the MAC address of directly connected hosts. Rebooting the switch changed nothing.

Removing the "switchport port-security" command on the individual interfaces and then re-applying it, solved the problem for me.

View solution in original post

Jan Hrnko · ‎03-01-2012

Hi Erik,

please can you also post output from this command?

show port-security interface g2/0/11

Best regards,

Jan

esamaniego · ‎03-01-2012

Jan, here is the output:

SWITCH#show port-security int g2/0/11

Port Security : Enabled

Port Status : Secure-up

Violation Mode : Shutdown

Aging Time : 0 mins

Aging Type : Absolute

SecureStatic Address Aging : Disabled

Maximum MAC Addresses : 1

Total MAC Addresses : 0

Configured MAC Addresses : 0

Sticky MAC Addresses : 0

Last Source Address:Vlan : 0021.9b38.1ea9:1

Security Violation Count : 0

Reza Sharifi · ‎03-01-2012

When a bridge receives a BPDU with the TC bit set from a neighbor, these occur:

It clears the MAC addresses learned on all its ports, except the one that receives the topology change.

It starts the TC While timer and sends BPDUs with TC set on all its designated ports and root port (RSTP no longer uses the specific TCN BPDU, unless a legacy bridge needs to be notified).

So, do a sh spann vlan xx detail and see if there is a lots of topology changes

http://www.cisco.com/en/US/tech/tk389/tk621/technologies_white_paper09186a0080094cfa.shtml

I also have seen some IP camera decoders that in order to see the MAC in the MAC table, you have to ping the device first.

HTH

Carlos190 · ‎03-01-2012

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 63000 bits/sec, 51 packets/sec

0 packets input, 0 bytes, 0 no buffer

Received 0 broadcasts (0 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

56272056 packets output, 9223565276 bytes, 0 underruns

From that output I can see that there are no packets coming in to the interface, and many packets out of it. A switch will not learn a MAC address if it doesn´t receive any packet from the host.

I´m curious on why it is not receiving any packets, I´ve seen this in a load-balancing configuration on the NICs(on servers for example, one NIC receives packets and the other NIC sends) but since this is a workstation I really doubt this is the cause.

So, the first question to ask is why the host is not sending any packets through this interface?

You may check NIC config, if you are using any sort of redundancy let us know.

Carlos

esamaniego · ‎03-01-2012

Carlos, I noticed that also, however since there are several ports behaving this way (on different switches) I check the other ports and saw that input packets are detected. All of these ports have workstations (Windows 7) connected to them. I checked a few of the workstations and there are no network issues (just the MAC capture issue).

Carlos190 · ‎03-05-2012

I see, then it would be interesting to see the output of a debug arp command if it´s supported by your switch, as always recommended, if you are going to issue that command be careful of not doing it when the switch already has a lot of work to do because it overloads the processor, and the debug arp comes with an option of debugging just one interface so I would start there.

MichaelJAssels · ‎03-20-2012

Erik, we see exactly the same problem on stacks of 3750X switches running IOS 15.0 (1) SE2 (ipbase image). It appears to affect only ports on stack members 2 or higher; we've configured stack member 1 as master in all cases, but I don't know if this is relevant. In all cases the affected ports are "secured" to a single computer, and are configured with "spanning-tree portfast". Here is a typical case:

#sh ru int g 2/0/1

interface GigabitEthernet2/0/1

switchport access vlan 502

switchport mode access

switchport port-security

switchport port-security violation restrict

switchport port-security mac-address sticky

speed auto 100

spanning-tree portfast

#sh port-security int g 2/0/1

Port Security : Enabled

Port Status : Secure-up

Violation Mode : Restrict

Aging Time : 0 mins

Aging Type : Absolute

SecureStatic Address Aging : Disabled

Maximum MAC Addresses : 1

Total MAC Addresses : 0

Configured MAC Addresses : 0

Sticky MAC Addresses : 0

Last Source Address:Vlan : 0021.86fa.95c4:502

Security Violation Count : 0

#sh mac address-table int g 2/0/1

Mac Address Table

-------------------------------------------

Vlan Mac Address Type Ports

---- ----------- -------- -----

These commands were run while the attached computer was happily answering "ping". There doesn't appear to be anything at all wrong with the computer's network connectivity.

Something that may or may not be relevant: With "debug port-security", logging shows many lines like this:

Mar 20 10:05:22.794: PSECURE: psecure_delete_address_not_ok: no port security subblock for Po48

Mar 20 10:05:23.817: PSECURE: psecure_delete_address_not_ok: no port security subblock for Po48

Mar 20 10:05:24.824: PSECURE: psecure_delete_address_not_ok: no port security subblock for Po48

Mar 20 10:05:25.830: PSECURE: psecure_delete_address_not_ok: no port security subblock for Po48

Po48 is the port-channel uplink, configured as a dot1q trunk, to a distribution switch. I haven't been able to find any reference to this debug message anywhere.

In short, we're seeing the same problem. The only data points I have to add are (a) that it happens only on stack members greater than 1, and (b) that the affected ports are all configured with "spanning-tree portfast", and bpduguard is on, so I don't think BPDUs should be an issue.

I'm stumped.

Michael Assels

vaib7av.shirkul · ‎03-20-2012

Hi Erik,

Hope ip routing is disabled on 3750 switch, use below method to find out interface using mac address database,

Ex- you are trying to find 172.16.1.20 host interface,

First ping from your core switch (L3) to 172.16.1.20 where host gateway (Ex.172.16.1.1) was configured,

then issue sh ip arp 172.16.1.20 on core,

here is your host mac address, then copy mac address and telnet to your host connected switch,

and issue sh mac address add xxxx.xxxx.xxxx

whoh! you have found interface of connected host,

HTH

Dan-Ciprian Cicioiu · ‎03-20-2012

Hi all,

The issue is that the host connected only receives traffic and does not sent - maybe does not send on this interface.

GigabitEthernet2/0/11 is up, line protocol is up (connected)

ARP type: ARPA, ARP Timeout 04:00:00

Last input never, output 00:00:00, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 5454502

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 63000 bits/sec, 51 packets/sec

Having in mind that the FDB is build by looking in the frame received on the port , and source mac address , and looking at the show interface output.

Regards

Dan

MichaelJAssels · ‎03-20-2012

Dan-Ciprian Cicioiu wrote:

The issue is that the host connected only receives traffic and does not sent - maybe does not send on this interface.

Dan,

I think the lack of traffic sent by the host on Erik's Gi2/0/11 interface is a red herring. I see dozens of ports with the same problem -- i.e., MAC addresses not captured on switch ports -- but all of them send and receive traffic normally. Again, I emphasize that this only happens on stacked 3750X switches, and never on the stack master (or at least, never on stack member 1, which happens to be the master). I should also add that it happens on very roughly half the access ports that have link and line protocol up, and there doesn't appear to be any obvious pattern to the distribution of ports with the problem.

I can't speak for Erik, who originally raised the issue, but it certainly appears that he and I are seeing the same thing.

Regards,

Michael

esamaniego · ‎03-23-2012

Thanks Kyle, removing and reapplying the port-security commands looks like it fixed the issue.

MichaelJAssels · ‎03-23-2012

Hi Erik,

Are you sure the problem is fixed? I was able to "bring back" a port's sticky MAC address by removing and reapplying the port-security commands, but it disappears again as soon as there's a significant change to the port's state. In particular, if you unplug that attached host and then plug it in again, the MAC address disappears again. I get the same result by changing the VLAN and then changing it back, or by shutting down the port and then bringing it back up.

Michael

Kyle McKay · ‎03-20-2012

I've recently experienced the same issue on a 3750 stack. The switch was initially installed however the first 12 ports would not capture the MAC address of directly connected hosts. Rebooting the switch changed nothing.

Removing the "switchport port-security" command on the individual interfaces and then re-applying it, solved the problem for me.

MichaelJAssels · ‎03-20-2012

Kyle McKay wrote:

[...]

Removing the "switchport port-security" command on the individual interfaces and then re-applying it, solved the problem for me.

Hmm. That looked really promising, but unfortunately, it didn't work for me.

In case in stimulates anyone's memory, I have more snippets of logging output from "debug port-security":

[...]

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/15

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/16

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning 0x7E3FE6C<001a.a015.6011:540> on Gi2/0/17

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/17

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/18

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning 0x7DEEE84<001a.a016.5892:540> on Gi2/0/19

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/19

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning 0x7DD808C<001a.a015.498a:540> on Gi2/0/20

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/20

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/22

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning 0x7B16A94<0000.a703.674b:601> on Gi2/0/23

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/23

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/33

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning 0x7800AD4<0021.86fa.93b1:540> on Gi2/0/34

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/34

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning 0x7F1D30C<0021.86fa.91f0:540> on Gi2/0/35

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/35

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/36

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/37

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning 0x7C4B634<0021.86fa.933a:540> on Gi2/0/38

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/38

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning 0x7C9AA84<0021.86fa.932d:540> on Gi2/0/39

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/39

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/40

Mar 20 15:08:47: PSECURE: psecure_get_next_mac_from_hat: returning NULL on Gi2/0/41

[... etc., up to Gi6/0/48 ...]

The cases where only "NULL" is returned correspond exactly to the ports that are up and working but apparently without recorded MAC addresses. The "missing" ports (e.g., Gi2/0/21, and Gi2/0/24-32) are not configured for port-security. Again, searching for "psecure_get_next_mac_from_hat" yields no information on either Cisco's site or elsewhere.

Michael