Switches doesn't learn mac addresses of desktops without arp req

r.gogunskiy · ‎02-10-2012

Hello Friends,

First of all I'm sorry for my bad English.

I have a very strange situation with unicast flooding. Switches doesn't learn mac addresses of desktops without arp requsts. The network has a star topology: one core switch (4507) and access switches (2960) are conected to the core switch. The laptop with wireshark recieves traffic with dst mac addresses of another desktops which are connected to another access switches. I've checked spaning tree topology - no loops, mac address tables on all switches is not full.

I found that the switches forget mac addresses in 300 seconds (default mac aging time) and don't learn mac addresses again without arp requests.

we have 2 vlans: data and voice vlans.

all desktops and all management ip addresses of all switches are in the data vlan

The test desktop (x.y.z.36) was connected to the SWITCH_C.

The desktop with wireshark was connected to the SWITCH_B

I pinged a test desktop and checked arp entries and mac entries on SWITCH_A, CORE and SWITCH_C switches

1. Everythins is OK, the switch knows the mac address of the x.y.z.36 desktop

SWITCH_A#show clock

12:00:25.172 UTC Fri Feb 10 2012

SWITCH_A#show mac address-table | i a.b.60d4

10 a.b.60d4 DYNAMIC Gi0/1

2. I tried to ping the x.y.z.36 desktop. Ping was success.

SWITCH_A#ping x.y.z.36

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to x.y.z.36, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/210/1040 ms

3. I checked an arp table and a mac table. Everything is OK

SWITCH_A#show arp | i x.y.z.36

Internet x.y.z.36 0 a.b.60d4 ARPA Vlan10

SWITCH_A#showshow mac address-table | i a.b.60d4

10 a.b.60d4 DYNAMIC Gi0/1

4. I waited about 6 minutes and checked an arp table and a mac table. The switch didn't have anymore mac address of the

x.y.z.36 desktop and had arp entry for that desktop. SWITCH_C and CORE switches removed the mac address also. Everything is OK

SWITCH_A#show mac address-table | i a.b.60d4

SWITCH_A#show clock

12:06:51.232 UTC Fri Feb 10 2012

SWITCH_A#show mac address-table | i a.b.60d4

SWITCH_A#show arp | i x.y.z.36

Internet x.y.z.36 5 a.b.60d4 ARPA Vlan10

5. I tried to ping the x.y.z.36 desktop. Ping was success.

SWITCH_A#ping x.y.z.36

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to x.y.z.36, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/40/184 ms

6. I checked an arp table and a mac table. The switch had arp entry for the desktop, but the switch didn't learn the mac address. SWITCH_C and CORE switches didn't learn the mac address too.

SWITCH_A#show arp | i x.y.z.36

Internet x.y.z.36 5 a.b.60d4 ARPA Vlan10

SWITCH_A#show mac address-table | i a.b.60d4

SWITCH_A#

SWITCH_A#show mac address-table | i a.b.60d4

7. I tried to ping the x.y.z.36 desktop again . Ping was success but the switches didn't learn the mac address again

SWITCH_A#ping x.y.z.36

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to x.y.z.36, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/7/17 ms

SWITCH_A#show mac address-table | i a.b.60d4

8. I cleared arp cache on the switch

SWITCH_A#clear arp

9. And the switches learnt the mac address

SWITCH_A#show mac address-table | i a.b.60d4

10 a.b.60d4 DYNAMIC Gi0/1

SWITCH_A#show clock

12:09:53.391 UTC Fri Feb 10 2012

I saw ping requests during 5-7 steps on the desktop with wireshark.

Software versions:

Access switches: c2960-lanbase-mz.122-53.SE1

The core switch: cat4500-ipbasek9-mz.122-54.SG

Does anyone have any ideas what is going on in my network?

Thank you.

Best regards,

Ruslan

rsimoni · ‎02-11-2012

Hi Ruslan,

what you write is pretty weird indeed.

First thing I think of is that the PC responds to ARP request with a given MAC [a.b.60d4] but replies to ICMP with another one.

We can easily check this out as you have a wireshark pc already connected.

the idea is that if no intermediate switch has the PC's MAC address in its table unkown unicast flooding for that mac will occur, therefore the wireshark pc, connected in vlan 10 (or in a trunk port) should receive it.

therefore if you repeat the test you should see the ping replies also on the wireskark, does this happen?

If yes, can configrm that the mac is the same you see in arp replies?

If not, it would mean that the PC replies, for some bizarre reason, using a different mac. You need then to check that mac against the switch mac address table.

Riccardo

amikat · ‎02-13-2012

Hi,

Can you please post the "show arp | i x.y.z.36" and "show mac address-table | i a.b.60d4" commands outputs from the SWITCH_A, CORE and SWITCH_C boxes when

1) both arp and mac information is present

and then

2) after the mac timeout expires and you succeed with ping.

Also will you please post the SWITCH_C port configuration where the x.y.z.36 host is connected. Can you please confirm that a.b.60d4 is indeed the mac address of the x.y.z.36 host.

Thanks & Regards,

Antonin

r.gogunskiy · ‎02-13-2012

Hello

Yes, the a.b.60d4 mac address was a mac address of the x.y.z.36 host.

I'm trying to reproduce the issue today without any success.

This behavior happens not every day. But when it happens we see a lot of unicast flood with different dst mac addresses.

============================================================================

Switches knew about the mac address:

SWITCH_A#show clock

12:00:25.172 UTC Fri Feb 10 2012

SWITCH_A#show mac address-table | i a.b.60d4

111 a.b.60d4 DYNAMIC Gi0/1

SWITCH_A#ping x.y.z.36

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to x.y.z.36, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/210/1040 ms

SWITCH_A#show arp | i x.y.z.36

Internet x.y.z.36 0 a.b.60d4 ARPA Vlan111

CORE#show clock

12:00:11.606 UTC Fri Feb 10 2012

CORE#show mac add | i a.b.60d4

111 a.b.60d4 dynamic ip GigabitEthernet3/34

CORE#

SWITCH_C#show clock

12:00:18.239 UTC Fri Feb 10 2012

SWITCH_C#show mac add | i a.b.60d4

111 a.b.60d4 DYNAMIC Fa0/1

============================================================================

Switches forgot the mac address and didn't learn it

SWITCH_A#show clock

12:06:51.232 UTC Fri Feb 10 2012

SWITCH_A#show mac address-table | i a.b.60d4

SWITCH_A#show arp | i x.y.z.36

Internet x.y.z.36 5 a.b.60d4 ARPA Vlan111

SWITCH_A#ping x.y.z.36

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to x.y.z.36, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/40/184 ms

SWITCH_A#show arp | i x.y.z.36

Internet x.y.z.36 5 a.b.60d4 ARPA Vlan111

SWITCH_A#show mac address-table | i a.b.60d4

SWITCH_A#

SWITCH_A#show mac address-table | i a.b.60d4

SWITCH_A#ping x.y.z.36

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to x.y.z.36, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/7/17 ms

SWITCH_A#show mac address-table | i a.b.60d4

CORE#show clock

12:07:32.522 UTC Fri Feb 10 2012

CORE#show mac address-table | i a.b.60d4

SWITCH_C#show clock

12:07:43.701 UTC Fri Feb 10 2012

SWITCH_C#show mac address-table | i a.b.60d4

=================================================================================

Arp cache was cleared on the SWITCH_A ( show clock output from SWITCH_A is more recent because I cleared arp cache and checked mac address table and than

checked mac address table on other switches. and only after that I checked time on the SWITCH_A).

SWITCH_A#clear arp

SWITCH_A#show mac address-table | i a.b.60d4

111 a.b.60d4 DYNAMIC Gi0/1

SWITCH_A#show clock

12:09:53.391 UTC Fri Feb 10 2012

CORE#show mac address-table | i a.b.60d4

111 a.b.60d4 dynamic ip GigabitEthernet3/34

CORE#show clock

12:09:45.306 UTC Fri Feb 10 2012

SWITCH_C#show mac address-table | i a.b.60d4

111 a.b.60d4 DYNAMIC Fa0/1

SWITCH_C#show clock

12:09:49.330 UTC Fri Feb 10 2012

Configuration of the port of the SWITCH_C switch :

interface FastEthernet0/1

switchport access vlan 111

switchport mode access

switchport voice vlan 222

srr-queue bandwidth share 10 10 60 20

priority-queue out

mls qos trust cos

storm-control broadcast level pps 240 120

auto qos voip trust

spanning-tree portfast

end

Best regards,

Ruslan

amikat · ‎02-13-2012

Hi,

Thank you for the information provided.

I understand that due to the fact this behaviour appears only sometimes you have used the outputs from 10th February in both of your posts. What confuses me a bit is this: the time for the SWITCH_A output appears to be identical for both your original and latest posts:

SWITCH_A#show clock

12:00:25.172 UTC Fri Feb 10 2012

SWITCH_A#show mac address-table | i a.b.60d4

10 a.b.60d4 DYNAMIC Gi0/1

SWITCH_A#show clock

12:00:25.172 UTC Fri Feb 10 2012

SWITCH_A#show mac address-table | i a.b.60d4

111 a.b.60d4 DYNAMIC Gi0/1

But while your host belongs to Vlan 10 in the original post, it is part of Vlan 111 in your latest post.

Can you please make any comment?

Thanks & Regards,

Antonin

r.gogunskiy · ‎02-13-2012

Hello,

Output in my second post is the same as in the first. I just added outputs from SWITCH_C and CORE switches.

> But while your host belongs to Vlan 10 in the original post, it is part of Vlan 111 in your latest post.

> Can you please make any comment?

I'm sorry, I had forgotten to do find/replace for vlans on my second post. So the data vlan is 111 in both posts. The mac address a.b.60d4 is in vlan 111.

amikat · ‎02-14-2012

Hi,

Thanks for the reply.

Next time you experience this issue will you please try:

1) ping from SWITCH_C for host x.y.z.36 and vice versa from host x.y.z.36 to SWITCH_C,

2) try traceroute from SWITCH_A instead of ping,

3) try telnet, ssh, http or https (whichever you have configured) from host x.y.z.36 to SWITCH_C and SWITCH_A,

and see if there is any change.

Thanks & Regards,

Antonin

Switches doesn't learn mac addresses of desktops without arp request.