cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
451
Views
0
Helpful
4
Replies

3550 switchport problems - brought on by Source Guard?

ICTNetworking
Level 1
Level 1

We have about 250 Catalyst 3550 48-port switches. We recently enabled IP Source Guard, and this change has caused very peculiar symptoms on about 5-10% of the switches. Everything works fine for a while, but after an unpredictable amount of time, the switchports appear to negotiate to an incorrect speed/duplex. The hosts connected to these switchports negotiate to 100/full, but the Catalyst switch uses 10/full. All of the expected results of a negotiation mismatch start to appear (output buffer failures, underruns). Then it gets stranger. A shut/no shut of the port does not resolve the problem. Half of the time, the port goes back to 10/full, and the other half of the time, it simply stops working (reporting notconnect). There doesn't seem to be any commonality on the client end, and all clients we've investigated are properly set for auto-negotiation. This problem definitely occurred after the Source Guard implementation. Simply removing 'ip verify source' also does not resolve the problem. Instead, a hardware reboot of the switch is necessary. We've also found that Source Guard must be removed from any interfaces that are connected prior to the reboot, or the switch will crash on boot (Source Guard can be re-enabled once the boot is complete). In some areas, we've had to remove Source Guard permanently in order to prevent the negotiation problem from recurring.

This problem has occurred for us on 12.2(44)SE2, 12.2(44)SE3 and 12.2(46)SE.

I'm hesitant to pursue hardware replacement on this many switches, because I expect that this is a bug of some sort. Has anyone else experienced this issue?

Below is an example of a common switchport config. The 'ip verify source' is the only addition that seems to bring this problem on, although I readily admit that if this is related to ASIC oversubscription in some way, Source Guard could theoretically have simply been the last straw. On the other hand, we don't get any error messages either, and this behavior seems to be more bug-like than a resource problem.

interface FastEthernet0/1

switchport access vlan 127

switchport mode access

switchport port-security

switchport port-security violation restrict

ip access-group al_127 in

no logging event link-status

wrr-queue bandwidth 5 70 25 1

wrr-queue min-reserve 1 8

wrr-queue min-reserve 2 6

wrr-queue min-reserve 3 5

wrr-queue min-reserve 4 7

wrr-queue cos-map 1 1

wrr-queue cos-map 2 2 3 4 6 7

wrr-queue cos-map 3 0

wrr-queue cos-map 4 5

priority-queue out

no cdp enable

spanning-tree portfast

spanning-tree bpduguard enable

service-policy input qos-standard-voice

ip igmp max-groups 8

ip igmp filter 101

ip verify source

ip dhcp snooping limit rate 100

4 Replies 4

Giuseppe Larosa
Hall of Fame
Hall of Fame

Hello Ralph,

I would open a TAC service request.

You have collected enough data and evidence of the problem.

I agree that I would expect other problems from ip source guard like high cpu usage or intermittent connectivity but again 25 switches of 250 seems too high percentage to be an hardware problem. Also you have seen that by disabling ip source guard and rebooting the ports are able again to negotiate at 100 full.

Also the fact that rebooting with ip source guard enabled causes a crash is a sign of a problem with this feature.

C3550 are good switches but I think they were designed before the introduction of ip source guard feature.

Another suggestion: what happens if you try to fix speed and duplex to 100 full ?

Hope to help

Giuseppe

Giuseppe,

to answer your question, the results of fixing speed/duplex are inconsistent. Some ports will remain connected at 100/full, while others will go into a notconnect state. Of the latter, some will return to 10/full when auto-negotiation is restored, and others will remain in a notconnect state thereafter until the next reboot.

Giuseppe,

I neglected to thank you for your reply: thanks!

I can also confirm that the switch CPU is normal. Other than the abnormal switchport behavior, everything seems fine.

mullzkBern_2
Level 1
Level 1

What Devices are behind the Faulty Switchports? We had the same speed-problem here (although no problems on switch-boot and our switches are c3560) - but not ip verify source was the culprit, but a firmware-upgrade on our alcatel iptouch-phones - since we are back on 3.71.20, everything works fine again.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: