Slow performance after replacing a Catalyst 2828 with a Catalyst2950-24SX

[l.mourits]

Hi Guys,

I have a problem at a customer site which is troubling me and I can't figure it out (allthough I have quite some experience on Catalyst switches). So, Im hoping that someone on this forum can help me out with usefull tips or ideas.

Here's the case:

My customer has one head office and about 26 smaller offices within one city. The sites are connected via Ethernet Leased Lines (ethernet over ATM) which is provided by Priority Telecom (PT). These ELL's have a committed access rate of 2Mbps. For this purpose PT has placed an 2900-XL Catalyst switch and delivers a 10Mb full duplex ethernet interface to the customer.

At the customer site there is an existing situation with HP Procurve hubs (12-port hubs) connected to a Cisco 2828 Catalyst (yep, one of the old days). Within the 2828 are two 100BaseTX modules available and they are conected to the local servers. Port 24 on the Catalyst 2828 is configured for full duplex and connected to the Catalyst 2900-XL of the provider. Client PC's are Windows 2000 and logging in to the doamin is local (but some things within the kix login script are downloaded from the head office).

All this is working fine for over two years now (so far, so good).

Because of a future change of provider and the end-of-support on the Catalyst 2828 customer has decided (based on my advice) to replace the Catalyst 2828 into a Catalyst 2950-24SX (the new provider will provide a Gigabit fiber uplink as physical interface). Later on this year the HP Provcurves will also be replaced but this has to be tendered, so, time forced us to replace the 2828's allready.

I configured the switch in a way that it just the same as the 2828, but prepared for the future new provider. I configured just one vlan (vlan 21 and all ports are set into that vlan. Interface vlan 21 is configured with an IP for maintenance purposes. I did not use vlan 1 bacause it give me the ability to have an easy migration to the new design with the new provider. The interfaces that connect to the HP Procurve hubs are configured as fixed 10Mb half duplex. The interface that are connect to the servers are fixed 100Mb full duplex and to the Catalyst 2900 of the existing provider is configured as 10Mb full duplex. So, everything is the same as on the Catalyst 2828.

But now the problem. As soon as I exchange the Catalyst 2828 with the new Catalyst 2950-24SX users are complaining about very slow performance. So, my first thoughts where a misconfigured link (duplex mismatch) or some not very well patched connection or something like that. But when I checked onsite this morning I saw no errors on the interfaces (no CSC, no alignment). There are of course collisions on the interfaces which the hubs where connected to, but this is also the case on the 2828 (so I assume there's no difference there). Also did check on duplex mismatches and physical cabling, but no luck. Then exchanged the 2950-24SX again with the older 2828 and thing where allright again (performance was okay again). Strange huh?

So, I checked what switching mode the 2828 was in, and this was configured for store-and-forward as it should be. I have searched on CCO and store-and-forward switching is also the default switching mode for the 2950-24SX.

But right now I'm running out of options, cause everything seems fine, but it is not. There is a very big difference on performance when the 2950-24SX or the 2828 is in place, while no errors are seen and no real differences can be discovered. Only thing that I could discover that there was a small difference with the number of timeouts when pinging a address on the headoffice. With the 2828 in place there's about 1 timeout on 40 packets (1500 byte packets) and with the 2950-24SX in place there's about 4 timeout on 40 packets (also 1500 byte packets). Testing with 64 byte packets gives no difference.

I also did the bug search and search on the interoperability issues, but didn't find anything usefull. Anyone has any usefull tips on troubleshooting this case? Or maybe someone has some clues on what could be happening here? I love to here, cause I really don't get it.

Thanks in advance,

Leo

[smcquerry]

Leo,

I have seen some compatibility issues with manually setting the duplex on the 2950's. For some reason they haven't performed well for me in the past, meaning when I set the speed and duplex for certain types of equipment the performance was worse than when it negotiated a-10/a-half and I was seeing no errors logged on the interfaces.

You may want to experiment with duplex settings on the servers and the provider switch if possible. I would focus on the connections between the servers and the switch. If the cards are auto capable let them negotiate if not then try 100/half.

I know this doesn’t sound logical, but as I mentioned I have seen strange behavior with the duplex and speed settings on the 2950 and 3550 products in the past.

Good luck.

[l.mourits]

Thanks for the tip. Indeed it doesn´t sound logical, cause one would expect to see at least some errors then. Experimenting with the provider switch is no option (I have to generate a change request for every change) but I can always do some triel and error on the interfaces on which the servers reside.

I have seen some bugs on earlier releases talking about the behaviour you described. So, just to be sure, was this recently that you experienced this problems?

Thanks in advance for you reply,

Leo

[skarundi]

Every time I troubleshoot slow performance issues, I ask the user to identify one action that is "slow". For example: " If i download this file from this server it take 2 minutes. Now when you use the 2950 downloading the same file takes 5 minutes".. From there I take a sniffer trace of the "fast" transfer, and then a sniffer trace of the "slow" transfer. I normally span the client port to begin with. Then I compare the 2 traces to see the difference.

I look to see if the "slow" transfer is missing frames. Then this would tell me for sure that packets are getting dropped. Switches do 2 things: forward the packet or drop it. switches add very little latency to a data conversation.

Basically when it comes to slow performance taking sniffer traces is usually the way to go if you have checked out all the other issues such as speed, duplex, and input errors.

[l.mourits]

(In reply to Skarundi)

The main thing that bothers me is that I checked the number of deferred packets versus the number of transmitted packets on both switches during a timeframe of 30 minutes exactly and acn´t find only small differences on some interfaces, while performance drops significantly when I replace the 2828 with the 2950-24SX. For example, just the login script takes about one minute when the 2828 is in place, but takes about 5 minutes when the 2950 is in place. I expect to see an significant difference in deferrals then when the two switches are compared.

I agree that sniffing would be the next step to troubleshoot, but because differences where this big I thought that maybe this could be some known issue (for example, incompatibilty between the new 2950 and the existing 2900 provider switch when connected to eachother on 10Mb full duplex).

If switches would just do switch or drop a packet why on earth would a 2950-24SX drop more packets then a 2828 on a similar configuration with the same traffic patterns?

Do not forget that this is an existing situation which is allright (with the 2828 in) and slow performance is seen when the 2828 is replaced with the newer 2950.

But, let´s say that I will discover that the 2950 does drop more packets then the 2828, what´s to do next?

[smcquerry]

The first time I encountered the problem was January 2003. However I also had some issues in the past couple of months.

The last problem I had was resolved by letting the server and switch auto-negotiate. I cannot remember what release of code we were using at the time.

[uupsala]

Is it right that on the 2828 no additional vlan exists and this is the whole difference between both configurations? I suppose your uplinks port trunking setting is misconfigured and flaps between trunk and access mode. Maybe this could help you.