Slow response issue

willemvwyk · ‎09-30-2007

Hi

I would like to know what debugging tools exist in the IOS of cisco catalysts as well as for a 6509?

I am sitting with a problem where there are 3 floors in the same building (admin building). Each floor has 3 switches. So there's a total of 9 switches. These switches are connected with fibre to the head office, which is about 5 minutes walking distance from the admin building. These fibre cables then connect to a 6509 gig blade. This 6509 then connects to a cisco router as the gateway. The head office use the exact same 6509 and gateway router, but head office is not complaining about slow response.

Basically what happens is intermittently the users on all floors in admin building will complain there's a very slow response to all their applications. This occurs like once a week or so but then it will be throughout the day that they will have slow connectivity and then fast connectivity etc.

I would like to know if there is a debugging tool in the IOS for the switches as well as for the 6509 in order to try and nail the problem?

Thanks

wvw

JORGE RODRIGUEZ · ‎09-30-2007

Willem, from the description you indicate that this happens in only (admin building) and not others, is this correct? I would start looking into uplinks ports from admin building to Core6509, do the switches have dual fiber link to single 6509 in head office or there are other 6509 in the head office like dual core where buildings interconnect into two 6509s.

There are few troubleshooting tools, most common is to first gather information on how admin building differs from other buildings,

what type of siwthces in admin building and amount of users.In your scenarion other building do not have same issue. Particularly I would focus on admin building uplink ports as well as users connections ports, look into uplink port trafic stats " show int g0/1 " and see if there are any indications or errors,. Check switches cpu utilization " show process cpu " or " show proc cpu " for CATOSs and note cpu util during normal days to get base line, do same when slowness occor. Look in switches logs including the head office logs, usually logs can tell whether there are flaping interfaces that may indicate bridging loops. For the most part slowness may be caused by bridging loops, but could also be switches in admin building may be undersubscribed.

Here is a link to refer to with regards to STP, and how to gather information on your layer 2 topology for troubleshooting.

http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a0080136673.shtml#intro

Rgds

Jorge

Jorge Rodriguez

willemvwyk · ‎10-03-2007

Hi Jorge

Thanks for the feedback. My apologies for only replying now. We are having some serious WAN issues this side. I have started reading the docs from the link you have posted. Just to answer some of your questions:

From checking the interfaces on the switches as well as the 6509 I do not see any errors. The other thing is when the admin bldg complains about slow response I get to the switches and the 6509 instantly which indicates there is no high cpu utilisation. But I checked it anyway, and the cpu for the switches were sitting at an average of 21%. The 6509 was less at about 5% ( if I remember correctly).

What do you mean with "For the most part slowness may be caused by bridging loops, but could also be switches in admin building may be undersubscribed."?

Thank you

willemvw

pciaccio · ‎09-30-2007

Within IOS version of switches and routers you can enable CEF (on by default) and use Netflow. Once CEF is enabled you can enable stats on the interfaces and view traffic patterns and loads on the interfaces. You will require a Netflow tool (some are free most cost some money). However that is the only IOS based traffic analyzer there is. Of course you can invest in a generic NMS system that gives you all the bells and whistles but that will cost a lot of money...Good Luck

willemvwyk · ‎10-02-2007

Hi pciaccio

Thank you for the help. CEF is not available on these switches in the admin bldg. The switches are all C2950G and when I tried to enable cef i didn't have the option (switch1#ip cef enabled). Any other suggestions, please?

thanks

wvw

paul.matthews · ‎09-30-2007

There are lots of debugging tools available, but first we have to try to find out which are the useful tools.

The very bad news. An intermittent performance problem is probably the most difficult type of problem to fix.

The first thing to do is get users to start recording what problems they have when and start to look for a pattern, either on times or what they do. Does the problem happen every three days at 14:25 for example. Is it every tine the sales director comes in with his XP notebook that he has been playing with and told to bridge?

look at the interfaces themselves covering the links to the core switches. Are any errors reported? When do errors increment?

What is the topology? Is your root bridge forced? Is Spanning tree stable? Have you followed best practice on the config - things like user ports to portfast, with BPDU guard? Are VLANs manually pruned where not needed? If going between offices, I would (if possible) force the topology - have you got redundant links? If not the access switches can be in a flat VLAN and the link back can be access rather than trunk. and spanning tree can be turned off.

willemvwyk · ‎10-02-2007

Hi Paul

Thanks for your assistance. As if this intermittent problem is not bad enough I also don't get to work on it when it occurs. It is handed to either myself or my colleague. THat makes life even more "interesting" :)

I have checked the interfaces and no errors reported. No incrementing of errors either.

Every switch has a vlan 200 configured and they have no errors either.

What do you mean with "Is your root bridge forced"?

What do you mean with "If not the access switches can be in a flat VLAN and the link back can be access rather than trunk. and spanning tree can be turned off."

Thanks

willemvw

paul.matthews · ‎10-02-2007

By root bridge forced, I mean have you configured your network such that you know where the root bridge is? Good practice is that you have decided on the location of the root, and have configured the network to enforce that location.

Whn I talk about the flat VLAN I am talking about the access switches. If you only need one VLAN on the switchthen you can simplify the spanning tree by having the uplink as an access port rather than trunk. It is not something I would normally recommend, but if you are having issues, it may be worth trying.