Solved: Catalyst 6509 CPU utilization at 95%

benny · ‎01-14-2010

We have a pair of Catalyst 6509, one is having very high cpu constantly, and the two cpu-intensive processes are ios-base and tcp.proc

Attached file shows:

1) show processes cpu

2) show processes detail ios-base

3) show processes detail tcp.proc

4) show version

Servers are distributed among the two Cat6509, but this one is the main switch for uplink connections.

Please kindly advise how to troubleshoot and find out the cause of this high cpu utilization.

Thanks.

Jon Marshall · ‎01-14-2010

benny@panamby wrote:

Jon,

Thanks very much for your reply.

I did actually go through that document before and didn't find anything that seems to apply to our network.

I am planning to do a packet capture on the switch to look at the traffic, but I am afraid the monitor session will bring the CPU to 100%, as it is already 96% now, so I haven't done it yet. And so I would like to get some ideas here as to how to proceed.

Thanks!

Benny

Can you run a "sh tcp brief" and see if there are any hungs sessions ?

Jon

View solution in original post

Jon Marshall · ‎01-14-2010

benny@panamby wrote:

We have a pair of Catalyst 6509, one is having very high cpu constantly, and the two cpu-intensive processes are ios-base and tcp.proc

Attached file shows:

1) show processes cpu

2) show processes detail ios-base

3) show processes detail tcp.proc

4) show version

Servers are distributed among the two Cat6509, but this one is the main switch for uplink connections.

Please kindly advise how to troubleshoot and find out the cause of this high cpu utilization.

Thanks.

Benny

Start with this doc -

http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a00804916e0.shtml

Jon

benny · ‎01-14-2010

Jon,

Thanks very much for your reply.

I did actually go through that document before and didn't find anything that seems to apply to our network.

I am planning to do a packet capture on the switch to look at the traffic, but I am afraid the monitor session will bring the CPU to 100%, as it is already 96% now, so I haven't done it yet. And so I would like to get some ideas here as to how to proceed.

Thanks!

Jon Marshall · ‎01-14-2010

benny@panamby wrote:

Jon,

Thanks very much for your reply.

I did actually go through that document before and didn't find anything that seems to apply to our network.

I am planning to do a packet capture on the switch to look at the traffic, but I am afraid the monitor session will bring the CPU to 100%, as it is already 96% now, so I haven't done it yet. And so I would like to get some ideas here as to how to proceed.

Thanks!

Benny

Can you run a "sh tcp brief" and see if there are any hungs sessions ?

Jon

francisco.holguin · ‎01-14-2010

hi my friend, why don't you check if have any debug runing on your equipment, try with the command "undebug all", also you can check if you have a DoS attack (denial of services).

i hope it helps.

Regards.

benny · ‎01-14-2010

Francisco,

Thanks a lot for your help! I fixed the problem by removing hung telnet session to the switch.

Thanks!

benny · ‎01-14-2010

Jon,

Thanks a lot for your advice.

I did a show tcp brief, and saw a few 'established' connection to the switch itself, then i did a show line, and saw there are 6 telnet sessions sitting there for more than 9 weeks, which is strange... I configured the vty line to be expired in 60 mins. So not sure why they are still there. But once I clear all those old vty connection, the cpu is now down to 10% !!! What a big difference! I can't understand why a few telnet session to the switch will push the cpu all the way to 96%.

Thanks a lot for your help!

burleyman · ‎01-14-2010

We just had the exact same thing happen with our 6509's. I think this may be a new bug. Can you check something for me. Try and telnet back into the switch and then see if the CPU starts to go back up again with just one telnet session. That is what we are seeing and they have no fix for it. So that means we cannot telnet into the switch or it will send the CPU back up to 100%. If you see the same problem let me know.

Thanks,

Mike