High CPU on 11506

cbuzzard · ‎08-02-2003

We have a pair of 11506(s) running VIP and interface redundancy. We recently experienced periods of high CPU (95%+) during non-peak traffic times. Our SW ver is 7.1 bld 6; searching the bug toolkit doesn't show any known issues. The CPU doesn't stay up for too long, it does go back down on it's own without intervention, however we can't isolate what's causing the CPU to spike in the first place.

Our CSS(s) have just the 1 head unit and the 2 GE ports. One port is the ISC between the 2 for state and the other is the trunk link to the network.

We have about 80 services configured. Most of those are port 80 and 443 services. The 80 ones are monitored on a script keepalive pulling up and ASP page. The 443 ones are the KAL-AP-SSL script. There are a handful of other services that are just monitored via TCP keepalive type.

The last time it happened we went into debug mode and dropped a show tasks to send off to TAC, waiting on an answer for those. Anyone know of any better commands to drop to find out which process may be tapping the CPU?

Anyone seen this before? Any help would be appreciated.

Thanks

Chad

jfoerster · ‎08-03-2003

Hi Chad,

two questions at the beginning:

1) Are those CSSes monitored via HP Openview or something similiar?

2) Is this peak occuring regulariy? If yes what's the time difference between two peeks?

I had this kind of behaviour on routers and switches a lot of times and most of the times a SNMP-polling was the reason for this.

Another reason for this could be, even if I do not know where you placed the CSSes, that some kind of DoS is hitting you.

Are the CSSes behind or infront of a Firewall?

Hope this helps.

Kind Regards,

Joerg

cbuzzard · ‎08-04-2003

Hi,

We monitor via SNMP so the fact you mention that part below is interesting. We check about 5 different OIDs within a 10 minute period, 1 or 2 as often as 3 minutes.

They are located behind a firewall and completely 1918 addressed. They do provide load balancing for externally available web sites, but the traffic on the site at the time this last happened was very low.

We haven't been able to put together any good references on the peak times. The first time was last tuesday in the afternoon and the most recent was Friday night in the middle of the night. We're tracking it to see if there isn't something else that correlates to when this happens, but right now there's nothing.

Thanks for the input.

chad

Gilles Dufour · ‎08-07-2003

Chad,

Another command is 'cpu monitor'

First do a 'symbol-table load SPRITZ' then a 'cpu monitor '

The threshold is a minimum value of percentage to display a process. ie: if you put a threshold of 10, the command will only display the processes that consume more than 10%.

Capture this info many times (you just need to type 'd' to collect many results) and send us the summary.

Once we know the process that consume the memory we can tell you where to look.

Gilles.

cbuzzard · ‎08-07-2003

Thanks for the info, next time it happens we will definitely capture that. We were also given the following as well if it will also help:

symbol-table load SPRITZ

shell 1 1 spy

shell 1 1 spyReport

shell 1 1 spyStop

symbol-table unload SPRITZ

Gilles Dufour · ‎08-08-2003

the document I gave you is safer and will directly display what we need.

Gilles.