IDS 4215 CPU 100 % high utilization issue

Unanswered Question
Mar 13th, 2010
User Badges:

Hi there, I upgraded my IDS 4215 to the 6.0.1  version, I got 512 Mb of ram, but the system is very slow, so I created a  service user, then I ran top, and I saw that near 100% of CPU is used  (see code below) by the sensor app, and then it crashes.
The CPU  values below are after a fresh recovery of the IPS, and there is no  traffic sent to the sensor, only management interface is connected.

I  checked some logs:
ids4215# show clock
*11:47:09 UTC Sat Mar 13  2010

top - 11:24:47 up 0 min,  1 user,  load average: 0.71, 0.20, 0.06
Tasks:  48 total,   5 running,  43 sleeping,   0 stopped,   0 zombie
Cpu(s):   0.3% user,  49.3% system,  50.0% nice,   0.3% idle
Mem:    490260k total,   485500k used,     4760k free,     1608k buffers
Swap:        0k total,        0k used,        0k free,    91600k cached

  PID USER      PR  NI  VIRT  RES  SHR S
%CPU %MEM    TIME+  COMMAND           
 
388 cids      18   5 10584 398m 394m R 81.0 83.2   0:17.22 sensorApp         
 
368 cids      17  15  9348 9284 69362.6  1.9   0:00.13 mainApp           
 
398 admin     14   0   988  988  8001.7  0.2   0:00.16 top               
   
4 root       9   0     0    0    01.3  0.0   0:00.10 kswapd            
  
47 root       9   0     0    0    00.3  0.0   0:00.08 kjournald         
 
305 cids      13   5  9348 9284 69360.3  1.9   0:00.71 mainApp           
 
306 cids       9   0  9348 9284 69360.3  1.9   0:00.03 mainApp           
 
339 cids      13   5  9348 9284 69360.3  1.9   0:00.11 mainApp           
 
366 cids      17  15  9348 9284 69360.3  1.9   0:00.02 mainApp           
   
1 root       8   0   576  576  4920.0  0.1   0:09.91 init              
   
2 root       9   0     0    0    00.0  0.0   0:00.00 keventd           
   
3 root      18  19     0    0    00.0  0.0   0:00.00 ksoftirqd_CPU0    
   
5 root       9   0     0    0    00.0  0.0   0:00.00 bdflush           
   
6 root       9   0     0    0    00.0  0.0   0:00.00 kupdated          
  
72 root       9   0     0    0    00.0  0.0   0:00.06 kjournald         
 
104 root       9   0   584  584  5040.0  0.1   0:00.01 syslogd           
 
107 root       9   0   572  572  4960.0  0.1   0:00.00 klogd

//or :

sensor
# sh stat host | begin CPU
CPU
Statistics
  
Usage over last 5 seconds = 100
  
Usage over last minute = 98
  
Usage over last 5 minutes = 98

//on the sh ver:
Using 405282816 out of 502026240 bytes of available memory (80% usage)
system
is using 17.8M out of 29.0M bytes of available disk space (61% usage)
application
-data is using 35.0M out of 166.8M bytes of available disk space (22% usage)
boot
is using 37.7M out of 68.6M bytes of available disk space (58% usage)



I reimaged with a 6.04 image, and I got the same error.

everytime, I use a .img file, booting on rommon, so I assume it format the flash.

I checked the log and it shows me this:


-bash-2.05b$ tail -n 50 /var/log/messages
Mar 13 12:21:32 sensor user.notice /etc/init.d/S80cids: Clean up:
Mar 13 12:21:34 sensor user.notice /etc/init.d/S80cids: mainApp -d startup
Mar 13 12:21:35 sensor user.notice root: mainApp (cids) started
Mar 13 12:21:35 sensor user.notice /etc/init.d/S80cids: Starting CIDS:
Mar 13 12:21:36 sensor user.notice kernel: e100: fe0_0 NIC Link is Up 100 Mbps Full duplex
Mar 13 12:21:38 sensor daemon.info init: ^MStarting pid 385, console /dev/tty1: '/sbin/getty'
Mar 13 12:21:38 sensor daemon.info init: ^MStarting pid 386, console /dev/tty2: '/sbin/getty'
Mar 13 12:21:38 sensor daemon.info init: ^MStarting pid 387, console /dev/ttyS0: '/sbin/getty'
Mar 13 12:21:38 sensor daemon.info init: ^MStarting pid 388, console /dev/ttyS1: '/sbin/getty'
Mar 13 12:22:58 sensor daemon.info init: ^MStarting pid 416, console /dev/null: '/etc/init.d/rc.down'
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 389 (sensorApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 389 (sensorApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 411 (sensorApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 412 (sensorApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 413 (sensorApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 414 (sensorApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 346 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 328 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 331 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 332 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 333 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 346 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 347 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 348 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 363 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 364 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 365 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 366 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 367 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 368 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 369 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 370 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 371 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 372 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 373 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 374 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 375 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 376 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 377 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 390 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 392 (mainApp).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 121 (inetd).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 121 (inetd).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 1 (init).
Mar 13 12:22:58 sensor user.err kernel: Out of Memory: Killed process 1 (init).
Mar 13 12:22:59 sensor auth.info sshd[125]: Received signal 15; terminating.
Mar 13 12:22:59 sensor user.notice /etc/init.d/S60ssh: sshd -TERM
Mar 13 12:23:00 sensor auth.info login(pam_unix)[387]: session opened for user admin by (uid=0)
Mar 13 12:23:00 sensor auth.info  -- admin[387]: DIALUP AT ttyS0 BY admin
Mar 13 12:23:00 sensor auth.info  -- admin[387]: LOGIN ON ttyS0 BY admin


I have just upgraded to 512M to upgrade to 6.0x version, So i guess the sensorapp is doing some nasty things that hangs up all the CPU and then it crashes... I supect a signature error, wouldn't it be erased after a reimaging or a recovery ?


Thanks for your help.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
bastien.migette Sat, 03/13/2010 - 10:37
User Badges:

can someone try these commands and see that I really got a problem : ?


sensor(config)# username admin passw password priv service
sensor(config)# exit
sensor# exi
login: admin
Password:
bash-2.05b$su
Password: !same pass as admin
bash-2.05b# cd /usr/cids/idsRoot/etc/config/analysisEngine/
bash-2.05b# ls
current.xml       default.xml       typedefs.xml
default.sig.xml   typedefs.sig.xml


bash-2.05b# cat current.xml
£ [email protected]@á½S4Ý3õ³’_
                               ˜“Њ)·Äî[½g1†‘&%×æyð;S
                                                      ®Œi_Sç¦Ôê\—ÀžE¶ ¥T 1Æã
HŽÜ“Ñ}¦”eÙ”—ðÛ·鳇spbash-2.05b#
bash-2.05b#

Actions

This Discussion