LMS 3.2.1 ICServer push CPU to 100%

[mladenp]

Hello,

Recently I have problem with Inv. collection jobs because some time after job start CPU goes to 100% and stay like that. I can only do "pdterm ICServer" and than CPU goes to normal. I have done some troubleshoot and discovers that ICServer like to stuck on particular devices (switches), while on other devices (routers) whole job is complete without any problems. I have attached IC_Server.log with example of one unsuccessful job done on 3750 switch. Job is stuck 2 min after start. Any help would be appreciated.

Best regards,

Mladen

[AFROJ AHMAD]

Hi Maiden,

Run this command >> pdterm EnergyWise

wait for few minutes and check the CPU .

If it goes normal ,let me know.

Thanks

[mladenp]

Hello Afroj,

There isn't EnergyWise process:

C:\Users\cwsadmin>pdshow -brief

  Process               State                                     Pid

  *******               *****                                     ***

  Tomcat                Program started - No mgt msgs received    10524

  TomcatMonitor         Running normally                          12928

  Apache                Program started - No mgt msgs received    11080

  RMEDbEngine           Program started - No mgt msgs received    11172

  Proxy                 Program started - No mgt msgs received    13168

  ProcSysBus            Program started - No mgt msgs received    17264

  OpsxmlDbEngine        Program started - No mgt msgs received    10144

  OpsXMLRuntime         Program started - No mgt msgs received    4648

  CWAPublisher          Administrator has shut down this server   0

  NameServer            Program started - No mgt msgs received    9824

  NameServiceMonitor    Running normally                          4588

  EDS                   Running normally                          11256

  LicenseServer         Program started - No mgt msgs received    6784

  INVDbEngine           Program started - No mgt msgs received    13316

  FHPurgeTask           Transient terminated                      0

  FHDbEngine            Program started - No mgt msgs received    4172

  ESS                   Program started - No mgt msgs received    10896

  EssMonitor            Running normally                          12388

  EventFramework        Program started - No mgt msgs received    15756

  PMServer              Running normally                          14984

  SyslogCollector       Program started - No mgt msgs received    15716

  EPMDbEngine           Program started - No mgt msgs received    1368

  EPMServer             Running normally                          3616

  FHServer              Running normally                          10740

  diskWatcher           Running normally                          15052

  DFMMultiProcLogger    Program started - No mgt msgs received    17048

  DFMLogServer          Program started - No mgt msgs received    13124

  DFMCTMStartup         Administrator has shut down this server   0

  DfmBroker             Program started - No mgt msgs received    4152

  DfmServer             Running normally                          15852

  DfmServer1            Running normally                          4900

  DCRDevicePoll         Transient terminated                      0

  CSSCPServer           Program started - No mgt msgs received    5148

  CSRegistryServer      Running normally                          7616

  CSDiscovery           Never started                             0

  CmfDbEngine           Program started - No mgt msgs received    9904

  CmfDbMonitor          Running normally                          11720

  DCRServer             Running normally                          8484

  CMFOGSServer          Program started - No mgt msgs received    14232

  CampusOGSServer       Program started - No mgt msgs received    8820

  EssentialsDM          Running normally                          7872

  ConfigMgmtServer      Program started - No mgt msgs received    14548

  ICServer              Running normally                          6396

  RMEOGSServer          Program started - No mgt msgs received    16004

  PMCOGSServer          Program started - No mgt msgs received    12992

  TISServer             Program started - No mgt msgs received    8616

  DFMOGSServer          Program started - No mgt msgs received    9024

  InventoryCollector    Running normally                          16668

  Interactor            Program started - No mgt msgs received    9368

  InventoryCollector1   Program started - No mgt msgs received    7516

  Interactor1           Program started - No mgt msgs received    9708

  NOSServer             Running normally                          11108

  PTMServer             Running normally                          13172

  EDS-GCF               Running normally                          11856

  jrm                   Running normally                          12620

  2820                  Never started                             0

  2821                  Never started                             0

  3295                  Never started                             0

  CTMJrmServer          Running normally                          3848

  ChangeAudit           Program started - No mgt msgs received    11224

  SyslogAnalyzer        Program started - No mgt msgs received    7456

  DataPurge             Administrator has shut down this server   0

  ANIDbEngine           Program started - No mgt msgs received    6340

  ANIServer             Running with busy flag set                14464

  UTMajorAcquisition    Transient terminated                      0

  MACUHIC               Running normally                          3968

  UTLITE                Running normally                          8088

  UTManager             Running normally                          4432

  VNMServer             Program started - No mgt msgs received    16680

  WlseUHIC              Running normally                          4712

  AdapterServer1        Program started - No mgt msgs received    6652

  AdapterServer         Program started - No mgt msgs received    3344

Thanks,

Mladen

[AFROJ AHMAD]

Hi Mladen,

what is the Size of RME database.

Can you send the screenshot of /NMSROOt/CSCOPx/database/rmeng

Also have you seen any pattern or any specific device like 3750 which is causing issue while inventory polling.

as there are Bugs related to this.

Thanks

[mladenp]

Hello Afroj,

If I made right conclusion ICServer have problems with switches where some changes in inventory are detected, inventory polling jobs never stuck, only inventory collection...

2960 and 3750 cause problems.

Thanks,

Mladen

[AFROJ AHMAD]

Hi Mladen,

Enable the debugging for the Inventory Collection process as follows:
    Please look for the IC_Server.log (NMSROOT\log\), clean the contents of
the files, 
    and save them and after that enable debugging:
    a. RME -> Admin -> System Preferences -> Loglevel Settings, specify the
following:
    -- Application = ICServer
    -- Inventory Collection Service = DEBUG
    -- Inventory User Interface = DEBUG
    -- Inventory Collection Jobs = DEBUG
    -- Click Apply
   b. Redo the Inventory Collection for 1 of the (3750 and 2960) device : RME ->
Devices -> Inventory -> Inventory Jobs.
   c.  Once the job is finished, send me the following:
   -- $NMSROOT\log\IC_Server.log

Thanks-

[mladenp]

Hello Afroj,

Job is started at 1:05pm, cpu goes to 100% around 1:07pm/1:08pm. Pdterm ICServer is executed at 1:15pm.

[AFROJ AHMAD]

Hi ,

Configure the below SNMP VIEW on one of the 3750 and then run the inventroy and check the CPU.

snmp-server view cutdown iso included
snmp-server view cutdown ciscoFlashMIB excluded
snmp-server community  view cutdown RO

Thanks-
Afroz

[mladenp]

Hello Afroj,

Done, but without improvement.

Regards,

Mladen