CiscoWorks ANI server cannot connect, Joe Clark please help

Answered Question
Dec 28th, 2007

Joe, if you see this, please help

I noticed yesterday I was getting some Java errors when opening different modules of Cisco Works, Campus Manager, RME.

I rebooted the server and everything seemd to be working fine. I tried to set up Notification Services after the reboot and it seemed to go without incident.

This morning Common Services shows

"DCA Server is down or inaccessable"

Campus Manager shows:

"Cannot connect to ANI Server since it is down"

Here is a clip from ani.log:

2007/12/28 08:00:08 Thread-76 ani ERROR DBConnection$ConnectionCreator: Failed to establish DB connectivity because: java.sql.SQLException: JZ006: Caught IOException: java.net.ConnectException: Connection refused: connect

2007/12/28 08:00:08 Discovery ani ERROR DiscoveryTimeBaseStat: Failed to enumerate Wbu table names because null

2007/12/28 08:00:08 Discovery ani DiscoveryTimeBaseStat: {NumDevices=31}{NumDiscovered=0}{NumDeleted=0}{DPH=14695}{OPH=0}

2007/12/28 08:00:08 Discovery ani DiscoveryTimeBaseImpl: completed: Discovery in 8578 ms

1198846847568 December 28, 2007 8:00:47 AM ES_SVRCWLMS__1198788044958(127) SVRCWLMS Error EDSSY0012: Cannot connect to EDS.

1198846910568 December 28, 2007 8:01:50 AM ES_SVRCWLMS__1198788044958(127) SVRCWLMS Error EDSSY0012: Cannot connect to EDS.

1198846973568 December 28, 2007 8:02:53 AM ES_SVRCWLMS__1198788044958(127) SVRCWLMS Error EDSSY0012: Cannot connect to EDS.

Reboot has not helped this morning.

I just found out we did some WIndows patches a couple of days ago.

I have this problem too.
0 votes
Correct Answer by Joe Clarke about 8 years 11 months ago

Shutdown dmgtd. Verify all the dbsrv9.exe processes are down. Run the following commands:

del /f D:\CSCOpx\databases\cmf\cmf.log

del /f D:\CSCOpx\databases\ani\ani.log

del /f D:\CSCOpx\databases\dfmEpm\dfmEpm.log

del /f D:\CSCOpx\databases\dfmInv\dfmInv.log

D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\cmf\cmf.db

D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\ani\ani.db

D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\dfmEpm\dfmEpm.db

D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\dfmInv\dfmInv.db

Then restart dmgtd. All the processes should come up.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (9 ratings)
Loading.
wilson_1234_2 Fri, 12/28/2007 - 07:28

There were some security patches installed, I will have to check that.

I have noticed it does look like there could be an issue with permissions in the logs.

I just checked, that patch is on there, but not one of the ones recently installed, that one (921883)was installed on 11/23/2006

David Stanford Fri, 12/28/2007 - 07:37

You'll want to see what the highest level process is that is failing. If you run pdshow which processes are down that should be up? Is tomcat running?

ANIServer depends on EDS and ANIDbEngine, are they up?

wilson_1234_2 Fri, 12/28/2007 - 07:55

Tomcat- Program Started - No mgt msgs received

EDS-GCF - never started

jrm - never started

aniDBEngine - never started

aniServer - never started

David Stanford Fri, 12/28/2007 - 08:04

Were the any changes made to the casuser account? Or services changed from automatic to manual?

CAn you try a resetCasuser and restart the daemons?

wilson_1234_2 Fri, 12/28/2007 - 08:09

no changes made to causer account.

How would I reset causer?

I do see the CiscoWorks services are manual, but starting ani does not work.

Attempts to restart services not started fails, I see no authentication errors in TACACS+, where the causer account is authenticated.

Joe Clarke Fri, 12/28/2007 - 10:09

Please provide the entire pdshow output and the output of pdreg -l ANIDbEngine.

wilson_1234_2 Fri, 12/28/2007 - 10:48

Joe,

Here are the files.

I have also reset the casuser account with no change.

I see several proccesses of the dbsrv9.exe running, is this normal?

Attachment: 
Joe Clarke Fri, 12/28/2007 - 11:08

Yes, this is normal. What files do you have under the following directories:

D:\CSCOpx\databases\cmf

D:\CSCOpx\databases\ani

D:\CSCOpx\databases\dfmEpm

D:\CSCOpx\databases\dfmInv

wilson_1234_2 Fri, 12/28/2007 - 11:14

All folders look the same with a folder and files as shown

D:\CSCOpx\databases\cmf:

"orig" folder

cmf.db

cmf.log

D:\CSCOpx\databases\ani

"orig" folder

ani.db

ani.log

D:\CSCOpx\databases\dfmEpm

"orig" folder

dfmEpm.db

dfmEpm.log

D:\CSCOpx\databases\dfmInv

"orig" folder

dfmInv.db

dfmInv.log

Correct Answer
Joe Clarke Fri, 12/28/2007 - 11:21

Shutdown dmgtd. Verify all the dbsrv9.exe processes are down. Run the following commands:

del /f D:\CSCOpx\databases\cmf\cmf.log

del /f D:\CSCOpx\databases\ani\ani.log

del /f D:\CSCOpx\databases\dfmEpm\dfmEpm.log

del /f D:\CSCOpx\databases\dfmInv\dfmInv.log

D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\cmf\cmf.db

D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\ani\ani.db

D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\dfmEpm\dfmEpm.db

D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\dfmInv\dfmInv.db

Then restart dmgtd. All the processes should come up.

wilson_1234_2 Fri, 12/28/2007 - 11:42

You are the best duder,

That fixed it, but I have some questions:

1. What did I do (looks like deleted log files and databases and created new ones), with the above commands?

2. What could have caused the inittial problem?

3. Is there any sequence that needs to be performed to reboot the Cisco Works server?

4. What can I do for you, box of cigars, maybe enroll you in the Jelly of the Month club or something?

Thanks Joe, your help has been invaluable once again

Joe Clarke Fri, 12/28/2007 - 11:52

Your transaction logs were damaged, and you deleted them, then forced the database to a quiet state by ignoring any outstanding transactions. This typically occurs when the server crashes or is shutdown improperly. To properly shut things down, first perform a net stop crmgmtd, then reboot the server once dmgtd has completed its shutdown.

wilson_1234_2 Fri, 12/28/2007 - 12:18

Joe,

Is this a common thing?

Are there any more databases that you can use these commands on?

Have I lost any historical information by doing this?

Joe Clarke Fri, 12/28/2007 - 12:40

Not very common considering the number of deployments of CiscoWorks and the number of times CiscoWorks servers are rebooted.

These commands apply to all of the CiscoWorks Sybase databases. Basically, any dsn under the NMSROOT/databases directory.

You would have list some information that was not checkpointed before the server was rebooted. However, for users that have been told to do these recovery steps, none have reported any obvious data loss to my knowledge.

wilson_1234_2 Fri, 12/28/2007 - 12:46

Joe,

Is there any way this could have been related to me setting up Notification Services?

Something I noticed yesterday is that when setting up "Event Sets", nothing shows up.

Per the document (and I saw this yesterday after a reboot of the server (most likely what caused the problem), I should see a list of items to trigger notification and I can configure nine different sets.

But, I see "No Records" on the Event Sets page.

I see the A,B,C etc, but nothing under them.

After a reboot yesterday, then I saw the items and checked off for the different event sets.

My goal is to have an e-mail alert when switch or router interfaces go down , or high utilization of fragmentation.

Joe Clarke Fri, 12/28/2007 - 12:51

No, this was caused by the database engines being improperly shutdown which most likely occurs when the server is shutdown improperly. Nothing done through the CiscoWorks GUI would ever cause transaction log corruption.

If you tried to do anything with events in DFM, you would have gotten errors or no results since the EPM database was down.

At this point, assuming the EPM database and EPMServer are up, you should see events in your event set configuration, and you should be able to assign events to those sets.

wilson_1234_2 Fri, 12/28/2007 - 13:31

Joe,

I have noticed in Alerts and Activities (I saw this yesterday also) numerous alerts some active for more than 200 days.

I have tried to clear them and it seems to hang the application. Then going back in, I see no alrts.

Can I clear all of these active alerts at once?

Joe Clarke Fri, 12/28/2007 - 13:43

Not yet. The ability to clear alerts en masse is coming in DFM 3.1 part of LMS 3.1 due out late this spring.

If you are going to pursue DFM troubleshooting, please start a new thread.

wilson_1234_2 Fri, 12/28/2007 - 10:56

Joe,

Another thing is the dcmaservice and the ANIServerlog logs stopped updating at 3:41 yesterday afternoon.

Nothing since then.

Actions

This Discussion