12-28-2007 06:02 AM
Joe, if you see this, please help
I noticed yesterday I was getting some Java errors when opening different modules of Cisco Works, Campus Manager, RME.
I rebooted the server and everything seemd to be working fine. I tried to set up Notification Services after the reboot and it seemed to go without incident.
This morning Common Services shows
"DCA Server is down or inaccessable"
Campus Manager shows:
"Cannot connect to ANI Server since it is down"
Here is a clip from ani.log:
2007/12/28 08:00:08 Thread-76 ani ERROR DBConnection$ConnectionCreator: Failed to establish DB connectivity because: java.sql.SQLException: JZ006: Caught IOException: java.net.ConnectException: Connection refused: connect
2007/12/28 08:00:08 Discovery ani ERROR DiscoveryTimeBaseStat: Failed to enumerate Wbu table names because null
2007/12/28 08:00:08 Discovery ani DiscoveryTimeBaseStat: {NumDevices=31}{NumDiscovered=0}{NumDeleted=0}{DPH=14695}{OPH=0}
2007/12/28 08:00:08 Discovery ani DiscoveryTimeBaseImpl: completed: Discovery in 8578 ms
1198846847568 December 28, 2007 8:00:47 AM ES_SVRCWLMS__1198788044958(127) SVRCWLMS Error EDSSY0012: Cannot connect to EDS.
1198846910568 December 28, 2007 8:01:50 AM ES_SVRCWLMS__1198788044958(127) SVRCWLMS Error EDSSY0012: Cannot connect to EDS.
1198846973568 December 28, 2007 8:02:53 AM ES_SVRCWLMS__1198788044958(127) SVRCWLMS Error EDSSY0012: Cannot connect to EDS.
Reboot has not helped this morning.
I just found out we did some WIndows patches a couple of days ago.
Solved! Go to Solution.
12-28-2007 11:21 AM
Shutdown dmgtd. Verify all the dbsrv9.exe processes are down. Run the following commands:
del /f D:\CSCOpx\databases\cmf\cmf.log
del /f D:\CSCOpx\databases\ani\ani.log
del /f D:\CSCOpx\databases\dfmEpm\dfmEpm.log
del /f D:\CSCOpx\databases\dfmInv\dfmInv.log
D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\cmf\cmf.db
D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\ani\ani.db
D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\dfmEpm\dfmEpm.db
D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\dfmInv\dfmInv.db
Then restart dmgtd. All the processes should come up.
12-28-2007 07:23 AM
Did you install 921883 (MS06-040) Microsoft security patch by chance?
12-28-2007 07:28 AM
There were some security patches installed, I will have to check that.
I have noticed it does look like there could be an issue with permissions in the logs.
I just checked, that patch is on there, but not one of the ones recently installed, that one (921883)was installed on 11/23/2006
12-28-2007 07:37 AM
You'll want to see what the highest level process is that is failing. If you run pdshow which processes are down that should be up? Is tomcat running?
ANIServer depends on EDS and ANIDbEngine, are they up?
12-28-2007 07:55 AM
Tomcat- Program Started - No mgt msgs received
EDS-GCF - never started
jrm - never started
aniDBEngine - never started
aniServer - never started
12-28-2007 08:04 AM
Were the any changes made to the casuser account? Or services changed from automatic to manual?
CAn you try a resetCasuser and restart the daemons?
12-28-2007 08:09 AM
no changes made to causer account.
How would I reset causer?
I do see the CiscoWorks services are manual, but starting ani does not work.
Attempts to restart services not started fails, I see no authentication errors in TACACS+, where the causer account is authenticated.
12-28-2007 10:09 AM
Please provide the entire pdshow output and the output of pdreg -l ANIDbEngine.
12-28-2007 10:48 AM
12-28-2007 11:08 AM
Yes, this is normal. What files do you have under the following directories:
D:\CSCOpx\databases\cmf
D:\CSCOpx\databases\ani
D:\CSCOpx\databases\dfmEpm
D:\CSCOpx\databases\dfmInv
12-28-2007 11:14 AM
All folders look the same with a folder and files as shown
D:\CSCOpx\databases\cmf:
"orig" folder
cmf.db
cmf.log
D:\CSCOpx\databases\ani
"orig" folder
ani.db
ani.log
D:\CSCOpx\databases\dfmEpm
"orig" folder
dfmEpm.db
dfmEpm.log
D:\CSCOpx\databases\dfmInv
"orig" folder
dfmInv.db
dfmInv.log
12-28-2007 11:21 AM
Shutdown dmgtd. Verify all the dbsrv9.exe processes are down. Run the following commands:
del /f D:\CSCOpx\databases\cmf\cmf.log
del /f D:\CSCOpx\databases\ani\ani.log
del /f D:\CSCOpx\databases\dfmEpm\dfmEpm.log
del /f D:\CSCOpx\databases\dfmInv\dfmInv.log
D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\cmf\cmf.db
D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\ani\ani.db
D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\dfmEpm\dfmEpm.db
D:\CSCOpx\objects\db\win32\dbsrv9.exe -f D:\CSCOpx\databases\dfmInv\dfmInv.db
Then restart dmgtd. All the processes should come up.
12-28-2007 11:42 AM
You are the best duder,
That fixed it, but I have some questions:
1. What did I do (looks like deleted log files and databases and created new ones), with the above commands?
2. What could have caused the inittial problem?
3. Is there any sequence that needs to be performed to reboot the Cisco Works server?
4. What can I do for you, box of cigars, maybe enroll you in the Jelly of the Month club or something?
Thanks Joe, your help has been invaluable once again
12-28-2007 11:52 AM
Your transaction logs were damaged, and you deleted them, then forced the database to a quiet state by ignoring any outstanding transactions. This typically occurs when the server crashes or is shutdown improperly. To properly shut things down, first perform a net stop crmgmtd, then reboot the server once dmgtd has completed its shutdown.
12-28-2007 12:18 PM
Joe,
Is this a common thing?
Are there any more databases that you can use these commands on?
Have I lost any historical information by doing this?
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: