/var/adm/CSCOpx running out of space on LMS 2.5

Unanswered Question
Apr 28th, 2008

Hello,

we are using LMS 2.5, RME 4.0 and the filesystem /var/adm/CSCOpx is running out of space. It is growing when configurstions are successfully collected.

I think the problem might be that the jobs archive update, archive poller, system inventory polling and system inventory collection are failing.

The logfile /var/adm/CSCOpx/log/dmgtd.log shows errors like

Apr 27 21:32:04 [hostname] SQLAnywhere(rmengeng): [ID 702911 local0.notice] Connection terminated abnormally

Apr 27 21:32:04 [hostname] SQLAnywhere(cmfeng): [ID 702911 local0.notice] Connection terminated abnormally

Thank you in advance for your help

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
yjdabear Mon, 04/28/2008 - 06:18

Believe it or not, "Connection terminated abnormally" is usually benign and probably has nothing to do with the space issue. I bet it's your /var/adm/CSCOpx/files/rme/dcma/difffiles that's filling up the file system. You can verify that with "du -skr /var/adm/CSCOpx/files/rme/dcma/difffiles" and "/var/adm/CSCOpx/files/rme/dcma". If so, this is attributed to CSCse88215, that will hopefully be addressed in RME 4.1 of LMS 3.x. See these two old threads dealing with this subject.

http://forum.cisco.com/eforum/servlet/NetProf?page=netprof&forum=Network%20Infrastructure&topic=Network%20Management&topicID=.ee71a02&fromOutline=&CommCmd=MB%3Fcmd%3Ddisplay_location%26location%3D.2cbfc165

http://forum.cisco.com/eforum/servlet/NetProf?page=netprof&forum=Network%20Infrastructure&topic=Network%20Management&topicID=.ee71a02&fromOutline=&CommCmd=MB%3Fcmd%3Ddisplay_location%26location%3D.1ddf4921

netzeatkw Mon, 04/28/2008 - 07:41

you are right there are lots of difffiles.

This will solve the space problem but there is still a problem with the database left.

Thanks a lot

yjdabear Mon, 04/28/2008 - 07:56

What does your /opt/CSCOpx/bin/pdshow output say?

"Connection terminated abnormally" alone doesn't actually indicate you have a database problem. I've got tons of these in my dmgtd.log:

Apr 28 06:25:12 nms SQLAnywhere(cmfeng): [ID 702911 local1.notice] Connection terminated abnormally

Apr 28 06:25:12 nms SQLAnywhere(rmengeng): [ID 702911 local1.notice] Connection terminated abnormally

netzeatkw Tue, 04/29/2008 - 04:21

Please find the pdshow file attached.

What bothers me are the following messages in the /var/adm/CSCOpx/log/dmgtd.log after the start of dmgtd:

Apr 29 13:34:05 hostname dmgt[25600]: [ID 947059 local0.info] #3007:TYPE=INFO:Started application(RmeOrb) "/opt/CSCOpx/lib/vbroker/bin/osa

gent -p 42342" pid=25789.

Apr 29 13:34:07 hostname SQLAnywhere(cmfeng): [ID 702911 local0.notice] Note: Duplicate index "CSI1OgsGroupPropertiesTable" for table "CSO

gsGroupPropertiesTable" in database "cmfDb"

Apr 29 13:34:07 hostname SQLAnywhere(cmfeng): [ID 702911 local0.notice] Note: Duplicate index "mykey" for table "CSUserGroupAssociationTab

le" in database "cmfDb"

Apr 29 13:34:07 hostname SQLAnywhere(cmfeng): [ID 702911 local0.notice] Note: Duplicate index "mykey" for table "CSOgsTagTable" in databas

e "cmfDb"

Apr 29 13:34:08 hostname SQLAnywhere(cmfeng): [ID 702911 local0.notice] Note: Duplicate index "CampusI1OgsGroupPropertiesTable" for table

"CampusOgsGroupPropertiesTable" in database "cmfDb"

Apr 29 13:34:08 hostname SQLAnywhere(cmfeng): [ID 702911 local0.notice] Note: Duplicate index "mykey" for table "CampusUserGroupAssociatio

nTable" in database "cmfDb"

Apr 29 13:34:08 hostname SQLAnywhere(cmfeng): [ID 702911 local0.notice] Note: Duplicate index "mykey" for table "CampusOgsTagTable" in dat

abase "cmfDb"

How can I get additional information about failed jobs (archive poller, archive update)that are scheduled?

Attachment: 
yjdabear Tue, 04/29/2008 - 07:19

The pdshow output looks fine to me.

As far as I understand, the "duplicates" notices are harmless too, though I have no idea why they appear at the moments they do:

Apr 11 16:30:34 nms SQLAnywhere(cmfeng): [ID 702911 local1.notice] Note: Duplicate index "mykey" for table "CampusOgsTagTable" in database "cmfDb"

Apr 11 16:31:33 nms SQLAnywhere(cmfeng): [ID 702911 local1.notice] Note: Duplicate index "Index1" for table "OgsDcrDeviceTable" in database "cmfDb"

Apr 11 16:31:33 nms SQLAnywhere(cmfeng): [ID 702911 local1.notice] Note: Duplicate index "mykey" for table "OgsDcrDeviceTable" in database "cmfDb"

Apr 11 16:55:51 nms SQLAnywhere(cmfeng): [ID 702911 local1.notice] Note: Duplicate index "CampusI1OgsGroupPropertiesTable" for table "CampusOgsGroupPropertiesTable" in database "cmfDb"

Apr 11 16:55:51 nms SQLAnywhere(cmfeng): [ID 702911 local1.notice] Note: Duplicate index "mykey" for table "CampusUserGroupAssociationTable" in database "cmfDb"

Apr 11 16:55:51 nms SQLAnywhere(cmfeng): [ID 702911 local1.notice] Note: Duplicate index "mykey" for table "CampusOgsTagTable" in database "cmfDb"

Apr 11 16:56:37 nms SQLAnywhere(cmfeng): [ID 702911 local1.notice] Note: Duplicate index "Index1" for table "OgsDcrDeviceTable" in database "cmfDb"

Apr 11 16:56:37 nms SQLAnywhere(cmfeng): [ID 702911 local1.notice] Note: Duplicate index "mykey" for table "OgsDcrDeviceTable" in database "cmfDb"

Apr 27 22:32:52 nms SQLAnywhere(cmfeng): [ID 702911 local1.notice] Note: Duplicate index "mykey" for table "CampusOgsGroupCacheTable" in database "cmfDb"

You can get more info on each failed job by clicking on their respective Job ID in the DCR and/or RME homepage, as well as RME's Job Management tab. If you want to look at them via CLI, I find the convenient approach is to locate the job log using "find /var/adm/CSCOpx/ -name [JobID#].log -print".

Actions

This Discussion