Re: LMS 3.1 installation/migration issues - Page 2

yjdabear · ‎09-24-2008

1. Can LMS 3.1 function without tftp? I'm not so much concerned about LMS installing /opt/CSCOpx/bin/in.tftpd when it finds the Sol 10 tftpd binary is intentionally removed, but whether the product has any dependency on /tftpboot/ later on during day-to-day operations, even with TFTP not used in RME and assorted places.

2. On one of the boxes, wrapper.pl keeps ending prematurely with the following, even though RBackup.sh is right there:

...

Backing up CiscoView data now

Backing up CiscoView done

IPM backup starts now

IPM BackUp Loaction /product/ehealth/backups/lms26backup/0/ipm

Exiting the backup process.

Reason: RBackup.sh is not available

The exact same steps backed up the data fine on another box.

3. On the above box where wrapper.pl functions fine, "dmgtd stop" takes hours but still fails to shut down all LMS processes:

Daemon Management stopping. This may take a few minutes.

WARNING: Daemon Manager terminated with SIGKILL.

INFO : Stopping DBEngine processes registered to Daemon Manager

WARNING: Please check if all processes have been terminated using

WARNING: the command - "ps -ef|grep CSCOpx" and

WARNING: terminate them if any processes are running.

casuser 21339 20353 0 15:58:40 ? 0:00 /opt/CSCOpx/objects/wfengine/program/GGScript

casuser 21356 20353 0 15:58:45 ? 0:00 /opt/CSCOpx/objects/wfengine/program/DataStore

casuser 20353 1 0 15:35:17 ? 0:18 /opt/CSCOpx/objects/wfengine/program/PolicyServer

casuser 21342 20353 0 15:58:40 ? 0:01 /opt/CSCOpx/objects/wfengine/program/GGLogger

casuser 21355 20353 0 15:58:45 ? 0:05 /opt/CSCOpx/objects/wfengine/program/OpsBREngine

casuser 21343 20353 0 15:58:41 ? 0:00 /opt/CSCOpx/objects/wfengine/program/GGProxy

casuser 21344 20353 0 15:58:41 ? 0:00 /opt/CSCOpx/objects/wfengine/program/GGHelper

casuser 21357 20353 0 15:58:45 ? 0:01 /opt/CSCOpx/objects/wfengine/program/PolicyScheduler

Joe Clarke · ‎09-25-2008

Looks like I was mistaken. Our server switch was actually in transparent mode. When I put it in server mode, the VTP/VLAN data was added to the vlan.dat file. As it turns out, only extended VLAN information is stored in the running config. When the device in is VTP server mode, the rest of the config is stored in a vlan.dat file even though "vlan database" should not be used to configure VLANs and VTP information.

So, yes, TFTP will be required for your VTP server devices.

Joe Clarke · ‎09-25-2008

And for my own edification, I found the doc explaining just this:

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/vlans.html#wp1020848

yjdabear · ‎09-29-2008

I ran truss against the wrapper.pl process on one box, but it seems there's an error that stopped truss from capturing anything after wrapper.pl started backing up cmfDb. As a result, there's nothing on why wrapper.l couldn't locate RBackup.sh in the same directory:

/product/CSCO/CSCOpx/campus/etc/cwsi/ReportArchives

/product/CSCO/CSCOpx/htdocs/campus/maps

Backing up database cmfDb:Adaptive Server Anywhere Command File Hiding Utility Version 9.0.0.1364

Adaptive Server Anywhere Backup Utility Version 9.0.0.1364

(2843 of 2843 pages, 100% complete)

(4 of 4 pages, 100% complete)

Database backup completed

Command file created using Adaptive Server Anywhere Command File Hiding Utility.

Adaptive Server Anywhere Command File Hiding Utility Version 9.0.0.1364

Adaptive Server Anywhere Validation Utility Version 9.0.0.1364

No errors reported

Command file created using Adaptive Server Anywhere Command File Hiding Utility.

Backing up database aniDb:Adaptive Server Anywhere Command File Hiding Utility Version 9.0.0.1364

Adaptive Server Anywhere Backup Utility Version 9.0.0.1364

(30629 of 30629 pages, 100% complete)

(2 of 2 pages, 100% complete)

tail -f wrapper.out

24213: 3 2 2 5 4 0 ] ,\0\0 6\0FC iEB W\00105\0\0\0 ~9402\0 \0\0\01C\f

24213: \0\0 <\0\0\004 2 6 1 610 c 2 5 9 0 0 0 e d 6 6 5 d 5 ) <\0\0 J\0

24213: getpid() = 24213 [24211]

24213: send(12, 0x000634AD, 30, 0) = 30

24213: 1801\01E\t 3 * <\0\0\0\0030107 N E\0\0\0\0\007 S\010\0\0\0\0

24213: write(9, "\0\0\001\0\0\0\f\0\0\004", 12) = 12

24213: poll(0x00061760, 2, -1) = 2

24213: fd=8 ev=POLLIN rev=POLLIN

24213: fd=12 ev=POLLIN rev=POLLIN

24213: read(8, "\0\0\001\0\0\0\f\0\0\004", 12) = 12

24213: read(8, 0xFE603BC8, 12) Err#11 EAGA

I'm repeating the same step on another box right now.

Joe Clarke · ‎09-29-2008

It you run truss as truss -a -f -vall -wall -rall -o truss.out COMMAND , then truss will follow all forks, so you should see the relevant details. I should be able to get a copy of wrapper.pl tomorrow to investigate.

yjdabear · ‎09-29-2008

The command I use is:

truss -a -f -vall -rall -wall -o /path/to/wrapper.out /opt/CSCOpx/bin/perl /path/to/wrapper.pl /path/to/lms26backup

yjdabear · ‎09-30-2008

Same repeatable error on the second box. Doesn't seem like there's any clue why "RBackup.sh is not available" though:

Database backup completed

Command file created using Adaptive Server Anywhere Command File Hiding Utility.

Adaptive Server Anywhere Command File Hiding Utility Version 9.0.0.1364

Adaptive Server Anywhere Validation Utility Version 9.0.0.1364

No errors reported

Command file created using Adaptive Server Anywhere Command File Hiding Utility.

Backup completed: at [Mon Sep 29 16:24:36 2008]

************************************************************

Return status of CS backup script is: 0

Backing up CiscoView data now

Backing up CiscoView done

IPM backup starts now

IPM BackUp Loaction /path/to/lms26backup/0/ipm

Exiting the backup process.

Reason: RBackup.sh is not available

Joe Clarke · ‎09-30-2008

The error is misleading. It has nothing to do with the inability to find the RBackup.sh script. Instead, the error is seen if IPM is not installed on the LMS server. If IPM is installed, then there must be an error detecting it. The script checks /opt/CSCOpx/setup/ipm.info and /opt/CSCOpx/setup/ipm-s.info. Depending on which file exists (ipm-s.info is preferred), then the file is search for a line starting with VERSION. If both files or not found, or the existing file does not have a VERSION line, then wrapper.pl will terminate.

yjdabear · ‎09-30-2008

The two files look normal.

less /opt/CSCOpx/setup/ipm.info

PKGCATEGORY=Application

CATEGORY="application"

VERSION=2.6

NAME=Internetwork Performance Monitor

DEPENDS="CSCOcwbS, cdone"

PROD_INFO=ipm

PRODNAME=Internetwork Performance Monitor

INSTDATE=04-24-2007 0:41:49

SSL_COMPLIANT=YES

SUITE=ipm

TAGS=CSCOdummy

less /opt/CSCOpx/setup/ipm-s.info

VERSION=2.6

NAME=Internetwork Performance Monitor Server

DEPENDS="CSCOcwbS"

PRODNAME=Internetwork Performance Monitor

INSTDATE=04-24-2007 0:41:49

IPM is running, but there's been no active collector for a long time. So old data may well have aged out by now. Could this explain why wrapper.pl fails to detect IPM?

ps -ef |grep -i cscoipm

casuser 3401 1 0 Aug 17 ? 16:11 /product/CSCO/CSCOcwbS/db/bin/dbsrv9 -n ipm /product/CSCO/CSCOcwbS/db/CSCOipm/I

casuser 3370 1 0 Aug 17 ? 0:14 /product/CSCO/CSCOipmClient/jre/bin/../bin/sparc/native_threads/java -DIPM:Gate

casuser 3611 3512 0 Aug 17 ? 0:01 /product/CSCO/CSCOipm/bin/CWB_ipmRTPServer -ORBagentPort 44342 -PMCserverName I

casuser 3577 3512 0 Aug 17 ? 0:01 /product/CSCO/CSCOipm/bin/CWB_ipmConfigServerd -ORBagentPort 44342 -PMCserverNa

root 22889 13432 0 14:36:10 pts/2 0:00 grep -i cscoipm

casuser 3550 3512 0 Aug 17 ? 0:02 /product/CSCO/CSCOipm/bin/CWB_ipmSnmpd -ORBagentPort 44342 -PMCserverName IPMPr

casuser 3567 3512 0 Aug 17 ? 0:03 /product/CSCO/CSCOipm/bin/CWB_ipmData_colld -ORBagentPort 44342 -PMCserverName

casuser 3618 3512 0 Aug 17 ? 0:01 /product/CSCO/CSCOipm/bin/CWB_ipmDataViewServer -ORBagentPort 44342 -PMCserverN

Joe Clarke · ‎09-30-2008

Actually, I was looking at the LMS 3.0 version of wrapper.pl. The LMS 3.1 version is different. In that version, the error is accurate. Wrapper.pl is looking for RBackup.sh in the same directory as wrapper.pl. It does a Perl "-e" test for file existence. The code looks good.

yjdabear · ‎09-30-2008

Any clue in truss.out why it can't find RBackup.sh?

Joe Clarke · ‎09-30-2008

It stats the file in the current working directory, and the OS returns ENOENT indicating it doesn't exist. The only thing I can think of is that wrapper.pl changes its cwd, but I can't see where that happens in the code. There's not enough in the truss to see what the cwd is.

yjdabear · ‎09-30-2008

I'll put wrapper.pl in Perl debug to see if that gets more details into truss.out.

Joe Clarke · ‎09-30-2008

It suddenly occurred to me another reason why stat is returning ENOENT. Where is wrapper.pl and RBackup.sh (i.e. what is the fully-qualified path)?

yjdabear · ‎09-30-2008

They were in /product/CSCO/cwkbackups/wrapper/, along with the original SolarisLMSBackup.tar. Right now, I've deleted SolarisLMSBackup.tar. I've also moved the scripts to /product/CSCO/backupscript/. Something like the two above caused the installation of another third-party commercial app to fail.

Joe Clarke · ‎09-30-2008

I assume /product/CSCO is your NMSROOT. If so, then my theory is probably wrong since you'd be seeing other problems with LMS. However, you could unmount the underlying file system, and check the mount point perms. They must be 0755.