Questions after LMS 2.6 upgrade from LMS 2.2

Answered Question
Apr 3rd, 2007
User Badges:
  • Gold, 750 points or more

1. Is there a GUI counterpart to modifying NMSROOT/MDC/tomcat/webapps/rme/WEB-INF/classes/com/cisco/nm/rmeng/csc/data/Collector.properties? I can't find it. Is LMS 2.5/2.6 (RME 4.x) supposed to pick up the custom syslog_info location from LMS 2.2 (RME 3.5)? If there's no GUI, how do I make LMS 2.6 re-read Collector.properties, without restarting dmgtd?


2. Is DCR supposed to pick up the entire RME 3.5 inventory and all custom "Views" during the LMS 2.2 -> 2.6 (via Dec05 update) upgrade? DCR is missing about 20% of the devices after the upgrade. Aftering importing the full inventory via a csv file, I get duplicates: one with just the hostname, one with hostname.fqdn.com. Even if I have All Devices folder checked, my search for which devices with names containing "fqdn.com" (the ones imported via the csv) returns nothing.

Correct Answer by Joe Clarke about 10 years 3 months ago

What is PID 1006? I don't think this is SyslogCollector.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (3 ratings)
Loading.
Joe Clarke Tue, 04/03/2007 - 20:28
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

1. There is no GUI counterpart, but LMS 2.6 does offer a script, NMSROOT/bin/syslogConf.pl that allows one to configure many of the properties in Collector.properties. This script requires dmgtd to be down. However, if you modify Collector.properties directly, all you need to restart is SyslogCollector.


2. Yes, the upgrade should preserve the full inventory from RME 3.5. Importing a CSV file from RME 3.5 into a clean DCR should not result in duplicates. However, if you already had some of the devices in DCR, then duplicates can occur. DCR does not do any resolution of duplicates itself. It leaves that task up to the other applications. How and where did you do the fqdn.com search?

yjdabear Wed, 04/04/2007 - 06:00
User Badges:
  • Gold, 750 points or more

1. My syslog_info is on a read-only NFS export mounted on the LMS server. I had modified Collector.properties and restarted dmgtd earlier. It didn't work. I ran the pretty barebone syslogConf.pl, got this output:


Enter Your Choice: 4



INFO: You have opted to change Syslog File Location

Old Sylog Directory : /var/log/


Enter Full Path of New Syslog Directory: /var/adm/CSCOpx/syslog/

chown: /var/adm/CSCOpx/syslog/syslog_info: Read-only file system

chmod: WARNING: can't change /var/adm/CSCOpx/syslog/syslog_info


Sylog file location changed from: /var/log/

to: /var/adm/CSCOpx/syslog


Here's SyslogConf.log:


----------------Change Sylog File Location-----------------

Create new Syslog Directory

newSyslogLoc /var/adm/CSCOpx/syslog/

newSylsogLoc /var/adm/CSCOpx/syslog

Cannot Open File /var/adm/CSCOpx/syslog/syslog_info

Cannot Close File /var/adm/CSCOpx/syslog/syslog_info

---------Change Sylog File Location in syslog.conf------

Updated /etc/syslog.conf.

Updated /opt/CSCOpx/MDC/tomcat/webapps/rme/WEB-INF/classes/com/cisco/nm/rmeng/cs

c/data/Collector.properties.

----------------Change Sylog File Location in logstat.conf-----------------


Must I mount the syslog as read-write? What permissions does SyslogCollect expect syslog_info to be? The syslog_info itself is a symlink pointing to the real syslog file on the other box, since another app is using the original and Solaris won't let one file read by two procs concurrently. The symblink's permission is 777, while the original is as follows:


-rw-r--r-- 1 root other


This arrangement has worked fine with LMS 2.2.


Does installing a Device Update pkg for CiscoView have the same effect as restarting dmgtd?



Joe Clarke Wed, 04/04/2007 - 07:57
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Using an external syslog_info on a read-only file system was not test, and thus not supported. I can't say if this will work in LMS 2.6 or not. The error above indicates that it will not. However, the perms that syslogConf.pl tries to set are casuser:casusers 0664. However, this isn't needed as the standard syslog_info perms are root:sys 0664. You might try enabling debugging in Collector.properties, and see if there are any obvious errors when SyslogCollector tries to use this file.


As for CiscoView package installation, it is not quite the same as a dmgtd restart. PSU will actually stop all the daemons by hand, then restart them ALL (even the transients). dmgtd itself never stops. But for purposes of restarting the application daemons, it is sufficient.

yjdabear Wed, 04/04/2007 - 08:21
User Badges:
  • Gold, 750 points or more

Re-reading the error message, it seems as if SyslogCollector was trying to create a directory.


Is setting Logging Level to Deubg in RME->System Pref->Loglevel Settings-> SyslogAnalyzer the GUI equivalent to turning on SyslogCollector debugging?


A new question: There's a way to modify a config file so CM 4.x User Tracking displays more than the default max of 1000 entries per page, right? I recall the max is 3000, though a search for "3000 tracking" turns up nothing.

Joe Clarke Wed, 04/04/2007 - 08:31
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

No, the only way to enable SyslogCollector debugging is in Collector.properties. This will be changed in LMS 3.0.


The maximum number of entries per page in UT cannot be changed. This value is hard-coded.

Joe Clarke Wed, 04/04/2007 - 09:23
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

There is no indication of any problem here. It just looks like no new messages arrived in the syslog_info file. The daemon is properly starting, and does appear to have successfully opened the file. You can confirm that by using lsof on the PID of SyslogCollector.

yjdabear Wed, 04/04/2007 - 10:57
User Badges:
  • Gold, 750 points or more

I see the NFS mount opened:


cwjava 24267 casuser 11r VREG 322,1 84716087 172 /var/adm/CSCOpx/syslog (prod-nms:/var/logs/)


I can see the /var/adm/CSCOpx/syslog/syslog_info getting new syslogs from the CLI, so could LMS 2.2. Doesn't this mean it's a defect in LMS 2.6's SyslogCollector?


Can Remote SyslogCollector read a local file such as /var/logs/blah/blah/infolog, or is it hardcoded to /var/log/syslog_info? I think I might have to start exploring the RSC route.


Joe Clarke Wed, 04/04/2007 - 11:07
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

There can be no defect since this configuration isn't officially supported. But I've seen no errors so far to indicate the cause of the problem.


SyslogCollector can read from any file. It is not hardcoded to syslog_info. But it looks like to specify a different file name, you will have to modify Collector.properties directly, then restart SyslogCollector.


The RSC option would be the supported way of doing this.

yjdabear Wed, 04/04/2007 - 12:30
User Badges:
  • Gold, 750 points or more

I must be missing a step or something. Even having SyslogCollector reading a local original /var/log/syslog_info doesn't seem to make it collect. Nothing is recorded by SyslogCollector's debug either.

Joe Clarke Wed, 04/04/2007 - 12:54
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

How are the new messages getting into this local syslog_info? Provided that the Collector.properties points to this file, SyslogCollector has been restarted since making the modifications, and the permissions allow SyslogCollector to read the file, you should get the messages processed (or at least filtered).

yjdabear Thu, 04/05/2007 - 07:03
User Badges:
  • Gold, 750 points or more

The issue with local /var/log/syslog_info seems related to a problem with NMSROOT/bin/syslogConf.pl. I don't know why I'm the only one that seems to experience this, but apparently if I use NMSROOT/bin/syslogConf.pl to configure syslog location, it puts in blank spaces instead of tabs when entering "local7.info /var/log/syslog_info" in /etc/syslog.conf (seen with ":set list" in vi).



******************

/opt/CSCOpx/bin/perl /opt/CSCOpx/bin/syslogConf.pl

...

[Q] Quit

Enter Your Choice: 4



INFO: You have opted to change Syslog File Location

Old Sylog Directory : /var/log/


Enter Full Path of New Syslog Directory: /var/log


Sylog file location changed from: /var/log/

to: /var/log


[1] Change Syslog Analyzer Port

[2] Change Syslog Collector Port

[3] Configure Remote Syslog Collector(RSAC) Address and Port

[4] Change Syslog File Location

[Q] Quit

Enter Your Choice: Q


INFO: You have opted to Quit..

INFO: For more information, please check the log file /var/adm/CSCOpx/log/SyslogConf.log


INFO: Please restart syslog service [/etc/init.d/syslog stop , /etc/init.d/syslog start].


INFO: Please restart daemon manager [/etc/init.d/dmgtd start].



/usr/sbin/syslogd -d

...

writemsg(7): Logging msg 'syslogd: line 35: unknown priority name "info /var/log/syslog_info"' to FILE /var/adm/messages

init(1): accepting messages from local system

set_udp_buffer(1): allocate 262144 for fd 3

init(1): accepting messages from remote

init(1): syslogd: started

main(1): off & running....

hostname_lookup(17): hostname_lookup started

net_poll(18): net_thread started

net_poll(18): received message from 172.18.1.4.197.232



172.18.1.4 is a lab router on which I did a "conf t" to generate a sys-5 syslog, of course.



vi /etc/syslog.conf

...

$

# Added for Cisco Syslog Analyzer (begin)$

local7.info /var/log/syslog_info$

# Added for Cisco Syslog Analyzer (end)$

local1.info^I^I^I^I/var/adm/messages$

#BEGIN CSCOmd - DO NOT EDIT THESE COMMENTS OR CONTENTS CONTAINED WITHIN - local2 1$

#$

local2.emerg;local2.alert;local2.crit;local2.err;local2.warning;local2.notice;local2.info;local

2.debug^I/var/adm/CSCOpx/log/dmgtd.log$

#$

#END CSCOmd DO NOT EDIT BEFORE THIS LINE 1$

:set list


Upon switching to tabs:


net_poll(19): received message from 172.18.1.4.197.232

writemsg(12): Logging msg '132: *Apr 5 11:01:42.524: %SYS-5-CONFIG_I: Configured from console by my-tacacs-id on vty0 (10.254.74.1)' to FILE /var/log/syslog_info


Also, I notice the #BEGIN CSCOmd section used to be all local0 with LMS 2.2. Is LMS 2.6 switching to local2 instead?

Joe Clarke Thu, 04/05/2007 - 07:14
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Wow, nice catch on the spaces vs. tabs thing. This is definitely a bug. CSCOmd will use local0 if it is free. Do you have local0 used for something else?

Joe Clarke Thu, 04/05/2007 - 07:39
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

I filed CSCsi42348 to track this problem. A quick fix is to modify line 334 of syslogConf.pl, and change the line to:


$newValue="\t\t".$newValue."\n";

yjdabear Thu, 04/05/2007 - 08:30
User Badges:
  • Gold, 750 points or more

Does Syslog Analysis report on devices not managed (question mark on my lab router)? I'm still not seeing anything in Syslog report.


Can you attach a screenshot of what a working RME->Tools->Syslog->Syslog Collector Status looks like (preferrably without RSC), and a debug SyslogCollector.log of that? Particularly, I'm not sure these are positive or negative messages:


SyslogCollector - [Thread: main] DEBUG, 05 Apr 2007 12:19:02,354, Entered resurrectSubscribers()...

SyslogCollector - [Thread: main] DEBUG, 05 Apr 2007 12:19:02,354, SCRecoveryAssistant - Entering retrieveSubscriptionInfo()

SyslogCollector - [Thread: main] DEBUG, 05 Apr 2007 12:19:02,359, FcssEmblemProcessor - setPause() has been called with true

SyslogCollector - [Thread: main] DEBUG, 05 Apr 2007 12:19:02,359, FilterProcessor - setPause() has been called with true

SyslogCollector - [Thread: main] DEBUG, 05 Apr 2007 12:19:02,359, SyslogCollectorEngine - Service has been paused.

SyslogCollector - [Thread: main] DEBUG, 05 Apr 2007 12:19:02,360, No subscriber data file found. Not attempting a resurrection.

SyslogCollector - [Thread: main] DEBUG, 05 Apr 2007 12:19:02,360, Initializing Cleaner...

SyslogCollector - [Thread: main] DEBUG, 05 Apr 2007 12:19:02,361, Assigning shutdown hook...

SyslogCollector - [Thread: main] DEBUG, 05 Apr 2007 12:19:02,362, Syslog Collector Engine is running...

SyslogCollector - [Thread: main] DEBUG, 05 Apr 2007 12:19:04,082,




Joe Clarke Thu, 04/05/2007 - 08:42
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Attached is a screenshot, but I cannot get a debug right now. But when it's working, you will see SyslogCollector read then ew messages right out of the syslog file. You will see the raw messages being read in SyslogCollector.log, then you will see SyslogCollector decide if it needs to be filtered, or passed on to a subscribed analyzer.


For Sysloganalyzer to write the messages to the database for purposes of standard reporting, the messages need to be coming from RME managed devices.



yjdabear Thu, 04/05/2007 - 08:58
User Badges:
  • Gold, 750 points or more

Based on what you said, I think it might be because of the mix-bag of managed vs unmanaged state (and duplicates) of my DCR inventory.


Do you think these pkgchk failure could have anything to do the syslog problem too?



Attachment: 
Joe Clarke Thu, 04/05/2007 - 09:20
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

These pkgchk messages are fine. Even if the devices were not managed, you should still see SyslogCollector detecting new messages in the syslog file. It looks like you might not have subscribed your analyzer to the collector. What does YOUR Syslog Collector status look like?

yjdabear Thu, 04/05/2007 - 09:59
User Badges:
  • Gold, 750 points or more

I was wondering about that. There's no entry where "rtp-main" is on your screenshot, just "No Records". When I tried to add a subscription to "127.0.0.1" or "the external IP addr in gatekeeker.cfg" or "hostname", I got:


SLCA0126: Could not subscribe to the Collector.

1.Check whether the collector is running.

2.Make sure that SSL certificates are imported/exported correctly and process restarted.

3.Check whether the Certificates exported/imported are valid and have not expired.

4.Check whether SyslogAnalyzer process is running.


I just assume something else is broken, when that gets fixed, Syslog Analyzer will subscribe to Syslog Collector on the localhost automatically.

Joe Clarke Thu, 04/05/2007 - 10:34
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Attach an output of lsof for the SyslogCollector process as well as the AnalyzerDebug.log.

Joe Clarke Thu, 04/05/2007 - 11:00
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Please send your current Collector.properties and the output of lsof -i :4444 and lsof -i :3333.

yjdabear Thu, 04/05/2007 - 11:12
User Badges:
  • Gold, 750 points or more

vi /opt/CSCOpx/MDC/tomcat/webapps/rme/WEB-INF/classes/com/cisco/nm/rmeng/csc/data/Collector.properties

"/opt/CSCOpx/MDC/tomcat/webapps/rme/WEB-INF/classes/com/cisco/nm/rmeng/csc/data/Collector.prop

erties" 25 lines, 1000 characters

[Hit return to continue]

# Timezone related properties

TIMEZONE=EST

COUNTRY_CODE=USA

TIMEZONE_FILE=$NMSROOT/MDC/tomcat/webapps/rme/WEB-INF/classes/com/cisco/nm/rmeng/fcss/data/Time

Zone.lst


# General properties

SYSLOG_FILES=/var/log/syslog_info

#SYSLOG_FILES=/var/adm/CSCOpx/syslog/syslog_info


DEBUG_CATEGORY_NAME=SyslogCollector

DEBUG_FILE=/var/adm/CSCOpx/log/SyslogCollector.log

#DEBUG_LEVEL=INFO

DEBUG_LEVEL=DEBUG

DEBUG_MAX_FILE_SIZE=15MB

DEBUG_MAX_BACKUPS=3

DOWNTIME_DIR=$NMSROOT/MDC/tomcat/webapps/rme/WEB-INF/classes/com/cisco/nm/rmeng/csc/data

FILTER_DUMP_FILE=$NMSROOT/MDC/tomcat/webapps/rme/WEB-INF/classes/com/cisco/nm/rmeng/csc/data/fi

lters.dat


# Miscellaneous properties. These are not important to users.

READ_INTERVAL_IN_SECS=1

QUEUE_CAPACITY=100000

PARSER_FILE=$NMSROOT/MDC/tomcat/webapps/rme/WEB-INF/classes/com/cisco/nm/rmeng/fcss/data/Format

Parsers.lst

SUBSCRIPTION_DATA_FILE=$NMSROOT/MDC/tomcat/webapps/rme/WEB-INF/classes/com/cisco/nm/rmeng/csc/d

ata/Subscribers.dat

FILTER_THREADS=1

COLLECTOR_PORT=4444


lsof -i :4444

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

java 1006 adsc01d 21u IPv4 0x3001860b0e0 0t0 TCP *:4444 (LISTEN)


lsof -i :3333

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

cwjava 4557 casuser 27u IPv4 0x30029b86360 0t0 TCP *:3333 (LISTEN)

Correct Answer
Joe Clarke Thu, 04/05/2007 - 11:20
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

What is PID 1006? I don't think this is SyslogCollector.

yjdabear Thu, 04/05/2007 - 11:23
User Badges:
  • Gold, 750 points or more

axxx##d 1006 998 0 Mar 22 ? 424:14 /product/confignia/dist/external/jdk-1.4.2-SunOS-sparc/bin/java -ea -DCOLLATION


It appears to be a Confignia product. It's install directory /product/confignia is 100% full. This is hogging port 4444?


Joe Clarke Thu, 04/05/2007 - 11:26
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Yeah. If you can't free up tcp/4444, that's not a problem. Either use syslogConf.pl or just modify Collector.properties directly to pick a new, free, TCP port between 1024 and 5000 (modify COLLECTOR_PORT). If you modify Collector.properties directly, then just restart SyslogCollector, else restart dmgtd.


Once that's done, you should be able to subscribe your Analyzer to your Collector (use the short hostname of the server and NOT 127.0.0.1). then you should be good to go.

yjdabear Thu, 04/05/2007 - 11:29
User Badges:
  • Gold, 750 points or more

Thanks, I'll try it out. Is the subscription to local Collector always a manual process? I might have missed it, but I don't think I saw any mention of that in the documentation.

Joe Clarke Thu, 04/05/2007 - 11:33
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

No, it's usually automatic. Had the port not been occupied, it would have just worked.

yjdabear Fri, 04/13/2007 - 09:35
User Badges:
  • Gold, 750 points or more

It works out great, even with NFS-mounted syslog.


Will Cisco take a request to have line 316-ish of syslogConf.pl either removed or substantially improved? As you noted before, this step is rather pointless, since the syslog file on Sol should be owned by root. As it is, this section has no other practical purpose than [unintentionally, I think] preventing syslogConf.pl from working on a read-only NFS mount (unsupported and all).


#else

#{

#

# system("chown -R casuser:casusers $newSyslogFile");

# system("chmod 664 $newSyslogFile");

#}


A new question: Should all the NetConfig jobs be preserved by the LMS 2.6 upgrade? The Job list is empty.


Joe Clarke Fri, 04/13/2007 - 10:17
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Actually, the problem is worse than you think. This is a security hole. We should not chown the syslog file to casuser. The perms are also too open. This will be fixed, but we will need to do a chmod to at least 0644. If this fails, though, it would not be fatal.


As for Netconfig jobs, no jobs or job-related data is migrated from 2.2 to 2.6.

Joe Clarke Sat, 04/14/2007 - 19:58
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

I filed CSCsi52968 to track the issue with permissions on syslogConf.pl adjusting the permissions on the syslog message file.

Actions

This Discussion