Syslog on remote storage

aieremias · ‎12-09-2009

Hi all,

We use LMS 3.2 with RME 4.3.1 on Solaris 10 and tried to change the parameter SYSLOG_FILES=/var/log/syslog_info in Collector.properties to a file on NFS mount. The remote syslog file resides on a NetApp Storage System and is accessible by casuser, we can do:
su - casuser
tail -100 remote_file
tail -f remote_file
and see logs.

We restart SyslogCollector and SyslogAnalyzer after editing Collector.properties, however RME SyslogCollector can't get logs from it.
We check this from RME > Tools > Syslog > Collector Status, columns are N/A

We can use any local file without problem and append logs from remote_file to local_file with:
tail -100 remote_file >> local_file. This works.

Normally SyslogCollector should not care if the file is local or remote.

The reason we want to use this configuration is that there is already a syslog server for all devices and don't want to send logs to a second one, the RME server. We prefer to direct SyslogCollector to read the remote_file.

All suggestions welcome!

Alex

Joe Clarke · ‎12-09-2009

Post the SyslogCollector.log and AnalyzerDebug.log.

aieremias · ‎12-09-2009

Hi jclarke,

attached is the relevant part of SyslogCollector.log. I did several tests with local and remote files, so I attach only a failed test with remote file.

I will post AnalyzerDebug.log as soon as I can get it from server.

Thank you,

Alex

Joe Clarke · ‎12-10-2009

The error points to SyslogCollector not being able to open /netlog/cisco/info.log for reading. I know you said you tested this cas casuser, but it's not working from SyslogCollector. SyslogCollector will attempt to open the file for reading, then seek to the end of the file. If either of these two operations fail, you will see the error you're seeing. Re-verify the permissions not only on the info.log file, but also on all of the subdirectories leading up to root. Also, unmount the NFS volume, and check the perms on the mount point. Make sure they are 0755.

aieremias · ‎12-11-2009

I checked that when I login as casuser I can do

cd /

ls

cd netlog

ls

cd cisco

ls

tail info.log

all directories have permission 0755, also the mount point /netlog when unmount has permission 0755

The end user has Cisco Support Contract and opened a TAC case. Of course all suggestions for the solution of this puzzle are welcome!

Thank you

Alex

yjdabear · ‎12-11-2009

Is remote_file on the remote syslog server 1) the real syslog file itself, or 2) a symbolic link to the syslog file? In the case of 1), is remote_file being read/opened by anything (process or product) on the remote syslog server? If the answer is "yes", you need to make it 2), and point CiscoWorks LMS' Syslog Analyzer to the symbolic link on the NFS mount.

aieremias · ‎12-12-2009

Remote_file is the real syslog file. I tried creating remote_file1 with the command tail -100 remote_file > remote_file1 and SyslogCollector can't read from remote_file1 too. Certainly no process reads from remote_file1.

Also tried a local symlink to remote_file, this failed. I will try remote symlink to remote_file when I go to this site again.

But I think that an application should work same way whether it reads the file directly or thru a symlink. Please correct me if wrong.

Joe Clarke · ‎12-13-2009

A symlink won't help. If SyslogCollector cannot read from the actual file, it won't be able to get it via a symlink. As I said in my previous post, truss would be very helpful at this point.

Joe Clarke · ‎12-13-2009

At this point, I would want to see the SyslogCollector.log with debugging enabled, and probably truss the startup of SyslogCollector to see exactly why it cannot operate on the log file. Therefore, opening a TAC SR was a good course of action.

aieremias · ‎12-13-2009

The SyslogCollector.log that I posted is with DEBUG_LEVEL=DEBUG but does not enlighten us.

Using truss seems a good approach, I can try it. Do you suggest something like:

pdterm SyslogCollector SyslogAnalyzer

truss "pdexec SyslogCollector"

pdexec SyslogAnalyzer

Another approach that I considered is snooping the nfs traffic. But both truss and nfs snoop need an internal understanding of what SyslogCollector does.

You are already ahead TAC in understanding the issue...

Joe Clarke · ‎12-13-2009

No, running truss like that will not work. You will need to run the process manually as casuser, and truss the actual VM execution (with appropriate arguments). This is not a very straight-forward process. TAC should be able to walk you through the necessary steps.

aieremias · ‎12-24-2009

Update:

After TAC suggestion I experimented a little on the nfs mount file and the minimum permissions that work.

With 644 SyslogCollector works

With 640 SyslogCollector does not work, although casuser is able to read it from "group"

So SyslogCollector somehow requires to get read access from "others" while standard utilities like tail work if they get read access from "group"

This is not optimal setting for a syslog file from a security viewpoint.

Happy Christmas to all!

Alex

Joe Clarke · ‎12-24-2009

There must be something going on in your UID/GID mapping. This should work. If you haven't restarted dmgtd since adding casuser to the others group, you might try that. SyslogCollector should behave like any other process except that it is spawned from dmgtd after dmgtd sets the UID and GID to casuser and casusers respectively.

aieremias · ‎01-09-2010

Hi Joe,

Yesterday I was at the customer site and checked the modification date of /etc/passwd and /etc/group, LMS was restarted two days after that.

Anyway we modified the file permissions to 644 as suggested by TAC and the problem was solved.

Thank you for your suggestions. Your posts in this forum have helped me a lot to understand LMS better!

Happy New Year,

Alex

Joe Clarke · ‎01-09-2010

That makes sense. If /etc/group was not readable by casuser, then the proper GIDs could not get assigned. I've never seen a really compelling reason to restrict access to /etc/passwd, either. Many applications require this file to be readable to get attributes such as home directory, shell, and GECOS.