Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

Verifying NFS

Hello all,

Hopefully someone can help me out as I dont know linux at all to verify some information.

I have MARS configured to backup to a NFS server. I recently received a notification from MARS stating that one of the backups failed.

Because I do not have access to the NFS server (Managed Services is why), what other ways can I verify that the NFS backup is succeeding? Whenever I attempt to use the GUI to pull raw messages from the NFS server it essentially times out.

Anything on the cli available? Any and all help/ideas are appreciated.

MARS is running 4.2.6 ( 2458 ). Yea, old, but customer absolutely refuses to give us the time of day to upgrade it.

16 REPLIES

Re: Verifying NFS

You can setup a second NFS server temporarity to see if that works. You can even run NFS on windows with the help of some software/add-on (as per MARS docs).

Alternatively you could go to MARS >> Admin >> System Maintenance >> Set Runtime Logging Levels

pnarchiver = DEBUG

And see if you get any more meaningful logs.

Regards

Farrukh

New Member

Re: Verifying NFS

FYI,

It isn't until a later release that Cisco actually includes

A) more meaningful logs for NFS

B) Event corrolation for NFS failures for the MARS appliance so it will notify more then just the account associated with "pnadmin"

Gold

Re: Verifying NFS

"MARS is running 4.2.6 ( 2458 ). Yea, old, but customer absolutely refuses to give us the time of day to upgrade it."

well, they still get to check the box when the auditor visits;-) sad isn't it. The NFS logging enhancements is one of MANY reasons to make sure MARS is kept up to date. FWIW, we implemented our own NFS monitoring years ago because it would frequently break with no alerts being generated.

New Member

Re: Verifying NFS

Mind if I ask how you monitor the NFS?

Gold

Re: Verifying NFS

Each day has it's own directory in the archive.

We just check the modify time on the directory. The NFS archiving has gotten MUCH better, but back in the day this worked pretty good to let us know when it was broken.

New Member

Re: Verifying NFS

Ah. Makes sense, and was my first instinct. Unfortunately, we're a managed service provider. For liability issues, we do not have access to view the server.

Re: Verifying NFS

Also there is a 'pnexp' command available on the CLI. Even tough it is meant to export 4.x database before importing it into a new 5.x box (Generation 1). It can be used for 'immediate backups' as per the Cisco engineer on this thread:

http://forum.cisco.com/eforum/servlet/NetProf?page=netprof&forum=Expert%20Archive&topic=Security&topicID=.ee7f99a&CommCmd=MB%3Fcmd%3Dpass_through%26location%3Doutline%40%5E1%40%40.2cbeaa98/18#selected_message

Once you enter the pnexp CLI

Do the following:

export config 10.1.1.1:/archive.

Then enter the following command:

pnexp> status

Data exporting process is currently running, please use command 'log {all|recent}' to view running logs and/or progress.

pnexp> log

Aug 25 09:44:47.192 2008@LM_INFO@Thread 1024:Number of events exported: 1227522, ~0.18% completed, overall speed = 30688.05 eps

Aug 25 09:44:47.192 2008@LM_INFO@Thread 1024:Estimated-Time-To-Complete: 6 hours 18 minutes

CTRL-C

As you can see I'm doing a full backup here (config+event/report data etc), don't run that just to 'test' NFS. Its better to use the 'config' option :)

Regards

Farrukh

Gold

Re: Verifying NFS

You can use tcpdump from the command line. Try this:

tcpdump -X host

I'm not sure how intelligent it is, but some level of NFS decoding is performed by tcpdump and it may give you enough of a hint to get you going in the right direction.

New Member

Re: Verifying NFS

Thats actually a really good idea. Of course, I keep thinking about how NFS is UDP so that a packet capture wouldn't be all that great. But If i do the dump at the same time requesting a file, i can confirm that data is indeed on the server and they're talking.

Thanks for the input!

Gold

Re: Verifying NFS

Actually, it's TCP...at least the version I'm on now. I do think it used to be UDP though. Another reason to patch;-) Even with UDP, there will be application level acknowledgments, so tcpdump should work fine.

New Member

Re: Verifying NFS

So, interesting enough, here is that tcpdump for communication to my NFS server. As you can see, it'd not really seeing a response from the NFS server. But the amusing part is, this is actually from when the NFS - Get Archived Files worked.

[pnadmin]$ tcpdump -i eth0 ip host 10.17.1.252

tcpdump: listening on eth0

16:08:36.407598 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)

16:08:37.400025 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)

16:08:39.400024 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)

16:08:43.400466 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)

16:08:51.400025 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)

16:09:07.400358 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)

16:09:39.400034 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)

16:10:39.400033 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)

16:11:39.400041 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)

16:13:07.541864 10.17.1.252.netbios-dgm > 10.17.1.255.netbios-dgm: NBT UDP PACKET(138)

Gold

Re: Verifying NFS

interesting. pulling files should definitely result in a whole lot more activity than that. You don't really know what is happening "behind the scenes". MARS may not actually fetch data it has local for example (i.e. previously requested data, or data in the DB). The getattr should result in a response too.

New Member

Re: Verifying NFS

I've actually expanded the tcpdump capture to get everything and then looked through it. For example, when I am doing that it isn't capturing my browsing the webgui. So how broken is tcpdump?

Gold

Re: Verifying NFS

I'm not aware of it ever being broken in MARS, but I wouldn't have paid that much attention if it was. The MARS box isn't using both interfaces is it?

Re: Verifying NFS

Perhaps you did this capture at a time at which the MARS was just sending 'some' of its data to the NFS? The real thing happens at 1 or 2 AM, but some types of data are backed up after a certain amount of time throughout the day. As per the guide:

"While you cannot schedule when the data backup occurs, the MARS Appliance

performs a configuration backup every morning at 2:00 a.m. and events are archived every hour. The

configuration backup can take several hours to complete"

http://www.cisco.com/en/US/docs/security/security_management/cs-mars/4.3/installation/guide/admin.html#wpxref73260

Also I hope there is no firewall/filtering device between the MARS and NFS server?

Regards

Farrukh

New Member

Re: Verifying NFS

The Mars has both interfaces assigned. But the NFS server is on the local network of the NIC we're connecting to. In addition, just to make sure I actually did packet captures on both interfaces to see what was going on. Still not seeing the traffic I would be generating by using the web gui.

Farrukh,

Thing about the NFS/MARS TCPDump is that this was when retrieving data from the NFS server worked through the GUI. In other words, it was successful in pulling a raw events log from the server, and we can see it talking to the device to do just that, just not any responses.

311
Views
9
Helpful
16
Replies