Verifying NFS

Unanswered Question
Aug 21st, 2008
User Badges:

Hello all,


Hopefully someone can help me out as I dont know linux at all to verify some information.


I have MARS configured to backup to a NFS server. I recently received a notification from MARS stating that one of the backups failed.


Because I do not have access to the NFS server (Managed Services is why), what other ways can I verify that the NFS backup is succeeding? Whenever I attempt to use the GUI to pull raw messages from the NFS server it essentially times out.


Anything on the cli available? Any and all help/ideas are appreciated.


MARS is running 4.2.6 ( 2458 ). Yea, old, but customer absolutely refuses to give us the time of day to upgrade it.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4.5 (2 ratings)
Loading.
Farrukh Haroon Fri, 08/22/2008 - 21:38
User Badges:
  • Red, 2250 points or more

You can setup a second NFS server temporarity to see if that works. You can even run NFS on windows with the help of some software/add-on (as per MARS docs).


Alternatively you could go to MARS >> Admin >> System Maintenance >> Set Runtime Logging Levels


pnarchiver = DEBUG


And see if you get any more meaningful logs.


Regards


Farrukh



spellluck Mon, 08/25/2008 - 06:24
User Badges:

FYI,


It isn't until a later release that Cisco actually includes


A) more meaningful logs for NFS

B) Event corrolation for NFS failures for the MARS appliance so it will notify more then just the account associated with "pnadmin"

mhellman Mon, 08/25/2008 - 06:29
User Badges:
  • Blue, 1500 points or more

"MARS is running 4.2.6 ( 2458 ). Yea, old, but customer absolutely refuses to give us the time of day to upgrade it."


well, they still get to check the box when the auditor visits;-) sad isn't it. The NFS logging enhancements is one of MANY reasons to make sure MARS is kept up to date. FWIW, we implemented our own NFS monitoring years ago because it would frequently break with no alerts being generated.

mhellman Mon, 08/25/2008 - 07:07
User Badges:
  • Blue, 1500 points or more

Each day has it's own directory in the archive.

We just check the modify time on the directory. The NFS archiving has gotten MUCH better, but back in the day this worked pretty good to let us know when it was broken.

spellluck Mon, 08/25/2008 - 07:24
User Badges:

Ah. Makes sense, and was my first instinct. Unfortunately, we're a managed service provider. For liability issues, we do not have access to view the server.

Farrukh Haroon Mon, 08/25/2008 - 04:39
User Badges:
  • Red, 2250 points or more

Also there is a 'pnexp' command available on the CLI. Even tough it is meant to export 4.x database before importing it into a new 5.x box (Generation 1). It can be used for 'immediate backups' as per the Cisco engineer on this thread:


http://forum.cisco.com/eforum/servlet/NetProf?page=netprof&forum=Expert%20Archive&topic=Security&topicID=.ee7f99a&CommCmd=MB%3Fcmd%3Dpass_through%26location%3Doutline%40%5E1%40%40.2cbeaa98/18#selected_message


Once you enter the pnexp CLI


Do the following:


export config 10.1.1.1:/archive.


Then enter the following command:


pnexp> status

Data exporting process is currently running, please use command 'log {all|recent}' to view running logs and/or progress.


pnexp> log

Aug 25 09:44:47.192 2008@[email protected] 1024:Number of events exported: 1227522, ~0.18% completed, overall speed = 30688.05 eps

Aug 25 09:44:47.192 2008@[email protected] 1024:Estimated-Time-To-Complete: 6 hours 18 minutes


CTRL-C


As you can see I'm doing a full backup here (config+event/report data etc), don't run that just to 'test' NFS. Its better to use the 'config' option :)


Regards


Farrukh

mhellman Mon, 08/25/2008 - 06:12
User Badges:
  • Blue, 1500 points or more

You can use tcpdump from the command line. Try this:


tcpdump -X host


I'm not sure how intelligent it is, but some level of NFS decoding is performed by tcpdump and it may give you enough of a hint to get you going in the right direction.

spellluck Mon, 08/25/2008 - 07:17
User Badges:

Thats actually a really good idea. Of course, I keep thinking about how NFS is UDP so that a packet capture wouldn't be all that great. But If i do the dump at the same time requesting a file, i can confirm that data is indeed on the server and they're talking.


Thanks for the input!

mhellman Mon, 08/25/2008 - 07:22
User Badges:
  • Blue, 1500 points or more

Actually, it's TCP...at least the version I'm on now. I do think it used to be UDP though. Another reason to patch;-) Even with UDP, there will be application level acknowledgments, so tcpdump should work fine.

spellluck Tue, 08/26/2008 - 13:14
User Badges:

So, interesting enough, here is that tcpdump for communication to my NFS server. As you can see, it'd not really seeing a response from the NFS server. But the amusing part is, this is actually from when the NFS - Get Archived Files worked.



[pnadmin]$ tcpdump -i eth0 ip host 10.17.1.252


tcpdump: listening on eth0


16:08:36.407598 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)


16:08:37.400025 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)


16:08:39.400024 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)


16:08:43.400466 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)


16:08:51.400025 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)


16:09:07.400358 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)


16:09:39.400034 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)


16:10:39.400033 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)


16:11:39.400041 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)


16:13:07.541864 10.17.1.252.netbios-dgm > 10.17.1.255.netbios-dgm: NBT UDP PACKET(138)


mhellman Tue, 08/26/2008 - 13:56
User Badges:
  • Blue, 1500 points or more

interesting. pulling files should definitely result in a whole lot more activity than that. You don't really know what is happening "behind the scenes". MARS may not actually fetch data it has local for example (i.e. previously requested data, or data in the DB). The getattr should result in a response too.

spellluck Tue, 08/26/2008 - 14:03
User Badges:

I've actually expanded the tcpdump capture to get everything and then looked through it. For example, when I am doing that it isn't capturing my browsing the webgui. So how broken is tcpdump?

mhellman Tue, 08/26/2008 - 14:05
User Badges:
  • Blue, 1500 points or more

I'm not aware of it ever being broken in MARS, but I wouldn't have paid that much attention if it was. The MARS box isn't using both interfaces is it?

Farrukh Haroon Tue, 08/26/2008 - 18:31
User Badges:
  • Red, 2250 points or more

Perhaps you did this capture at a time at which the MARS was just sending 'some' of its data to the NFS? The real thing happens at 1 or 2 AM, but some types of data are backed up after a certain amount of time throughout the day. As per the guide:


"While you cannot schedule when the data backup occurs, the MARS Appliance

performs a configuration backup every morning at 2:00 a.m. and events are archived every hour. The

configuration backup can take several hours to complete"


http://www.cisco.com/en/US/docs/security/security_management/cs-mars/4.3/installation/guide/admin.html#wpxref73260


Also I hope there is no firewall/filtering device between the MARS and NFS server?


Regards


Farrukh

spellluck Wed, 08/27/2008 - 05:25
User Badges:

The Mars has both interfaces assigned. But the NFS server is on the local network of the NIC we're connecting to. In addition, just to make sure I actually did packet captures on both interfaces to see what was going on. Still not seeing the traffic I would be generating by using the web gui.


Farrukh,


Thing about the NFS/MARS TCPDump is that this was when retrieving data from the NFS server worked through the GUI. In other words, it was successful in pulling a raw events log from the server, and we can see it talking to the device to do just that, just not any responses.

Actions

This Discussion