cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
19420
Views
29
Helpful
9
Replies

Critical Alarm : User space core files were detected (WAAS)

ohaddad
Level 1
Level 1

I have major alarms on WAVE-274 (Edge-WAAS) and WAVE-7341(Core-WAAS).

Alarm Name: Core_dump

Alarm informations: "User space core files were detected"

Is it possible to learn more about this - what caused it, ...ect

Any ideas on how to proceed?

9 Replies 9

claudiols
Level 1
Level 1

Go to subdirectory crash, probably there is a dump file that you need to open a TAC case and send it

Hi Claudiols;

Thank you for your reply,

But I wanted to know the exact subdirectory crash, and the dump file name if it's possible.

thank

the name of  file can be very diferent, but the directory are

/local1/core_dir or /local1/crash

the file would be something like core.java.4.1.3.b55.cnbuild.5940.gz

What can I extract of this file core.java.4.1.3.b55.cnbuild.5940.gz ? It´s only information of
crash?

Only the TAC to resolve this problem?

regards

Hi,

This is correct; you should open a TAC case and attach not only the core or crash files found in the directories mentioned, but also a sysreport generated in the device where the issue was observed as this investigation encompasses the analysis of several logs and possible correlation with known issues.

To generate a sysreport and store it to an FTP server, use this form of the command: copy sysreport ftp server-ip remote-directory remote-file-name

Best regards,

Fabio Bergamo

Cisco TAC

hi! 4 years after I have the same problem, but in to directory core_dir no have files, in to the crash have the files above:

     size          time of last change             name
--------------  -------------------------          -----------
   17053068576  Tue Nov 11 11:18:43 2014           vmcore
           434  Tue Nov 11 13:53:42 2014           waas_version.txt

Can I delete vmcore? That file don't have extension .gz

Regards

Felix Arrieta
Cisco Employee
Cisco Employee

when you see alarm "core file detected" most of the time means that  an internal process crash, it is hard to tell what exactly happens by  looking at the core file itself  so you better open a TAC case.

If you still want to learn about it, I can help with the  following steps ( this is not documented anywhere so I don't promise  covering all the steps, you'd need to be already familiar with reviewing  WAAS logs to give it more sense)

scenario sample:

1- 'core files detected alarm shows up and you have no  idea what it means.

2-  We need to know when exactly the core file was created.

do WAE#dir   find the folders  "core_dir" and "crash"  and open them

with core_dir folder :

WAE#dir core_dir    

      size          time of last change             name              

--------------  -------------------------          -----------       

        618747  Sat Apr 19 06:30:14 2012  core.cnbuild.29405.gz

        361568  Mon Aug 08 13:20:22 2008  core.cnbuild.11689.gz

so we found two core files over here and it's respective  date.

***

delete the core files that are older ( deleting them fixes the  alarm issue, why would we need to be alarm for a core file from a long time ago?   corefiles are logs that help our developers to understand what  process failed so this means  whatever issue happen WAAS already recover  from it, at least that's what we expect.

< how to delete a core file>

WAE#cd core_dir

WAE#delfile core.cnbuild.11689.gz

give it a minute

WAE# show alarms < the alarm will go again >

***

So once we know the date when the core file was created go and check the *syslog*

If you don't know how to check the syslog:

a- log/save your putty session ( so you can later open the log on a wordpad or any application to simplify research)

b- WAE#type-tail syslog.txt 500 <  this number indicates how many lines you want the log to be, this can  vary depending on how much log lines your WAE generates or how long ago  the core file was created> for example if the core file was first  seen 24 hours ago, I guess a syslog of 200 lines should do it but it  could be more or less.

c- open the saved session(notepad or whatever..) search for the closer  date to the core file date's, and we should see  helpful logs that we could easily do research on the internet.

Most  of the time if the core file is coming from a problem that requires  urgent assistance you'll notice how the alarm that looks critical often repeats itself through the log ( because WAAS will try to recover from the unhealthy state) sometimes  you delete a core file*clear the alarm* and it comes back few hours  later, that's bad ..so you either help yourself trobleshooting it based on the logs or open a TAC case soon cause it might be causing trouble to your applications)

On the other hand if it was just one time behavior the core file can be easily deleted.

for example:

WAE device stops responding at 8:00 then it comes back online at 8:05,   when it came back  an internal process got crash, it then generates a core file when the service get's restore  (let's say at 8:15)   a week later you're trying to figure out what's the core file alarm about.

If you  follow the steps above  you'll end up reviewing the syslog and realize  that   before the core file was created the WAAS box  got rebooted  at 8:05!!  end of the research...you might think why was this box rebooting?  may  be a power outage,hardware issue or someone messing up  at the DC?

I am not saying this will give you an exact answer but  you'd have something more than a core file not saying much, I have help  many customers without asking  higher support to decrypt the core file itself.

note:

-There are many logs for many different reasons the syslog I mention on this post is the 'general' log to say it like that... for more detailed logs for specific processes go to WAE#dir errorlog

-Don't forget to  vist the WAAS troubleshooting page

http://docwiki.cisco.com/wiki/Cisco_WAAS_Troubleshooting_Guide_for_Release_4.1.3_and_Later_--_Troubleshooting_Optimization

-Don't start playing with WAAS on production devices!!!

I know this post is from months ago but I hope it can help other people too, cheers!

Felix

Hi,Felix

Thanks for clearing the air ,have one question on  how can I download the core damp file in my local directory using CLI Command or GUI just like "sysreport", using FTP Server and or tftp server.

Hi,

cd crash   (or cd core_dir)

copy disk ftp <ftp-srv IP> <remote dir> <remote filename> <local filename>

...it will prompt you for credentials for the FTP server !

you can use the "." (period) as <remote dir> for the current (working) directory of the ftp server

Best regards

Finn

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: