UC560 CUE disk issue

Unanswered Question
Feb 24th, 2012

All-

So we have a client with a UC560 that is reporting that it is running out of disk space:

"Feb 24 18:06:14 localhost platform_config: Critical amount of disk space used, please backup log files using the copy log command"

Below is the result of a sh sysdb /sw/info/filesys:

dev/root/usage/mountPoint               /

dev/root/usage/oneKblocks               2048124

dev/root/usage/usedBlocks               1960264

dev/root/usage/availBlocks              87860

dev/root/usage/usedPercent              96

I took the system offline and attempted a "database compact" and, after a minute or two, received the following:

INFO:  vacuuming "pg_catalog.pg_largeobject"

ERROR:  invalid page header in block 22545 of relation "pg_largeobject"

We just recently deleted and recreated all of the voice mail boxes, so we're not using space there.  The log files are all just a few hundred K, so nothing noteworthy there, either.

Without a decent disk interface, it is hard to tell WHAT is gobbling the space...?   Also, what is this invalid page header issue?

Any assistance would be appreciated.

Thanks,

Rick

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 3 (2 ratings)
econsystems Sun, 02/26/2012 - 11:40

Rick, you MUST call support and open a trouble ticket.  I have gone through this issue 3 times.  1 time the whole unit had to be RMA'd.  1 time, the client lost everything and I had to rebuild the whole UC from scratch.  The 3rd time, i raised enough hell and it got escalated to the moon.

For almost a year, i have an engineer from cisco, california assigned specifically to my case who installed special scripts and the cue is running diagnistic logging 24/7 onto an FTP and syslog server.  he checks on the cue status every few weeks.

At this state, if you are lucky, you MAY be able to save the cue configuration.

Call support right away and tell them this issue has been tracked, specifically for uc560 for a couple of years.

Below is my email address.  If they give you a run around, email me directly and I can give you my case number so that they can get an idea what is going on.

Chaudhry Manzoor

cm@econsystems.com

David Trad Sun, 02/26/2012 - 16:11

Hi Rick,

There is a good chance that the Compact Flash card has been hit with some bad blocks, this is not uncommon as CFC's don't have a very high means failure rate, and on busy systems with plenty of CUE action they can experience failures prematurely.

I would log a support case imediatly on this machine, in the mean time I would use CCA to create a full backup of the system just incase the CUE fails completly.

Please do not leave anything to chance, also dont run a check disk on it as this can kill it (Has happened to me more then 3 times) especially if the CFC is on its last legs.

Cheers,

David.

Brandon Turpin Mon, 02/27/2012 - 15:53

HI Rick,

As has been mentioned before, please open a case with SBSC and reference CSCtr07804.

Thanks,

Brandon

kevinsoliz Fri, 05/18/2012 - 05:04

Anyone have an update from Cisco regarding this issue/bug?  I have a UC system and a case open with this exact issue and they are telling me to reload the module to factory defaults and start from sratch which IS NOT AN OPTION at all.

Myself and the customer are pretty livid and will be pushing and escalating this case ALL the way up until someone can address this issue within Cisco.

Bwalters613 Tue, 06/26/2012 - 13:45

Am going through this right now.  Had no idea this 'bug' even existed until the CUE was full.  Have an escalation in place now, but really, wipe out the CUE and start over is the answer?  I cannot believe there is not method to moev a file out of the system to free up space, or delete a mailbox or two to do so and get it back online to upgrade and allegedly fix it.

Has anyone seen 8.6 in action to actually *resolve* this, or does the issue still exist beyond that?

Paolo Bevilacqua Wed, 06/27/2012 - 03:54

         

Has anyone seen 8.6 in action to actually *resolve* this, or does the issue still exist beyond that?

The bug mentioned above still shows as 'open', meaning is not fixed in any version.

Cisco should provide additional information on that.

Bwalters613 Wed, 06/27/2012 - 05:55

Per Cisco - 8.6 helps alleviate the issue, but does not resolve it, as you mention.  They are hopeful that the version they are currently working on will actually resolve it.  I am not really sure how you can code a database to grow until *all* disk space is used - and not provide a method to watch/verify it automatically (and not a manual investigation), but at least when  you get to the escalated support, Cisco is there to work hard to resolve. Brandon, if you are out there watching these threads - thank you for all your hard work and attempts to fix this!

If your database gets to 100%, as mine did, you are unlikely to be able to recover anything, as we were unable to.  Unfortunately, another symptom that the database is full and/or corrupted is that automated backups of CUE fail - and if no one is watching them...restores are ugly in terms of recovered data (though the CUE reinstall seemed pretty simple and straightforward).

Just FYI for anyone else out there. Set up automated CUE backups (and have it email you if you can get that to work) and make sure *someone* is watching that, and disk usage.  Otherwise,  you end up here.

      

Update - scheduled backups in this instance *are* sending emails to the local admin at my client's site - and oddly enough, they are all saying, every day, that backups were successful and completed, so this may not be a harbginer of failure.  The darn thing said it was backing up fine and hadn't for months...ugh.

Message was edited by: Brett Walters

Thomas Gruendler Tue, 08/21/2012 - 11:31

Has anybody heard anything new about this bug?  I spent 11 hours last Tueday night with a UC560 getting it back up and running with the newest software pack and the support tech sent me this:

Hello Thomas,

Right now it appears as though the database is not corrupt, but only the engineer build will fix the issue. If you are happy with the way it is now, then make sure a regularly scheduled backup is done, but if you would like to move forward with the engineer build, just give us a call. The information you sent me is now attached to the case.

I am just wanting to know if this bug creeps up on you as in the database fills up at a slow rate or if it just one day goes from 21% to 100%.  This unit that I had has been in production since November 2010 and Monday night was when the database showed this bug.  Right now, I am sitting at 21% usage and on the latest software pack 8.6.0 and everything seems to be running smoothly. 

Also, does anybody have any kind of scripts that I can run to get the information on the database used percentage automatically?  Having to log in everyday and run the command is not ideal.

Bwalters613 Tue, 08/21/2012 - 11:39

Thomas - talk to higher level support within the Small Business Support group.  There is a temporary fix (new CUE installation version) that will alleviate some or most of the growth causing this, but you will need their assistance in obtaining it and installing it, even if you are a CUE expert and CLI guru.  I am told there will be a new version released in the near future (no promised dates, but hopefully September) that includes this version.

In my experience with the issue, it is gradual, but it will depend heavily on voice mail usage on the system.  If there is a ton of voice messages left, then deleted, this will grow faster than if VM is only nominally used.

I do not have any scripts - I logged in manually, and understand your frustration with it.  Same ones I have with not being able to automatically backup CCA (yes, I know I can backup CUE regularly, and do).

Brett

      

PS: 8.6.0 does not fix it - the engineer build is better suited to alleviate more of the issue.  Been there, done that.

Thomas Gruendler Tue, 08/21/2012 - 12:00

Thanks for the info.  I have contacted support to get the engineering build installed.  Do you know if you can restore a backup from the 8.6.0 CUE to the engineering build?

Bwalters613 Tue, 08/21/2012 - 12:02

When we did my latest one, with some work, we were able to - but your engineer will explain it.  I don't want to make anything public they may not wany public - but yes, you will be able to backup your config, upgrade, and restore.  At least we were in my case - I was not on 8.6.0 for mine.

victorycf Fri, 01/18/2013 - 10:43

I just got hit with this bug and got the bad news from Cisco support that I have to upgrade and lose all of my voice mails (I'm on 8.2 now)

I'm amazed that there wasn't some sort of communication sent out warning people to upgrade to fix a critical bug.  Or is there such a list and I'm not on it for some reason?

Well, about to call my boss and tell him the bad news.

Bwalters613 Fri, 01/18/2013 - 11:24

Are you at 100% utilization of voice mail and have no backups of the current VM?  Escalate the case with Cisco - there are sometimes exceptions to help rectify without losing it all.  But if you are at 100%, I think you are out of luck unless you have a backup of it.

victorycf Fri, 01/18/2013 - 11:44

No, voice mail is only at around 30% usage.   It's basically CSCtr07804 that allegedly is fixed in 8.6.1

se-10-1-10-1# show sysdb /sw/info/filesys

dev/root/usage/mountPoint               /

dev/root/usage/oneKblocks               2048124

dev/root/usage/usedBlocks               2048124

dev/root/usage/availBlocks              0

dev/root/usage/usedPercent              100

Backups won't complete. I can back up the config, but not data....

se-10-1-10-1(offline)# backup category configuration

Backup progress: 6930782 bytes

Backup Complete.

Check Backup history for detailed information.

se-10-1-10-1(offline)# backup category data

Backup progress: 1979 bytes

Backup Failed.

Check Backup history for detailed information.

se-10-1-10-1(offline)#

Cisco support said I had to upgrade to 8.6.1 and warned me since I have 8.2 I will lose voice mail... Will tackle the upgrade shortly, but I'm worried that when the backup fails, the upgrade process won't continue.   Hopefully I can bypass it.   I've already told my boss voice mails are almost certainly going to be lost. 

Bwalters613 Fri, 01/18/2013 - 11:48

Ack - it is 100%.  Pretty sure they are right, then...without any previous backup, all your voice mail is history.   You must have an environment that goes through a bunch of voice mails (and properly deletes them) like mine did.  I have other UC560's with minimal issue in this area because their VM usage is minimal.  We upgrade them anyway so not to lose track of them, but they did not melt down.

victorycf Sat, 01/19/2013 - 10:42

I resolved this fairly painlessly (well, except for losing all voicemails).   My main concern was not being able to do a backup during the upgrade procedure so that procedure would restore the system after it was done.  I fixed it doing the following.

The first step was to go into the CUE web management page and delete a few AA prompt messages that I didn't really need. This freed up enough space on the CUE flash so it started working properly temporarily.

I then exectued the following in offline mode

  • backup category configuration  (I was unable to backup data without it failing)
  • restore factory default

After it rebooted I then went into offline mode and did:


  • backup server url ..... (point to my original backup ftp site)
  • show backup server (to get the latest id of the backup I did above)
  • restore id n category configuration (where n is the backup id)
  • reload

This dropped my flash usage down to about 30% and got the system working again.  At this point  I could have waited to upgrade to a later time, but went ahead and did the upgrade now that it could complete a backup.

The upgrade went fairly painlessly. I did have a problem with pilot numbers for the CUE not working (vm, AAs) but doing a reload on the CUE fixed that.    I also lost my remote phones. Hopefully I don't have to drag them back in to reconfigure. I'll worry about that tomorrow.

Hopefully this helps someone who finds this article the same way I did, by doing a search on the symptons.

Alexander Maroukian Sat, 01/19/2013 - 11:52

Hello Ken,

Thanks for sharing! Just to add if the remote phones are SPA525 then they are using SSL VPN.

After upgrade you need to recreate the certificate for the SSL. You may try to do so in CCA Configure> Security> SSL VPN-> Generate Certificate.

HTH,

Alex

victorycf Sun, 01/20/2013 - 09:12

Thank you. That fixed that issue, as well as an issue of CCA not connecting over SSL locally.

brapier@dmhs.ca Mon, 01/21/2013 - 11:46

I got bit by this a month ago.  Was not aware of any bug reports out on the issue.  We lost the entire voicemail/auto attendant setup.

It started the night before with a call that someone "could not use the voicemail menus" while listening to the messages.

Turned out that the message they were listening to was actually recording the the menus themselves (you know the press 1 to, press 2 to, press 3 to etc.) , so when they tried to push a button nothing happened as it wasn't really the menu, but a recording of the menu.  I chalked it up to user error and thought nothing of it.  Well 16 hours later the whole thing died.

Started with a user telling me they could not get into thier voicemail, once I looked at the logs I started seeing the disk space errors.  I also tried the disk cleanup utility to end up with a corrupt db.

Of course the first thing we did with TAC was reload and update the software from 8.6 to 8.6.5 and then realized none of my backups would work.  8.5 hours later the system was rebuilt and running.

So far things have been stable since we moved to 8.6.5.

Actions

Login or Register to take actions

This Discussion

Posted February 24, 2012 at 3:18 PM
Stats:
Replies:20 Avg. Rating:3
Views:3239 Votes:0
Shares:0
Tags: No tags.

Discussions Leaderboard