LMS 3.1 Backup failed Insufficient disk space

Unanswered Question
May 10th, 2010

Hi, I've got LMS 3.1 running over Solaris 9. I created a folder in /free to store the database backups. Usually backups are about 2 to 2.5 GB but during the last backup I received an insufficient disk space warning:

Backup failed on 2010/5/9 at 4:2:24. REASON: Insufficient disk space on

backup destination volume. Available Space is 10259260Kb and required space

for backup is 19879403Kb.

No mayor changes have been  made to CiscoWorks lately so, is it normal that all of a sudden a db backup requires almost 10 times its usual size? Your recommendation would only be "assign your backup location to a bigger partition" or am I dealing with a disk space consuming backup issue?

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Nael Mohammad Mon, 05/10/2010 - 10:06

The backup does require 18.9 GB's of space and you only have 9.2GB available.  Check your /var/adm/CSCOp/log, /opt/CSCOpx/database, and /var/adm/CSCOpx/files  diretory and its subdirectories file size.

1. Post the dbbackup.log from /var/adm/CSCOpx/log directory, this should give us an idea what's going on .

2. Post the "df -h" output .

jarosado Tue, 05/11/2010 - 08:50

Folders' utilization:

/var/adm/CSCOpx/log --> 5.2 gb

/opt/CSCOpx/database --> 4.1 gb

/var/adm/CSCOpx/files --> 4.7 gb

bash-2.05$ df -hk

Filesystem             size   used  avail capacity  Mounted on

/dev/md/dsk/d0          15G   9.8G   4.8G    67%    /

/proc                    0K     0K     0K     0%    /proc

mnttab                   0K     0K     0K     0%    /etc/mnttab

fd                       0K     0K     0K     0%    /dev/fd

/dev/md/dsk/d3          20G   8.1G    11G    42%    /var

swap                    11G    32K    11G     1%    /var/run

swap                    11G    62M    11G     1%    /tmp

/dev/md/dsk/d5          69G    52G    16G    77%    /opt

/dev/md/dsk/d4          15G   4.8G   9.8G    34%    /free

Nael Mohammad Tue, 05/11/2010 - 10:25

LMS itself needs working space for the backup to perform its task and currently from reading the data you provided /free partition has 9.8GB available. According to the logs you provided, you need to have at least 18GB of free space to complete this task on /free/CSCO_DB_Backup .

Backup to '/free/CSCO_DB_Backup' started at: [Sun May  9 04:00:02 2010]

avail=10259260
[Sun May  9 04:02:19 2010]  ERROR(322): Insufficient disk space on backup destination volume.
Available Space is 10259260Kb and required space for backup is 19879403Kb.
[Sun May  9 04:02:19 2010]  Backup failed: 2010/05/09 04:02:19
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[Sun May  9 04:02:19 2010] Removing directory /free/CSCO_DB_Backup/7

As to why the backup is greater than before really depends on a number of factors such :

  • Purge Policy
    • Syslog Purge
      • Verify your purge policy under RME --> Admin --> Syslog --> Set Purge Policy and also do a "Force Purge" for syslogs .  The number of days is defaults to 7 days for purging.

    • Archive Purge
      • Go to RME --> Admin --> Config Mgmt Archive Mgmt --> Purge Settings and ensure its enabled and define the value according to your needs and data retention policy.
    • Job Purge
      • Go to RME --> Admin --> System Preferences --> Job Purge --> ensure that schedules are enabled based on your data retention policy. 
    • Change Audit
      • Go to RME --> Admin --> System Preferences --> Change Audit --> Set Purge Policy

The above is just an example for the RME application purging policy. Each application has its own purging policy such Campus Manager, DFM, HUM, IPM, and so on. You will need to identify what your data retention policy is and set the purging accordingly.

jarosado Thu, 05/13/2010 - 11:00

I did as suggested:


- I changed the "Purge records older than" value from 30 to 7 days at RME > Admin > Syslog > Set Purge Policy. Then I forced a purge at RME > Admin > Syslog > Force Purge. The size limit for the syslog backup file is 512MB.


- In RME > Admin > Config Mgmt > Archive Mgmt > Purge Settings I changed the value "Maximum versions to retain" from 5 to 3.


- In RME > Admin > System Preferences > Job Purge I scheduled a purge for all applications every week which was run yesterday at 23:00.

- I changed the value of "Purge change audit records older than" and "Purge audit trail records older than" from 180 to 30 days at RME > Admin > ChangeAudit > Set Purge Policy, and forced a purge.

- In Campus Manager > Administration > User Tracking at Administration > Reports > User Tracking Purge Policy I changed the values "Purge Archives Older than" and "Purge Jobs Older than" from 180 to 30 days.

This serveer doesn't have HUM installed and IPM is not configured at all.

After doing these changes I tried to backup the database again and now it's requsting 24.21GB!

Am I missing something?

Nael Mohammad Thu, 05/13/2010 - 19:32

What syntax are you doing to run the backup ? Where have you defined the backup directory to be?

Ensure you are using this format:

Usage: /bin/perl  /bin/backup.pl  [LogFile] [Num_Generations]

The backup.pl script uses "freespace/1024" to determine the free space available and requirements for the task to run. If that is not available, you need to clear up space not just from CiscoWorks but the destination partition where the backup directory resides.  Start by clearing out logs and syslogs, tmp directory, home directories, and so on.

If you want and this is "NOT SUPPORTED", you can hack the /opt/CSCOpx/bin/backup.pl but make a backup copy of the file. Change the value for "$freeSpace = $freeSpace/1024;" and "$totalSize = $totalSize/1024; to a lower number. For example, instead of 1024, you can choose "512" for both variables freeSpace and totalSize. This should at least get the backup started but  you might run into space issues that will cause it to fail. This simply lowers the threshold for space requirements to start the backup.pl script.

jarosado Tue, 05/18/2010 - 10:54

I've scheduled the backup at CS > Server > Admin > Backup. It runs at 4:00AM every Sunday and the backup folder is /free/CSCO_DB_Backup, the number of generations is 3. Here is a screenshot. It's been this way for over a month.

Since I first created this folder to store the backups, I made as much space as possible in this partition but now, due to the massive size of backups, none of the actual patitions on this server seems to be quite enough.

Attachment: 

Actions

This Discussion