ISE 2.1.0.474 backup issue hangs at 20%

[rikerrobert]

I am backing up ISE via ise-config backup of the PAN. I tried doing the backup from the web and then moved to the CLI to get more info. I attempted an operational backup and that worked without issue to an FTP repository. I re-attempted the configuration backup with the same repository and it failed again. No logs to really speak that I could find. Being an R&S guy, I checked debugs for backup-restore backup and was presented with the below when I kicked off the config backup again.

wilisep1/admin# backup BACKUP repository UPGRADE ise-config encryption-key plain P@ssw0rd
6 [15324]:[info] backup-restore:backup: br_history.c[521] [admin]: ISE backup/restore initiated by CLI as ise.br.status is not 'in-progress' in /tmp/ise-cfg-br-flags
7 [15324]:[debug] backup-restore:backup: br_backup.c[585] [admin]: initiating backup BACKUP to repos UPGRADE
7 [15324]:[debug] backup-restore:backup: br_backup.c[629] [admin]: no staging url defined, using local space
7 [15324]:[debug] backup-restore:backup: br_backup.c[54] [admin]: flushing the staging area
7 [15324]:[debug] backup-restore:backup: br_backup.c[658] [admin]: creating /opt/backup/backup-BACKUP-1518626716
7 [15324]:[debug] backup-restore:backup: br_backup.c[662] [admin]: creating /opt/backup/backup-BACKUP-1518626716/backup/cars
% Internal CA Store is not included in this backup. It is recommended to export it using "application configure ise" CLI command
% Creating backup with timestamped filename: BACKUP-CFG10-180214-1045.tar.gpg
% backup in progress: Starting Backup...10% completed
7 [15324]:[debug] backup-restore:backup: br_backup.c[726] [admin]: creating /opt/backup/backup-BACKUP-1518626716/backup/ise
7 [15324]:[debug] backup-restore:backup: br_backup.c[752] [admin]: calling script /opt/CSCOcpm/bin/isecfgbackup.sh
% backup in progress: Validating ISE Node Role...15% completed
%Error: Aborting ISE configuration backup as it is not supported on this node type.
sed: couldn't flush /tmp/sed5CRQRo: No space left on device
% Application backup error

The issue I am running into is the last few lines. This is an ISE appliance and not a VM. Any insight is helpful.

Thanks,

Rob

[marce1000]

 - Check available disk space on your ISE server using : show disks

M.

[rikerrobert]

wilisep1/admin# show disks


Internal filesystems:
/ : 43% used ( 6009820 of 14987616)
/dev : 0% used ( 0 of 8110824)
/dev/shm : 0% used ( 0 of 8121068)
/run : 10% used ( 795620 of 8121068)
/sys/fs/cgroup : 0% used ( 0 of 8121068)
/storedconfig : 2% used ( 1585 of 95054)
/boot : 23% used ( 104933 of 487634)
/tmp : 100% used ( 1966668 of 1983056)
/opt : 15% used ( 75886760 of 549840808)
/run/user/440 : 0% used ( 0 of 1624216)
/run/user/301 : 0% used ( 0 of 1624216)
/run/user/0 : 0% used ( 0 of 1624216)
/run/user/303 : 0% used ( 0 of 1624216)
/run/user/321 : 0% used ( 0 of 1624216)
warning - /tmp is 100% used (1966668 of 1983056)

As you can see the /tmp file is full. I haven't found a way to clear it.

[marce1000]

 - Yeah, I always find  it poor from Cisco, to use linux (open source) and not give customers the benefit of it (root access and management would solve this in 2-secs).  It's agains open-source philosophy! Anyway following the link below ; there is a Rootpatch for ISE, unfortunately you will have to get it from TAC :

 https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvc95735/?referring_site=bugquickviewredir

M.

[ibrahim.sadiq]

Does this bug applicable only for the physical appliances (as it is mentioned for Cisco ISE 3300) or even for the Virtual Appliances? 

Cos, we have a similar issue in a Virtual Appliance.

[Jatin Katyal]

Yes, it is applicable for ISE running on VM's too.

"Show disks" on the ISE CLI will tell you more how the disk looks like.

Look for this line in the output.

/tmp : 100% used

Once you confirm that, open a TAC case to get this cleared.

[ibrahim.sadiq]

Looks we have sufficient free space available

ise/admin# show disks

Internal filesystems:
/ : 18% used ( 2509964 of 14987616)
/dev : 0% used ( 0 of 8124524)
/dev/shm : 0% used ( 0 of 8134196)
/run : 1% used ( 25384 of 8134196)
/sys/fs/cgroup : 0% used ( 0 of 8134196)
/boot : 24% used ( 107955 of 487634)
/storedconfig : 2% used ( 1582 of 95054)
/tmp : 1% used ( 6500 of 1983056)
/opt : 23% used ( 129154740 of 593333176)
/run/user/440 : 0% used ( 0 of 1626840)
/run/user/301 : 0% used ( 0 of 1626840)
/run/user/0 : 0% used ( 0 of 1626840)
/run/user/304 : 0% used ( 0 of 1626840)
/run/user/303 : 0% used ( 0 of 1626840)
/run/user/322 : 0% used ( 0 of 1626840)
/run/user/321 : 0% used ( 0 of 1626840)
all internal filesystems have sufficient free space

We get the following error in the backup status;

ise/admin# show backup status
%% Configuration backup status
%% ----------------------------
% backup name: ISE_Config_Backup
% repository: SFTP_Repository
% start date: Wed Feb 21 10:47:18 CET 2018
% scheduled: no
% triggered from: Admin web UI
% host: XYZ.ISE-INTERNAL.NET
% status: DB BACKUP FAILED : . Backup aborted

[Jatin Katyal]

then your problem is different...what code ( including patch) you have running on ISE.

Check the condition of the defect and see if the symptoms of your problem matching it.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvc79739/?reffering_site=dumpcr