01-29-2014 06:17 AM - edited 03-10-2019 09:20 PM
Hello
I received a call today from our Data Centre team reporting 100% cpu on a Cisco ACS VM running 5.5.0.46 (this ACS VM is configured as a primary but not as a log collector). They also had a report stating that the /var folder on this VM was out of diskspace (1MB free)
From the cli on the VM I ran the show disks command and found the available disk space looked ok:
Internal filesystems:
..
/var : 80% used ( 7783116 of 10315944)
..
all internal filesystems have sufficient free space
On restarting the VM the output showed:
Internal filesystems:
..
/var : 5% used ( 442856 of 10315944)
..
all internal filesystems have sufficient free space
If this incident reoccurs, what commands can I use to view the logs (and sizes) in /var and are they included in the ACS logrotate schedule?
Thanks
Andy
Solved! Go to Solution.
01-30-2014 05:25 PM
Andy,
You cannot delete them, but ACS should prune them over time as it manages disk space.
Javier Henderson
Cisco Systems
01-29-2014 03:34 PM
Andy,
Look at the output of "show tech".
Javier Henderson
Cisco Systems
01-30-2014 10:52 AM
Thanks for the reply Javier
The high cpu issue on the primary started when some LUNs were being decommisioned - when this work was carried out VCenter lost contact with some ESX hosts (the ACS log collector was on one of these hosts). The VMs on these hosts were all up and running fine and could see their storage ok but to resolve the issue the VMs and ESX hosts were restarted ( the log collector was properly shutdown with the halt command)
When the log collector powered on it started to do 100% cpu cycling, the logs in Logging and Monitoring had disappeared and the daily incremental log backups stopped. The primary then started to show the problems in my initial post.
Moving log collection to another ACS VM didn't resolve the issue - Still no logging and 100% cpu cycling. To resolve this I turned off logging on the primary (I used "No log collector" rather than suspend). This stopped the 100% cpu on the log collector. when I reenabled log collection, the log collector started to display logs ok (nothing historical - only logs from when logging was enabled) and the 100% cpu cycling stopped.
I restored the logs on the log collector from backup with any gaps being covered by syslog.
So all is back to normal. I had a look at the show tech output and I can see the size of the files on /var. For future reference I assume that I can't delete any of these?
Thanks
Andy
01-30-2014 05:25 PM
Andy,
You cannot delete them, but ACS should prune them over time as it manages disk space.
Javier Henderson
Cisco Systems
01-30-2014 10:13 PM
Thanks Javier
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: