Hello I received a call today from our Data Centre team reporting 100% cpu on a Cisco ACS VM running 188.8.131.52 (this ACS VM is configured as a primary but not as a log collector). They also had a report stating that the /var folder on this VM was out of diskspace (1MB free)
From the cli on the VM I ran the show disks command and found the available disk space looked ok:
Internal filesystems: .. /var : 80% used ( 7783116 of 10315944) .. all internal filesystems have sufficient free space
On restarting the VM the output showed:
Internal filesystems: .. /var : 5% used ( 442856 of 10315944) .. all internal filesystems have sufficient free space
If this incident reoccurs, what commands can I use to view the logs (and sizes) in /var and are they included in the ACS logrotate schedule?
The high cpu issue on the primary started when some LUNs were being decommisioned - when this work was carried out VCenter lost contact with some ESX hosts (the ACS log collector was on one of these hosts). The VMs on these hosts were all up and running fine and could see their storage ok but to resolve the issue the VMs and ESX hosts were restarted ( the log collector was properly shutdown with the halt command)
When the log collector powered on it started to do 100% cpu cycling, the logs in Logging and Monitoring had disappeared and the daily incremental log backups stopped. The primary then started to show the problems in my initial post.
Moving log collection to another ACS VM didn't resolve the issue - Still no logging and 100% cpu cycling. To resolve this I turned off logging on the primary (I used "No log collector" rather than suspend). This stopped the 100% cpu on the log collector. when I reenabled log collection, the log collector started to display logs ok (nothing historical - only logs from when logging was enabled) and the 100% cpu cycling stopped.
I restored the logs on the log collector from backup with any gaps being covered by syslog.
So all is back to normal. I had a look at the show tech output and I can see the size of the files on /var. For future reference I assume that I can't delete any of these?
We have configured the outside and inside Interface with official ipv6 adresses, set a default route on outside Interface to our router, we also have definied a rule , which also gets hits, to permit tcp from inside Interface to any6.
In Syslog I also se...