IMPORTANT: Linux Leap second bug -> high load on VCS => please reboot!

Unanswered Question
Jul 1st, 2012

Hello!

The Cisco VCS seems to be affected by the (generic) Linux leap second bug!

We noticed that we had some (not all) VCSs with a load of >100.

I would recomend as a quick fix to immidiate reboot you VCSs!

Also check your other linux systems. Video and non video related. I looked at some TC/TE (E20, C20, ...) video codecs

but I did not notice any to be affected, but I would not be surprised if there are some.

See also (this is generic linux info, I would still recommend just to reboot the VCS!)

http://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/

http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second

http://www.h-online.com/open/news/item/Leap-second-Linux-can-freeze-1629805.html

Update:

The cisco bug id is CSCua78996 (though its currently not public as it looks, at least I do not find it)

some Java related info:

http://blog.wpkg.org/2012/07/01/java-leap-second-bug-30-june-1-july-2012-fix/

http://destefano.wordpress.com/2012/07/02/java-linux-leap-second-bug/

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 0 (0 ratings)
Martin Koch Sun, 07/01/2012 - 17:23

Btw, you will see a log entry in your linux kernel messages:

kernel:Jul  1 01:59:59 nd-ve-vcse-01 kernel: Clock: inserting leap second 23:59:60 UTC

To be sure I would still recommend the reboot but I was also sucessfull bringing the load down with on VCSs with X7.1, so maybe as an option where you can not immediately reboot it for various reasons:

uptime; /etc/init.d/ntpd stop; date -s "`date`"; ntpdate pool.ntp.org ; hwclock -w ; /etc/init.d/ntpd start

What it does:

uptime -  shows the load of the system (here we see a load of 29 which is to high as well)

 02:02:17 up 124 days, 13:51,  1 user,  load average: 29.36, 29.26, 29.72

  if you would do a top, you would see that the java process is what is eating up resources:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                           

12502 root      20   0 1162m 334m 9816 S  131  8.5   1984:37 java                                                                                              

13077 root      20   0  794m 309m  10m S   35  7.8 524:33.67 java  

/etc/init.d/ntpd stop

stops the NTP deamon

date -s "`date`"

sets the time to the current value, thats the fix mentioned on various websites,the load shall normalize after that.

ntpdate pool.ntp.org

sets the time with the current ntp time, in general one date/ntpdate shall be fine

hwclock -w

sets the hardware clock (optional)

/etc/init.d/ntpd start

start up the NTP server again.

After that you shall see that the load will go down and the java process is behaving ok again:

(all values will get lower (they show the averages over 1, 5, 15 minutes), 5.84 is the 1 min average

and it will most likely get lower on your system (0/1) but thats dependent on how much it is used)

uptime

02:03:55 up 124 days, 13:53,  1 user,  load average: 5.84, 21.05, 26.72

tomtanig Sun, 07/01/2012 - 17:53

Please note that this issue seen on VCS that run with Legacy TMS Agent due to application run on VCS for provisioning service.

VCS already upgraded to TMS Provisioning Extension should not experience this issue.

Also possibly see on VCS run OCS relay service.      

justinferello Mon, 07/02/2012 - 05:52

Fix 1:

Reboot the VCS and should be fine

Fix 2:

If you cannot reboot the VCS, just run this command from the 'root' login:

date -s  “`date -u`”

Thanks,

Justin

Actions

Login or Register to take actions

This Discussion

Posted July 1, 2012 at 4:16 PM
Stats:
Replies:3 Avg. Rating:
Views:3516 Votes:0
Shares:0

Related Content

Discussions Leaderboard