This article explains potential cause of the SMC periodically showing high CPU usage during bulkstats collection
We often get calls for a condition of high cpu usage by the SMC/SPC. Quite often, the reason for the spike is due to bulkstats gathering. When bulkstats are gathered, a massive number of counters need to be queried and (s)ftp'd offline. The interval with which bulkstats are gathered is determined by the bulkstats mode variable "sample-interval", with the value in minutes. An example follows from a portion of the output of the "show configuration" command:
bulkstats collection
bulkstats historical collection
bulkstats mode
sample-interval 2
transfer-interval 15
If you are experiencing this bulkstat cpu behavior, please realize that this is an as designed behavior, and should not affect your subscriber traffic. In the example that follows, note that slot 8/0's cpu usage goes up just into the sample-interval timeframe, and goes down sometime thereafter. To determine whether your customers issue matches this behavior, please do the following:
- ssh to the mobile gateway in question.
- show configuration | grep sample
- timestamps
- show cpu table | grep 8/0
You will see that your cpu usage on slot 8/0 goes up for a small period of time every sample-interval.
In this annotated example, pertinent timestamps are noted, but clearly show the relationship of cpu usage to bulkstat sample-interval time.
Example:
[local]PDSN# show configuration | grep sample
sample-interval 2
[local]PDSN# timestamps
[local]PDSN# sho cpu table | grep 8/0
--------Load-------- ------CPU-Usage----- ---------Memory--------
cpu state now 5min 15min now 5min 15min now 5min 15min total
---- ----- ------ ------ ------ ------ ------ ------ ----- ----- ----- -----
Friday December 04 20:52:02 UTC 2009
8/0 Actve 0.58 0.56 0.50 10.5% 19.5% 19.5% 413M 415M 411M 1024M
Friday December 04 20:52:10 UTC 2009
8/0 Actve 0.57 0.56 0.50 32.8% 19.6% 19.6% 418M 415M 411M 1024M
Friday December 04 20:52:15 UTC 2009
8/0 Actve 0.64 0.57 0.50 60.4% 21.1% 20.1% 417M 415M 411M 1024M
Friday December 04 20:52:27 UTC 2009
8/0 Actve 0.54 0.55 0.50 24.6% 21.4% 20.3% 416M 415M 411M 1024M
Friday December 04 20:52:37 UTC 2009
8/0 Actve 0.53 0.55 0.49 11.4% 21.4% 20.4% 416M 415M 411M 1024M
Friday December 04 20:52:49 UTC 2009
8/0 Actve 0.52 0.55 0.49 12.0% 21.6% 20.4% 414M 415M 411M 1024M
Friday December 04 20:53:02 UTC 2009
8/0 Actve 0.44 0.53 0.49 11.2% 21.7% 20.4% 414M 415M 411M 1024M
Friday December 04 20:53:06 UTC 2009
8/0 Actve 0.53 0.55 0.49 30.8% 21.8% 20.5% 414M 415M 411M 1024M
Friday December 04 20:53:12 UTC 2009
8/0 Actve 0.53 0.55 0.49 30.8% 21.8% 20.5% 414M 415M 411M 1024M
Friday December 04 20:53:16 UTC 2009
8/0 Actve 0.45 0.53 0.49 17.3% 19.7% 20.0% 414M 415M 411M 1024M
Friday December 04 20:53:21 UTC 2009
8/0 Actve 0.45 0.53 0.49 17.3% 19.7% 20.0% 414M 415M 411M 1024M
Friday December 04 20:53:25 UTC 2009
8/0 Actve 0.38 0.51 0.48 13.9% 19.4% 20.0% 413M 415M 411M 1024M
Friday December 04 20:53:31 UTC 2009
8/0 Actve 0.38 0.51 0.48 13.9% 19.4% 20.0% 413M 415M 411M 1024M
Friday December 04 20:53:40 UTC 2009
8/0 Actve 0.32 0.49 0.48 14.6% 19.5% 20.1% 413M 415M 411M 1024M
Friday December 04 20:53:44 UTC 2009
8/0 Actve 0.27 0.47 0.47 12.1% 19.7% 20.1% 413M 415M 412M 1024M
Friday December 04 20:53:51 UTC 2009
8/0 Actve 0.27 0.47 0.47 12.1% 19.7% 20.1% 413M 415M 412M 1024M
Friday December 04 20:53:57 UTC 2009
8/0 Actve 0.23 0.46 0.46 13.7% 19.8% 20.2% 413M 415M 412M 1024M
Friday December 04 20:54:01 UTC 2009
8/0 Actve 0.23 0.46 0.46 13.7% 19.8% 20.2% 413M 415M 412M 1024M
Friday December 04 20:54:05 UTC 2009
8/0 Actve 0.50 0.51 0.48 29.2% 20.0% 20.2% 415M 415M 412M 1024M
Friday December 04 20:54:08 UTC 2009
8/0 Actve 0.50 0.51 0.48 29.2% 20.0% 20.2% 415M 415M 412M 1024M
Friday December 04 20:54:10 UTC 2009
8/0 Actve 0.50 0.51 0.48 29.2% 20.0% 20.2% 415M 415M 412M 1024M
Friday December 04 20:54:13 UTC 2009
8/0 Actve 0.50 0.51 0.48 29.2% 20.0% 20.2% 415M 415M 412M 1024M
Friday December 04 20:54:16 UTC 2009
8/0 Actve 0.58 0.52 0.48 67.1% 21.6% 20.8% 421M 415M 412M 1024M
Friday December 04 20:54:19 UTC 2009
8/0 Actve 0.58 0.52 0.48 67.1% 21.6% 20.8% 421M 415M 412M 1024M
Friday December 04 20:54:22 UTC 2009
8/0 Actve 0.58 0.52 0.48 67.1% 21.6% 20.8% 421M 415M 412M 1024M
Friday December 04 20:54:26 UTC 2009
8/0 Actve 0.65 0.54 0.49 41.8% 22.3% 21.1% 423M 415M 412M 1024M
Friday December 04 20:54:30 UTC 2009
8/0 Actve 0.65 0.54 0.49 41.8% 22.3% 21.1% 423M 415M 412M 1024M
Friday December 04 20:54:34 UTC 2009
8/0 Actve 0.63 0.54 0.49 23.3% 22.6% 21.1% 415M 415M 412M 1024M
8/0 Actve 0.63 0.54 0.49 23.3% 22.6% 21.1% 415M 415M 412M 1024M
Friday December 04 20:54:57 UTC 2009
8/0 Actve 0.82 0.60 0.51 14.3% 23.3% 21.3% 414M 415M 412M 1024M
Friday December 04 20:55:03 UTC 2009
8/0 Actve 0.82 0.60 0.51 14.3% 23.3% 21.3% 414M 415M 412M 1024M
[local]PDSN#
Imported from Starent Networks Knowledgebase Article # 10880
