Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
3111
Views
0
Helpful
2
Replies

High CPU Switch Processor and DFC's

peterappleton
Level 1
Level 1

‎11-17-2011 04:15 PM - edited ‎03-07-2019 03:27 AM

Hi,

I am a little stuck trying to troubleshoot unusally high cpu on 7600 Platform.This has been going on for many hours and not been causing a problem, but this doesn't seem replicated in our network and I am concerned about the cause.

I have attached some output below, and disabled netflow on interfaces to free some switching CPU. Is there any article which might help me track the cause of this?

We only have routing protocols and 5 static routes to Null 0, so I cant see how this can be a route loop, but willing to stand corrected. Routes are learned from BGP with OSPF/MPLS internally on the network. This switch also connects to access switches on RIP and EIGRP.

No change has been made to this device when this problem started to occur so I am suspecting something about the traffic has changed or there is a rougue user?

Anybody who can shed any light on this would be appriciated.

Thank you.

#sh proc cpu sort | e %.*0.00%.*%

CPU utilization for five seconds: 7%/0%; one minute: 13%; five minutes: 13%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

   7    30965684     1634251      18947  4.31%  2.14%  1.60%   0 Check heaps

149         276         567        486  1.27%  0.30%  0.09%   1 SSH Process

327       14604    51638155          0  0.15%  0.16%  0.15%   0 TCP Timer

  12     4410044    55050720         80  0.15%  0.19%  0.23%   0 ARP Input

222     1865168    32156320         58  0.15%  0.13%  0.10%   0 IP Input

   2        1596      410554          3  0.07%  0.01%  0.00%   0 Load Meter

563      485480    19404642         25  0.07%  0.05%  0.06%   0 BGP I/O

185        1728    61347854          0  0.07%  0.01%  0.00%   0 ACE Tunnel Task

488     1069864    16208110         66  0.07%  0.04%  0.05%   0 Port manager per

257        2832   243982753          0  0.07%  0.07%  0.07%   0 Ethernet Msec Ti

221        1300    61347660          0  0.07%  0.01%  0.00%   0 IP ARP Retry Age

302      534172     3630729        147  0.07%  0.07%  0.07%   0 XDR mcast

577     3843468    21791383        176  0.07%  0.22%  0.21%   0 BGP Router

564     3754744     2777096       1352  0.07%  0.04%  0.03%   0 BGP Task

  29      465868     4358622        106  0.00%  0.02%  0.00%   0 IPC Seat Manager

159      411312       34966      11763  0.00%  0.02%  0.00%   0 Per-minute Jobs

160        8652     2054181          4  0.00%  0.01%  0.00%   0 Per-Second Jobs

212    10319444       62680     164636  0.00%  1.13%  1.70%   0 Collection proce

261        1500    61347915          0  0.00%  0.01%  0.00%   0 IPAM Manager

310      980568     1734948        565  0.00%  0.06%  0.07%   0 IP RIB Update

322        8692     2908044          2  0.00%  0.04%  0.05%   0 CEF: IPv4 proces

381      847188     1026384        825  0.00%  0.05%  0.05%   0 HIDDEN VLAN Proc

554     1327956    60254368         22  0.00%  0.05%  0.05%   0 IPv6 Input

569      256056     2063117        124  0.00%  0.01%  0.00%   0 SNMP ENGINE

575       17888     2268957          7  0.00%  0.01%  0.00%   0 OSPF-10 Router

582    44846720      253765     176728  0.00%  1.79%  2.39%   0 BGP Scanner

583    16955364       88650     191266  0.00%  6.32%  5.79%   0 BGP Event

Switch Processor:

#remote command switch sh proc cpu sort | e %.*0.00%.*%

CPU utilization for five seconds: 44%/0%; one minute: 25%; five minutes: 23%
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
191    29359288     1878756      15627 19.34%  9.13%  8.16%   0 CFIB LC Main Tas
425    16019868      195075      82122 19.19%  5.51%  4.51%   0 Collection proce
136   158738908   253656109        625  4.04%  7.54%  7.67%   0 slcp process
180     9484452      258279      36722  0.93%  0.50%  0.47%   0 TYFIB Table Mana
283      352592     2062402        170  0.23%  0.07%  0.06%   0 XDR LC Backgroun
366     4353856     2511130       1733  0.15%  0.25%  0.27%   0 Env Poll
406       79264       39578       2002  0.15%  0.02%  0.00%   0 CMFI
426       11736     7441657          1  0.07%  0.02%  0.00%   0 DiagCard5/-1
  67      129784    16775780          7  0.07%  0.02%  0.00%   0 SCP async: LCP#7
282       12108      979957         12  0.07% 49.57% 46.45%   0 XDR LC PRIO Crit
  52      170268    27083427          6  0.07%  0.04%  0.05%   0 SCP Download Lis
154       12268    13202457          0  0.07%  0.01%  0.00%   0 Heartbeat
428      403028    35359414         11  0.07%  0.07%  0.07%   0 DiagCard7/-1
  69       93908    16797971          5  0.00%  0.02%  0.00%   0 SCP async: LCP#8
   7     7544016      452185      16683  0.00%  0.28%  0.33%   0 Check heaps
139       41096     3017037         13  0.00%  0.04%  0.05%   0 CEF: IPv4 proces
177    27519724      922433      29833  0.00%  1.26%  1.33%   0 Vlan Statistics
178       36572     1368736         26  0.00%  0.01%  0.00%   0 L2 PI_E consiste
231       11212    60978322          0  0.00%  0.01%  0.00%   0 IP ARP Retry Age
235       13068    62474189          0  0.00%  0.01%  0.00%   0 IPAM Manager
296       80848      467109        173  0.00%  0.02%  0.01%   0 Hardware API bac
214      837640      389447       2150  0.00%  0.04%  0.05%   0 QoS SP Process
429      319396    32800011          9  0.00%  0.06%  0.07%   0 DiagCard8/-1
433      336056    33407136         10  0.00%  0.07%  0.07%   0 DiagCard9/-1
435       99904    16818849          5  0.00%  0.01%  0.00%   0 SCP async: LCP#9

Module 9

#remote command module 9 sh proc cpu sort | e %.*0.00%.*%

CPU utilization for five seconds: 64%/3%; one minute: 39%; five minutes: 29%
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
150    50122780     1715427      29218 31.83% 21.60% 15.38%   0 CFIB LC Main Tas
276    25015148      123715     202204 27.17% 11.11%  7.52%   0 Collection proce
222     2978100     1878830       1585  0.46%  0.12%  0.09%   0 XDR LC Backgroun
153    24225104   126189531        191  0.31%  0.44%  0.58%   0 fw_lcp process
262      114600       51249       2236  0.15%  0.04%  0.00%   0 CMFI
   7    16263448      903451      18001  0.00%  0.91%  0.75%   0 Check heaps
118      483964       34401      14068  0.00%  0.01%  0.00%   0 Per-minute Jobs
136    36859936      806601      45698  0.00%  1.61%  1.70%   0 Vlan Statistics
146    16604056      256543      64723  0.00%  0.69%  0.75%   0 TYFIB Table Mana
172     1668880      410402       4066  0.00%  0.05%  0.05%   0 QoS SP Process
228      949872      392500       2420  0.00%  0.04%  0.05%   0 Hardware API bac
267      657872     2764849        237  0.00%  0.01%  0.00%   0 CEF: IPv4 proces

Module 8

#remote command module 8 sh proc cpu sort | e %.*0.00%.*%

CPU utilization for five seconds: 47%/3%; one minute: 28%; five minutes: 27%
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
150    50511492     1720462      29359 22.63% 14.69% 14.43%   0 CFIB LC Main Tas
277    24961912      124426     200619 15.88%  7.45%  7.03%   0 Collection proce
136    36856280      807534      45641  1.74%  1.75%  1.74%   0 Vlan Statistics
   7    16423484      909629      18055  1.58%  0.88%  0.76%   0 Check heaps
222     3227248     1890048       1707  0.39%  0.12%  0.14%   0 XDR LC Backgroun
153    20103744   123558943        162  0.31%  0.24%  0.25%   0 fw_lcp process
262      119292       45833       2602  0.15%  0.04%  0.01%   0 CMFI
133      342732     1461957        234  0.15%  0.02%  0.00%   0 L2 Control Task
147     1065556     3701319        287  0.07%  0.01%  0.00%   0 Online Linecard
118      488432       34392      14201  0.00%  0.01%  0.00%   0 Per-minute Jobs
146    16698832      256647      65066  0.00%  0.70%  0.75%   0 TYFIB Table Mana
172     1780728      410369       4339  0.00%  0.06%  0.05%   0 QoS SP Process
173      355540       69152       5141  0.00%  0.01%  0.00%   0 TCAM Manager pro
228      914376      389772       2345  0.00%  0.03%  0.02%   0 Hardware API bac

Module 7

#remote command module 7 sh proc cpu sort | e %.*0.00%.*%

CPU utilization for five seconds: 59%/3%; one minute: 33%; five minutes: 29%
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
150    49730524     1705529      29158 29.18% 17.96% 15.36%   0 CFIB LC Main Tas
275    24977824      124173     201161 23.60%  9.32%  7.52%   0 Collection proce
136    37196096      805087      46202  1.73%  1.74%  1.74%   0 Vlan Statistics
222     2840452     1859748       1527  0.47%  0.20%  0.14%   0 XDR LC Backgroun
262      117196       51928       2256  0.15%  0.04%  0.00%   0 CMFI
153    25529680   104373143        244  0.15%  0.22%  0.22%   0 fw_lcp process
173      373564       69144       5402  0.07%  0.01%  0.00%   0 TCAM Manager pro
   7    16666812      919009      18135  0.00%  0.64%  0.67%   0 Check heaps
146    16798576      256376      65523  0.00%  0.72%  0.76%   0 TYFIB Table Mana
172     1676900      410386       4086  0.00%  0.05%  0.05%   0 QoS SP Process
228      768808      397244       1935  0.00%  0.02%  0.00%   0 Hardware API bac

Labels:
0 Helpful
2 Replies 2

Barun Singhania
Cisco Employee
Cisco Employee

‎11-23-2011 03:27 PM 

Hi Peter,

The two processes that are taking up CPU in all the outputs you have pasted are 
"CFIB LC Main Tas" and  "Collection proce".


Collection proce : 

This is the collection process which called the Background process bg_process() to 
get the context of a given process. This would happen whenever there is *something* 
to do. In this case there must be another process (possibly cfib_lc_process) that 
called the collection process.  

CFIB LC Main Task : 

This is cfib_lc_process which processes events from other local processes and icc 
messages from RP. In this regards, possibly there are FIB entries that are changing 
which is triggering these continuous updates. Check the output of 'show cef events' 
from both RP and SP, which may give us additional information on what CEF is doing.


Regards,
Barun
0 Helpful

peterappleton
Level 1
Level 1
In response to Barun Singhania

‎11-23-2011 03:55 PM

OK,

I have the outputs below but I see no entries since the last reboot

#sh cef events
% Command accepted but obsolete, unreleased or unsupported; see documentation.

Oct 25 05:49:39.315: SubSys  ipv4fib_ios_def_cap init
Oct 25 05:49:39.315: SubSys  ipv4fib_rp_def_cap init
Oct 25 05:49:39.327: SubSys  ipv6fib_ios_def_cap init
Oct 25 05:49:39.327: SubSys  ipv6fib_rp_def_cap init
Oct 25 05:50:14.515: FS_REG  IPRM(0 - ft 0xBBDDA50) - ok
Oct 25 05:50:14.515: FS_REG  NetFlow(1 - ft 0xBBDDA50) - ok
Oct 25 05:50:14.515: FS_REG  Broker(2 - ft 0xBBDDA50) - ok
Oct 25 05:50:14.515: FS_REG  MPLS LFD(3 - ft 0xBBDDA50) - ok
Oct 25 05:50:14.515: Inst    unknown -> RP
Oct 25 05:50:14.515: SubSys  fib_ios_chain init
Oct 25 05:50:14.715: FS_REG  NetFlow(1 - ft 0xBBF6AEC) - ok
Oct 25 05:50:14.715: FS_REG  Broker(2 - ft 0xBC15F8C) - ok
Oct 25 05:50:14.759: FS_REG  MPLS LFD(3 - ft 0xC7D75F0) - ok
Oct 25 05:50:14.759: FS_REG  IPRM(0 - ft 0xC7F101C) - ok
Oct 25 05:50:14.759: FS_REG  NetFlow(1 - ft 0xBC178E0) - ok
Oct 25 05:50:14.767: FS_REG  MPLS LFD(3 - ft 0xC1A3700) - ok
Oct 25 05:50:14.767: FS_REG  IPRM(0 - ft 0xC1A5478) - ok
Oct 25 05:50:15.203: SubSys  fib init
Oct 25 05:50:15.211: Walk    recalc o/p chains for IPv4
Oct 25 05:50:15.211: Walk    recalc o/p chains for IPv6
Oct 25 05:50:15.211: SubSys  fib_ios init
Oct 25 05:50:15.215: SubSys  fib_rp init
Oct 25 05:50:15.227: SubSys  ipv4fib init
Oct 25 05:50:15.227: SubSys  fib_ios_if init
Oct 25 05:50:15.231: SubSys  ipv4fib_ios init
Oct 25 05:50:15.231: Flag    Common CEF enabled set to yes
Oct 25 05:50:15.231: Flag    IPv4 CEF enabled set to yes
Oct 25 05:50:15.231: Flag    IPv4 CEF switching enabled set to yes
Oct 25 05:50:15.231: Flag    IPv4 dCEF enabled set to yes
Oct 25 05:50:15.231: GState  CEF enabled
Oct 25 05:50:15.235: SubSys  ipv6fib_ios init
Oct 25 05:50:15.235: SubSys  fib_ios_table init
Oct 25 05:50:15.235: SubSys  ipv4fib_util init
Oct 25 05:50:15.235: SubSys  ipv4fib_rp init
Oct 25 05:50:15.235: SubSys  fib_lc_if_broker init
Oct 25 05:50:15.235: SubSys  fib_rrp init
Oct 25 05:50:15.355: SubSys  ipv6fib_rp init
Oct 25 05:50:15.355: SubSys  ipv6fib_issu init
Oct 25 05:50:15.359: SubSys  ipv4fib_les init
Oct 25 05:50:15.359: SubSys  cef_push_issu init
Oct 25 05:50:15.359: SubSys  fib_issu_utils init
Oct 25 05:50:15.359: SubSys  fibhwidb_issu init
Oct 25 05:50:15.359: SubSys  fibidb_issu init
Oct 25 05:50:15.363: SubSys  ipv4fib_issu init
Oct 25 05:50:16.927: Process Background created
Oct 25 05:50:16.927: Flag    IPv4 CEF running set to yes
Oct 25 05:50:16.927: Walk    re-resolve all paths
Oct 25 05:50:16.927: Process Background event loop enter
Oct 25 05:50:16.931: Flag    IPv4 dCEF running set to yes
Oct 25 05:50:16.931: GSB     Enabled
Oct 25 05:50:16.935: Process RP IPC created
Oct 25 05:50:16.935: Process RP IPC event loop enter
Oct 25 05:50:36.583: IPv6adm disabled -> enabled and distributed
Oct 25 05:50:36.643: IPv6opr init -> running
Oct 25 05:50:36.643: IPv6opr running -> dist init
Oct 25 05:50:36.643: IPv6opr dist init -> dist running

#remote command switch sh cef events

% Command accepted but obsolete, unreleased or unsupported; see documentation.

Oct 25 05:50:15.997: Inst    unknown -> LC
Oct 25 05:50:15.997: FS_REG  IPRM(0 - ft 0xA0FE650) - ok
Oct 25 05:50:15.997: FS_REG  NetFlow(1 - ft 0xA0FE650) - ok
Oct 25 05:50:15.997: FS_REG  Broker(2 - ft 0xA0FE650) - ok
Oct 25 05:50:15.997: FS_REG  MPLS LFD(3 - ft 0xA0FE650) - ok
Oct 25 05:50:16.001: FS_REG  NetFlow(1 - ft 0xA1176EC) - ok
Oct 25 05:50:16.001: FS_REG  MPLS LFD(3 - ft 0xA149504) - ok
Oct 25 05:50:32.805: FS_REG  NetFlow(1 - ft 0xA0F2688) - ok
Oct 25 05:50:32.805: FS_REG  MPLS LFD(3 - ft 0xA160178) - ok
Oct 25 05:50:33.285: SubSys  fib init
Oct 25 05:50:33.293: Walk    recalc o/p chains for IPv4
Oct 25 05:50:33.293: Walk    recalc o/p chains for IPv6
Oct 25 05:50:33.293: SubSys  ipv4fib init
Oct 25 05:50:33.293: SubSys  ipv4fib_lc init
Oct 25 05:50:33.301: SubSys  fib_lc init
Oct 25 05:50:33.309: SubSys  fib_lc_if_broker init
Oct 25 05:50:33.309: SubSys  ipv4fib_util init
Oct 25 05:50:33.309: SubSys  cef_push_issu init
Oct 25 05:50:33.309: SubSys  fib_issu_utils init
Oct 25 05:50:33.309: SubSys  fibhwidb_issu init
Oct 25 05:50:33.309: SubSys  fibidb_issu init
Oct 25 05:50:33.309: SubSys  ipv4fib_issu init
Oct 25 05:50:33.309: SubSys  ipv6fib_issu init
Oct 25 05:50:40.325: Process LC IPC created
Oct 25 05:50:40.325: Process LC IPC event loop enter
Oct 25 05:50:43.241: GSB     Enabled
Oct 25 05:50:43.241: GSB     Registered Adjacency update, mask : 7
Oct 25 05:50:43.241: GSB     Registered IPv4 table broker, mask : 7,9
Oct 25 05:50:43.241: GSB     Registered IPv6 table broker, mask : 7,9-10
Oct 25 05:50:43.245: Flag    Common CEF enabled set to yes
Oct 25 05:50:43.245: Flag    IPv4 CEF enabled set to yes
Oct 25 05:50:43.245: Flag    IPv4 dCEF running set to yes
Oct 25 05:50:43.245: GState  CEF enabled
Oct 25 05:50:43.253: Process LC Stats created
Oct 25 05:50:43.253: Process LC Stats event loop enter
Oct 25 05:50:43.257: Process Background created
Oct 25 05:50:43.257: Flag    IPv4 CEF running set to yes
Oct 25 05:50:43.257: Walk    re-resolve all paths
Oct 25 05:50:43.257: Process Background event loop enter
Oct 25 05:50:43.261: IPv6adm disabled -> enabled and distributed
Oct 25 05:50:43.465: Flag    IPv4 CEF switching running set to yes
Oct 25 05:50:43.469: IPv6opr init -> running
Oct 25 05:50:43.469: IPv6opr running -> dist init
Oct 25 05:50:43.469: IPv6opr dist init -> dist running

sh ip cef events also doesn't mean much to me:

Nov 23 23:31:35.784: [v4:Default] 141.170.64.0/19'05    FIB mod src RIB          [OK]

Nov 23 23:31:35.784: [v4:Default] 141.170.96.0/22'05    FIB mod src RIB          [OK]

Nov 23 23:31:35.784: [v4:Default] 141.170.100.0/23'05   FIB mod src RIB          [OK]

Nov 23 23:31:35.948: [v4:Default] 86.104.160.0/21'05    FIB mod src RIB          [OK]

Nov 23 23:31:35.948: [v4:Default] 193.25.104.0/23'05    FIB mod src RIB          [OK]

Nov 23 23:31:38.856: [v4:Default] 88.222.100.0/22'05    FIB add src RIB (ins)    [OK]

Nov 23 23:31:38.856: [v4:Default] 88.222.104.0/22'05    FIB add src RIB (ins)    [OK]

Nov 23 23:31:39.032: [v4:Default] 203.84.136.0/23'05    FIB add src RIB (ins)    [OK]

Nov 23 23:31:39.456: [v4:Default] 109.200.16.136/32'05  FIB add src RR           [OK]

Nov 23 23:31:39.456: [v4:Default] 109.200.16.137/32'05  FIB mod src RR           [Ignr]

Nov 23 23:31:39.456: [v4:Default] 31.3.224.0/19'05      FIB mod src RIB          [OK]

Nov 23 23:31:40.064: [v4:Default] 77.245.64.0/20'05     FIB mod src RIB          [OK]

Nov 23 23:31:40.064: [v4:Default] 80.84.48.0/20'05      FIB mod src RIB          [OK]

Nov 23 23:31:40.064: [v4:Default] 203.84.136.0/23'05    FIB mod src RIB          [OK]

Nov 23 23:31:40.064: [v4:Default] 218.32.58.0/24'05     FIB add src RIB (ins)    [OK]

Nov 23 23:31:40.272: [v4:Default] 80.243.176.0/20'05    FIB mod src RIB          [OK]

Nov 23 23:31:40.272: [v4:Default] 81.94.192.0/20'05     FIB mod src RIB          [OK]

Nov 23 23:31:40.476: [v4:Default] 109.73.64.0/20'05     FIB mod src RIB          [OK]

Nov 23 23:31:40.476: [v4:Default] 109.200.0.0/19'05     FIB mod src RIB          [OK]

Nov 23 23:31:40.476: [v4:Default] 218.32.58.0/24'05     FIB mod src RIB          [OK]

Nov 23 23:31:40.880: [v4:Default] 149.3.128.0/21'05     FIB mod src RIB          [OK]

Nov 23 23:31:40.880: [v4:Default] 149.3.136.0/22'05     FIB mod src RIB          [OK]

Nov 23 23:31:41.080: [v4:Default] 91.103.56.0/24'05     FIB add src RIB (ins)    [OK]

Nov 23 23:31:41.284: [v4:Default] 188.227.160.0/19'05   FIB mod src RIB          [OK]

Nov 23 23:31:41.692: [v4:Default] 194.24.236.0/24'05    FIB mod src RIB          [OK]

Nov 23 23:31:41.692: [v4:Default] 194.24.237.0/24'05    FIB mod src RIB          [OK]

Nov 23 23:31:42.096: [v4:Default] 80.83.128.0/24'05     FIB add src RIB (ins)    [OK]

Nov 23 23:31:42.096: [v4:Default] 80.83.128.0/20'05     FIB add src RIB (ins)    [OK]

Nov 23 23:31:42.096: [v4:Default] 80.83.132.0/24'05     FIB add src RIB (ins)    [OK]

Nov 23 23:31:42.096: [v4:Default] 80.83.142.0/24'05     FIB add src RIB (ins)    [OK]

Nov 23 23:31:42.096: [v4:Default] 80.241.248.0/24'05    FIB add src RIB (ins)    [OK]

Nov 23 23:31:42.096: [v4:Default] 91.103.56.0/24'05     FIB mod src RIB          [OK]

Nov 23 23:31:42.096: [v4:Default] 91.184.106.0/24'05    FIB add src RIB (ins)    [OK]

Is this any help?

Thank you for helping me.

0 Helpful
Customers Also Viewed These Support Documents
Review Cisco Networking products for a $25 gift card