cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
5089
Views
0
Helpful
6
Replies

WS-C2960S-48FPD-L with high memory usage.

leocneves
Level 1
Level 1

I have in my network has a WS-C2960S-48FPD-L that is using more than 90% of memory. The IOS version installed is 150-2.SE2 (c2960s-universalk9-mz.150-2.SE2.bin), could not find out if this is a BUG. The processing is around 23%. Below are the "show processes cpu sorted 5min" and "show processes memory sorted allocated". Thanks for help.

SW_ACC_02_TR#show processes cpu sorted 5min

CPU utilization for five seconds: 24%/0%; one minute: 26%; five minutes: 25%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

155  4195888138   639952845       6556 15.59% 16.41% 16.21%   0 Hulc LED Process

   4   248001934    13719097      18077  0.00%  1.14%  0.94%   0 Check heaps     

124   169412938    25660800       6602  0.49%  0.53%  0.56%   0 hpm counter proc

212   118535249   574852125        206  0.59%  0.51%  0.48%   0 Spanning Tree   

120    41576380   332054932        125  0.29%  0.31%  0.27%   0 hpm main process

166    59299274     5119156      11583  0.19%  0.22%  0.20%   0 HQM Stack Proces

  85    31122391  1161331958         26  0.39%  0.17%  0.18%   0 RedEarth Rx Mana

  10    46775005      426883     109574  0.00%  0.13%  0.15%   0 Licensing Auto U

284    19497943    56623180        344  0.19%  0.14%  0.15%   0 Marvell wk-a Pow

103    20399008   703273565         29  0.00%  0.13%  0.12%   0 HLFM address lea

  78    15964116   925562055         17  0.00%  0.10%  0.10%   0 Draught link sta

  84    20573751   272991015         75  0.09%  0.09%  0.09%   0 RedEarth Tx Mana

358     8190900     2585781       3167  0.00%  0.12%  0.08%   0 SNMP ENGINE     

  56    35021439    25699423       1362  0.00%  0.08%  0.07%   0 Per-Second Jobs 

198    12025895   703271618         17  0.09%  0.04%  0.06%   0 IP ARP Retry Age

105    12471548   706772431         17  0.00%  0.05%  0.05%   0 HLFM address ret

186    19272066    15440290       1248  0.00%  0.06%  0.05%   0 CDP Protocol    

  44    14077734    31430059        447  0.00%  0.04%  0.05%   0 Net Background  

  12    11847905    35899234        330  0.00%  0.05%  0.04%   0 ARP Input       

199     7505111    33071459        226  0.00%  0.02%  0.03%   0 IP Input        

228     9706327    25585689        379  0.00%  0.02%  0.03%   0 PI MATM Aging Pr

356     3004646     4774701        629  0.00%  0.04%  0.01%   0 IP SNMP         

229     6628564   256439705         25  0.00%  0.02%  0.01%   0 UDLD            

357     1236671     2523911        489  0.00%  0.03%  0.00%   0 PDU DISPATCHER  

161         230        2446         94  0.39%  0.04%  0.00%   1 Virtual Exec    

147     4448561   125135068         35  0.00%  0.04%  0.00%   0 Hulc Storm Contr

  55     9622045      444117      21665  0.39%  0.04%  0.00%   0 Per-minute Jobs 

  53     1861943    34905451         53  0.09%  0.00%  0.00%   0 Net Input       

167     5308849    10236218        518  0.09%  0.01%  0.00%   0 HRPC qos request

  65     3560170   479944329          7  0.09%  0.01%  0.00%   0 DownWhenLooped  

_________________________________________________________________________________________________

SW_ACC_02_TR#show processes memory sorted allocated

Processor Pool Total:   66607236 Used:   57945988 Free:    8661248

      I/O Pool Total:   14680064 Used:   12340676 Free:    2339388

Driver te Pool Total:    1048576 Used:         40 Free:    1048536

PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process

  38   0 3637735668 3638354640       7692          0          0 crypto sw pk pro

167   0 3416215012 3415330192      90424     492372          0 HRPC qos request

166   0 2108866696  968241432     170224    3395844          0 HQM Stack Proces

356   0 2088379404 2074798724      25744    7004880          0 IP SNMP        

358   0 2069043972 2520358128      16416     258876          0 SNMP ENGINE    

  10   0 1800214384 1800214384       7160          0          0 Licensing Auto U

186   0  874988176     125196     865416   24938388          0 CDP Protocol   

357   0  418086688          0      33248          0          0 PDU DISPATCHER 

183   0  249277624  211662804   18074212          0          0 Auth Manager   

257   0  230142288  230141476       7972          0          0 Cluster Base   

271   0  225815492  225815492       7160          0          0 QoS stats proces

290   0  199977112  199958180       8732      10152          0 NIST rng proc  

  12   0  121517796  121481096      43860          0          0 ARP Input      

120   0   97428096   92337752      58976          0          0 hpm main process

  92   0   93195956 1196348440       7160          0      99828 hrpc <- response

   0   0   76977196   73891632    1827512   12832067     928178 *Dead*         

   0   0   73660260   27113924   41827516          0          0 *Init*         

   5   0   68336032  132544800      32920   61954432   99416795 Pool Manager   

214   0   64277948   60183624      68312          0          0 802.1x switch  

107   0   46027340   45332908       7160     406080          0 HRPC hulc misc r

  82   0   37271008   37271008      10160          0          0 Adjust Regions 

363   0   28303364   28300096      31596          0          0 hulc running con

212   0   18442748        876       7376          0          0 Spanning Tree  

199   0   17137088      22760      31028          0          0 IP Input       

103   0   13616600        232     157084          0          0 HLFM address lea

187   0    8341472    6196184       8300          0          0 SpanTree Helper

332   0    6298952   12438752      11568          0          0 PM Callback    

285   0    4720612   14241044      33776          0          0 MLDSN L2MCM    

278   0    4699604   14251332      30388          0          0 IGMPSN L2MCM   

258   0    3213116     856496      56308          0          0 DHCPD Receive  

303   0    2247272        600     129208    1248696          0 Crypto IKEv2   

346   0    2008824    2011716      13160          0       1692 Syslog         

  44   0    1973632    1837448      95588          0          0 Net Background 

252   0    1811708   17030976      13160          0          0 TCP Protocols  

  80   0    1754692    1744796      15896          0          0 OBFL TEMP obfl0

  89   0    1213028     150984     725664          0          0 Stack Mgr Notifi

333   0    1022008    1017008      18032          0          0 OBFL VOLT obfl0

   1   0     642016     470972     195248          0          0 Chunk Manager  

354   0     614096          0      15632          0          0 SSH Event handle

216   0     589100     227808      34860          0          0 DTP Protocol   

PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process

  32   0     549552     549552       7160          0          0 PrstVbl        

302   0     422492     421180       8472          0          0 crypto engine pr

  61   0     368228        600     377788          0          0 EEM ED Identity

176   0     294744          0     294744          0          0 HL2MCM         

177   0     294744          0     294744          0          0 HL2MCM         

325   0     265100          0     275260     100548          0 EEM ED Syslog  

315   0     236408     201388      48884          0          0 Crypto CA      

350   0     113924      90852      15020       3120          0 VLAN Manager   

336   0     107464          0     114624          0          0 EEM Server     

  49   0     101540      84896      63848          0          0 IF-MGR control p

161   1      93412      75972      42524          0          0 Virtual Exec   

108   0      92752          0      99912      53760          0 HRPC emac reques

  16   0      82600      79336      73296          0      11180 Entity MIB API 

  55   0      78164        412       7160      45684          0 Per-minute Jobs

204   0      76192      74416      14808          0          0 OBFL MSG obfl0 

311   0      71960       5324      13484          0          0 IPSEC key engine

231   0      71328          0      78488          0          0 Switch Backup In

337   0      68292      18904      48984          0          0 Call Home proces

  86   0      65972        232       7160          0          0 HULC Thermal Pro

143   0      65812        232      72740          0          0 REP Helper Proc

  64   0      58096        232      39864          0          0 USB Startup    

237   0      56128        380      69044          0          0 CEF background p

188   0      54936       9784      39956          0          0 CEF: IPv4 proces

254   0      52588      40728      22020          0          0 HTTP CORE      

  13   0      51560      51560       7176          0          0 ARP Background 

291   0      49584        232      56512          0          0 Timer Library  

180   0      49368          0      56528          0          0 ACCT Periodic Pr

  46   0      43612        232      13160      25380          0 Logger         

198   0      33020          0      40180          0          0 IP ARP Retry Age

  56   0      22152          0      27020          0          0 Per-Second Jobs

  84   0      21584      21836       7160          0          0 RedEarth Tx Mana

197   0      20388       3196      30352          0          0 IP ARP Adjacency

282   0      14912        232      11752          0          0 L2TRACE SERVER 

361   0      14764      10932      14836          0          0 SNMP Traps     

338   0      13020       1720      18460          0          0 EEM Policy Direc

  85   0      12844      22640       7504          0          0 RedEarth Rx Mana

301   0      12408          0      37440          0          0 LICENSE AGENT  

   0   0      12316    3117752      12316          0          0 *Sched*        

309   0      12268        232      25196          0          0 Crypto IKMP    

347   0      12164        464      24860          0          0 RBM CORE       

208   0      10724          0      35884          0          0 SMI Backup Proce

329   0      10652        392      20292          0          0 OBFL ENV obfl0 

274   0      10276        232      17204          0          0 REP BPA/EPA Proc

206   0       8204       3408      33364          0          0 SMI Director DB

112   0       5044          0      12204          0          0 HRPC vlan reques

PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process

320   0       4884        232      11812          0          0 VTP Trap Process

211   0       4884        232      17812          0          0 SMI IBC Download

294   0       4884        232      11848          0          0 Crypto Support 

   4   0       4772        500      11752          0          0 Check heaps    

340   0       4556          0      14716          0          0 EEM ED Config  

  63   0       4556  547248440      14716          0          0 EEM ED ND      

  62   0       4556          0      14716          0          0 EEM ED MAT     

341   0       4556          0      14716          0          0 EEM ED Env     

343   0       4556          0      14716          0          0 EEM ED OIR     

342   0       4556          0      14716          0          0 EM ED GOLD     

345   0       4556          0      14716          0          0 EEM ED Timer   

344   0       4556          0      14716          0          0 EEM ED Test    

  52   0       4140       4140      10160          0          0 IP Admission HA

209   0       3892        232      10820          0          0 SMI IBC server p

115   0       2244          0       7160          0          0 Hulc ILP Alchemy

300   0       1936        232       8864          0          0 TPLUS          

  48   0       1936       7620      14864          0          0 SXP CORE       

293   0       1936        232       8864          0          0 LDAP process   

  26   0       1688          0       8848          0          0 IPC Seat Manager

359   0       1156        232      14092          0          0 IP SNMPV6      

264   0       1120          0      14280          0          0 IP RIB Update  

283   0        924          0       8084          0          0 Inline Power   

193   0        868        708       7320          0          0 Dot1x Mgr Proces

  95   0        640        232       7160          0          0 HULC Device Mana

2 Accepted Solutions

Accepted Solutions

Leo Laohoo
Hall of Fame
Hall of Fame
The IOS version installed is 150-2.SE2 (c2960s-universalk9-mz.150-2.SE2.bin), could not find out if this is a BUG

Avoid using 15.0(2)SE2 at all cost.  There are so many CPU and memory bugs it's well known joke.  Hey, you should try 15.0(2)SE3, if you can still find this IOS, and then you'll be laughing all the way to the loonie-bin.

Use either 12.2(55)SE8 or 15.0(2)SE4 instead.

View solution in original post

Bhishma Khanna
Cisco Employee
Cisco Employee

Hi,

To troubleshoot memory please follow the below steps:

First, please check if the memory usage is genuine or not, this can be checked by checking the memory status of any other similar switch in the network having similar devices attached. This check is important because even a 95% of memory usage could be normal depending upon the requirements of your network.

If the memory usage is really abnormal and the switch is crashing then note the time between the two crashes.

If the two crashes are in  a GAP of 7 days then take the outpust of "show process memory sorted" 5 times at a gap of 24-36 hours.

It will give you an idea about the process which is constantly holding more and more memory without releasing it.

Once you know the process, you can track the relevant bug.

Moreover, on the 2960 switch the 15.X train has a lot of memory bugs, I would recommend you to downgrade it to 12.2(55) train as it is one of the most stable IOS images on this platform.

Thanks.

View solution in original post

6 Replies 6

Leo Laohoo
Hall of Fame
Hall of Fame
The IOS version installed is 150-2.SE2 (c2960s-universalk9-mz.150-2.SE2.bin), could not find out if this is a BUG

Avoid using 15.0(2)SE2 at all cost.  There are so many CPU and memory bugs it's well known joke.  Hey, you should try 15.0(2)SE3, if you can still find this IOS, and then you'll be laughing all the way to the loonie-bin.

Use either 12.2(55)SE8 or 15.0(2)SE4 instead.

Bhishma Khanna
Cisco Employee
Cisco Employee

Hi,

To troubleshoot memory please follow the below steps:

First, please check if the memory usage is genuine or not, this can be checked by checking the memory status of any other similar switch in the network having similar devices attached. This check is important because even a 95% of memory usage could be normal depending upon the requirements of your network.

If the memory usage is really abnormal and the switch is crashing then note the time between the two crashes.

If the two crashes are in  a GAP of 7 days then take the outpust of "show process memory sorted" 5 times at a gap of 24-36 hours.

It will give you an idea about the process which is constantly holding more and more memory without releasing it.

Once you know the process, you can track the relevant bug.

Moreover, on the 2960 switch the 15.X train has a lot of memory bugs, I would recommend you to downgrade it to 12.2(55) train as it is one of the most stable IOS images on this platform.

Thanks.

In my case, i have already know that HL2MCM process is holding the highest memory without releasing it.

What can i do next to resolve this memory leaking issue.

ios 15.2(2) E7

leocneves
Level 1
Level 1

Hello guys I did the replacement of 150-2.SE2 by the 12.2 (55) SE8, the problem was solved. I am very happy to have the help of you. Thank you.

Thank you Leo Laohoo!

Thank you Bhishma Khanna!

No Problem,

Thanks for rating the helpful posts.

Hello guys I did the replacement of 150-2.SE2 by the 12.2 (55) SE8, the problem was solved. I am very happy to have the help of you. Thank you.

Thank you for taking the time to give us the feedback(s) of your network improvement. 

And thank you for taking the time to rate our posts. 

FYI:  12.2(55)SE, starting from SE6, is considered to be one of the most stable IOS versions that ever came out of Cisco. 

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: