Memory leak - Sup1A CatOS 8.5.1

Unanswered Question

We have been experiencing a steady memory leak on our Cat6000 with Sup1/MSFC running hybrid CatOS/IOS. We suspect the issue is related to using the Sup1A as a VMPS server for the LAN, as the memory leak has worsened as the size of our vmps.cfg file grows (currently about 1100 lines).

The leak can be seen as an increase in the amount of memory used by "Dot1x_Dhcp" as shown by a 'show proc mem' command. Below is the output after switch has been running about 24 hours. When the Free memory drops to about 11,000,000, the switch will no longer service logins at the console or respond to vmps client switches, resulting in clients losing network connectivity. (when this occurs, Dot1xDhcp  is using 20Mb or more). The leak is now so severe that a reboot (or forced failover to 2nd Sup) is required every 4-5 weeks.

We have a C6509 with Sup2/MSFC2 waiting to replace this switch, but the lack of VMPS support in native IOS is a problem - we do not have the resources to switch to dot1x at this time, as it would require replacement of about 40 edge switches (2900/3500XL) that don't support dot1x. We're hoping for a solution to this problem that will allow us to keep this switch in service another 4-6 months while we plan for dot1x and native IOS migration. Any suggestions appreciated.

Memory Used:  15141408
       Free:  26421056
      Total:  41562464

PID        TTY        Allocated  Freed      Holding    Process
---------- ---------- ---------- ---------- ---------- ---------------
1          -2         13286272   3258112    10028160   Kernel and Idle
2          -2         192        0          192        Flash MIB Updat
3          -2         192        0          192        L2IntHdlr     
4          -2         0          0          0          EobRecoveryProc
5          -2         762304     704608     57696      SynDiags      
6          -2         128        0          128        GenMsgHndlr   
7          -2         2169024    1065312    1103712    SynConfig     
8          -2         64         0          64         TempMon       
9          -2         0          0          0          System_CrashInf
10         -2         3456       0          3456       IDLE_Tasks    
11         -2         320        128        192        PowerMgmt     
12         -2         15716416   15716064   352        Debug Port Coun
13         -2         137568     37088      100480     SL_TASK       
14         -2         1115328    185280     930048     RedundantTask 
15         -2         12073728   12071232   2496       Status Poll   
16         -2         64         0          64         SWPoll64bCnt  
18         -2         43904      42208      1696       VMPS_Task     
19         0          971904     970368     1536       HavailTask    
20         -2         4224       0          4224       SyncTask      
21         -2         338400     338368     32         Earl          
22         -2         16608      0          16608      TablefullTask 
23         -2         1109408    1101952    7456       DTP_Rx        
24         -2         489440     311808     177632     EthChnlRx     
25         -2         0          0          0          llcSSTPFlood  
26         -2         692800     251168     441632     EthChnlConfig 
27         -2         20256      18336      1920       FlexlinkRx    
28         -2         102560     0          102560     SecurityRx    
29         -2         55328      54016      1312       ACL           
30         -2         136224     108416     27808      DhcpSnooping  
31         -2         48128      28512      19616      Authenticator_S
32         -2         32         0          32         Backend_SM    
33         -2         160        0          160        dot1x_rx      
34         -2         775648     0          775648     Dot1x_Dhcp    
35         -2         160        0          160        GrpManager    
36         -2         32         0          32         Backend_Rx    
37         -2         32         0          32         Radius Acct Pro
38         -2         981280     976896     4384       NetFlow       
39         -2         0          0          0          L3Aging       
40         -2         64         0          64         MlsHa         
41         0          190464     188608     1856       SptTimer      
42         0          6368       6208       160        SptBpduRx     
43         0          160        0          160        SptBpduTx     
44         0          0          0          0          sptTraps      
45         -2         160        0          160        GL2Prot_Tunnel
46         -2         224        0          224        VtpTimer      
47         0          36928      0          36928      Read Stats Task
48         0          64         0          64         QDE Task      
49         0          64         0          64         BcastSuppr Task
50         -2         448        192        256        oam_tx_proc   
51         -2         160        0          160        oam_rx_proc   
52         -2         23282848   23282688   160        oam_monitoring_
53         -2         524352     0          524352     NvramMonitor  
54         -2         160        0          160        DeviceLinkChk 
55         -2         1504       0          1504       gentcp_act    
56         -2         974432     948320     26112      McastRx       
57         -2         185504     132000     53504      RouterConfig  
58         -2         13940288   13929856   10432      cdpd          
59         -2         32721568   32721344   224        cdpdtimer     
60         -2         0          0          0          sptTraps      
61         -2         224        0          224        ProtocolFilter
62         -2         544        0          544        MacMoveNotTask
63         -2         512        480        32         telnetd       
64         -2         32         0          32         connecthelper 
65         -2         32         0          32         tftpd         
66         -2         6559872    6559776    96         ProtocolTimer 
67         -2         64         0          64         ciscoRmonTimer
68         -2         64         0          64         ciscoUsrHistory
69         -2         96         0          96         rmonMediaIndep
70         -2         0          0          0          SnmpTraps     
71         -2         0          0          0          memPoolMain   
73         -2         64         0          64         RMON AlarmTimer
76         -2         32         0          32         Acct Send Bkg 
77         -2         128        0          128        l2t_server    
79         -2         32         0          32         SysLogTask    
80         -2         160        0          160        pinggateA     
81         -2         96         0          96         MacAuthProc   
82         -2         96         0          96         AuthControlMgr
83         -2         32         0          32         McastIgmpTimer
84         -2         32         0          32         IGMPQuerierProc
85         -2         96         0          96         M-MLS_stats   
86         -2         4416       64         4352       M-MLS_manager 
87         -2         67232      15840      51392      QoSTask       
88         -2         7840       7776       64         GBIC Process  
89         -2         32         0          32         HPActive      
90         -2         96         0          96         HPTrapMgr     
143        -2         288        0          288        VtpRx         
144        0          112352     57696      54656      Console       
145        -2         100227136  99821664   405472     snmpdm        
146        -2         19637440   19636640   800        SNMP_THREAD   
147        2241991444 97056      33568      63488      telnet147     
156        -2         2101504    2010944    90560      AclManager  

Memory Pool Utilization
Memory Pool Type 1Min  5Min  10Min
---------------- ----- ----- -----
DRAM               55%   55%   55%
FLASH              87%   87%   87%
NVRAM              67%   67%   67%
MBUF                4%    4%    4%
CLUSTER             9%    9%    9%
MALLOC             36%   36%   36%

show ver:

WS-C6006 Software, Version NmpSW: 8.5(1)
Copyright (c) 1995-2005 by Cisco Systems
NMP S/W compiled on Oct 22 2005, 11:28:54

System Bootstrap Version: 5.3(1)
System Boot Image File is 'bootflash:cat6000-supk8.8-5-1.bin'
System Configuration register is 0x2

Hardware Version: 2.0  Model: WS-C6006  Serial #: TBA04290439

PS1  Module: WS-CAC-1300W    Serial #: SNI06120229
PS2  Module: WS-CAC-1300W    Serial #: ACP03280942

Mod Port Model               Serial #    Versions
--- ---- ------------------- ----------- --------------------------------------
1   2    WS-X6K-SUP1A-2GE    SAD04260GEA Hw : 3.2
                                         Fw : 5.3(1)
                                         Fw1: 5.1(1)CSX
                                         Sw : 8.5(1)
                                         Sw1: 8.5(1)
         WS-F6K-PFC          SAD041701CC Hw : 1.1  
                                         Sw :
2   2    WS-X6K-SUP1A-2GE    SAD04510V1L Hw : 7.0
                                         Fw : 5.3(1)
                                         Fw1: 5.4(2)
                                         Sw : 8.5(1)
                                         Sw1: 8.5(1)
         WS-F6K-PFC          SAD04510GPE Hw : 1.1  
                                         Sw :
3   8    WS-X6408A-GBIC      SAD0429002D Hw : 1.3
                                         Fw : 0.0
                                         Sw : 8.5(1)
4   24   WS-X6224-100FX-MT   SAD03310103 Hw : 2.0
                                         Fw : 0.0
                                         Sw : 8.5(1)
5   16   WS-X6516-GE-TX      SAL08517EJK Hw : 2.6
                                         Fw : 0.0
                                         Sw : 8.5(1)
6   16   WS-X6316-GE-TX      SAD0732056W Hw : 1.3
                                         Fw : 0.0
                                         Sw : 8.5(1)
15  1    WS-F6K-MSFC         SAD04260JH6 Hw : 1.4
                                         Fw : 12.1(26)E4
                                         Sw : 12.1(26)E4
16  1    WS-F6K-MSFC         SAD04030BBG Hw : 1.4
                                         Fw : 12.1(26)E4
                                         Sw : 12.1(26)E4

       DRAM                    FLASH                   NVRAM
Module Total   Used    Free    Total   Used    Free    Total Used  Free
------ ------- ------- ------- ------- ------- ------- ----- ----- -----
1      130944K  73308K  57636K  15616K  13589K   2027K  512K  346K  166K
2      130944K

Uptime is 0 day, 22 hours, 37 minutes

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
glen.grant Sat, 01/30/2010 - 04:11

   Probably your only option is going to be reload the box if its memory leak .  You could try the very last version of code they made for this and see if it helps or check the bug toolkit for bugs for that version .  Catos is riding  that slow sunset into the sky and is EOL . All software fixes ended in Jan. of 2009 so they won't fix anything at this point .  Think 8.7.3 was the last release for catos.

Actions

This Discussion

Related Content