04-29-2010 11:16 AM - edited 03-13-2019 06:58 PM
We are running gatekeeper IOS version 12.3(11r)T2 where we have to reboot GK monthly to avoid memory leak issue causing GK to be nonfunctional after some time.
It was recommended to have monthly reboots to avoid the memory leaks. It has worked since it has avoided any additional memory leak. The reboots were recommended by Cisco to avoid the memory leak.
Does any one know if there is fix for this now or if any new IOS version has a fix.
Any help will be appreciated.
Regards,
Brajesh.
05-03-2010 01:07 PM
I think that this question should go to Cisco and one should not accept reload based workarounds.
05-17-2010 12:08 PM
Brajesh,
12.3(11r)T2 sounds like the IOS bootloader version. Can you copy the output of "show version." I can do a quick search to determine if there are any know leaks. Do you know which process actually leaks? I agree that you may want to open a TAC SR to start profiling memory utilization to determine the cause of the leak.
To view the current memory allocation use,
show process memory sorted
-Felipe
05-17-2010 07:56 PM
Felipe,
Here is show version.
vus-cvpgk1#sh ver
Cisco IOS Software, 3800 Software (C3825-ADVENTERPRISEK9_IVS-M), Version 12.4(15
)T7, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Wed 13-Aug-08 23:16 by prod_rel_team
ROM: System Bootstrap, Version 12.3(11r)T2, RELEASE SOFTWARE (fc1)
vus-cvpgk1 uptime is 2 weeks, 4 hours, 7 minutes
System returned to ROM by reload at 23:28:25 GMT Mon May 3 2010
System restarted at 23:30:02 GMT Mon May 3 2010
System image file is "flash:c3825-adventerprisek9_ivs-mz.124-15.T7.bin"
Cisco 3825 (revision 1.1) with 1011712K/36864K bytes of memory.
Processor board ID FTX1025A308
2 Gigabit Ethernet interfaces
1 Virtual Private Network (VPN) Module
DRAM configuration is 64 bits wide with parity enabled.
479K bytes of NVRAM.
1930760K bytes of ATA System CompactFlash (Read/Write)
Configuration register is 0x2102
=========================
The below bug seems to be close to our scenario. Please have a look.
CSCsw23397
1st Found-In
12.4T
12.4(15)T7
12.4(15)T8
Voice Gateway may leak memory for DSMP EVENT chu and DSMP DSP REQ chunk
Symptoms: A Cisco Router acting as a voice Gateway may leak memory for DSMP DSP REQ chunk and DSMP DSP REQ chunk
Conditions: The symptom appears to be triggered by calls that disconnect prematurely.
Fixed-In
12.4(22)MD1
12.4(20)T2
12.4(23.12)M
12.4(22)T1
12.4(23.12)T
12.4(23.12)PI10
12.4(23.15.1)PIX11
12.4(22)MDA
12.4(23a)M
12.4(23.15.4)PIC1
12.4(24.5.1)PIX11
12.4(15)T9
12.4(24.5.2)PIC1
12.4(22)XR
12.4(22)YE2
05-18-2010 01:52 PM
I wouldn't expect that to be the bug unless the gatekeeper gateway also has TDM/analog interface handling calls. Do you have any information on the process that's leaking? Can you get the output of "show proc mem sorted" or provide the previous TAC SR number opened to originally address the issue?
05-18-2010 07:58 PM
Earlier resolution what we found from Cisco is to reboot GK to fix this issue.
Please find the command output.
---------------------------
Processor Pool Total: 919148940 Used: 24673240 Free: 894475700
I/O Pool Total: 37748224 Used: 12959664 Free: 24788560
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 64213880 24742544 31780752 0 0 *Init*
68 0 659772 1328 640444 0 0 USB Startup
122 0 3760476856 3760018464 470240 0 0 gk process
118 0 365608 1052 367364 0 0 EAPoUDP Process
26 0 264500 0 274000 113400 0 EEM ED Syslog
214 0 247992 0 255196 0 0 QOS_MODULE_MAIN
254 0 161220 10404 167812 0 0 EEM Server
1 0 112392 13968 119596 0 0 Chunk Manager
0 0 61174996 63634536 104152 258728 0 *Dead*
4 0 65588 0 90792 0 0 EDDRI_MAIN
225 0 66852 0 80056 0 0 Crypto IKEv2
130 0 74304 0 74256 0 0 CEF process
248 0 75212 9024 73392 0 0 Syslog Traps
178 0 66416 252 73368 0 0 HTTP Process
229 0 45288 3204 67288 0 0 Crypto ACL
63 0 57904 252 64856 0 0 VNM DSPRM MAIN
212 0 940 0 62144 0 0 http client proc
95 0 49456 0 56660 0 0 ACCT Periodic Pr
101 0 49444 0 56648 0 0 IP ARP Retry Age
183 0 32136 252 39088 0 0 CCVPM_HDSPRM
143 0 39244 252 38060 0 0 DHCPD Receive
201 0 26200 0 36404 0 0 CCVPM_HTSP
134 0 10052 0 35256 0 0 COPS
102 0 234876 144492 34628 0 0 IP Input
271 0 17980 0 31252 0 0 IP SLAs Responde
46 0 45512 800 30712 0 0 rf task
175 0 16812 252 29764 0 0 IPS Auto Update
270 578 4164412 4144008 27816 0 0 Virtual Exec
18 0 22452 2400 27256 0 0 ARP Input
121 0 1068 252 26020 0 0 Border Element p
145 0 18944 252 25896 0 0 SCTP Main Proces
215 0 284 0 25488 0 0 RPMS_PROC_MAIN
267 0 252 0 25456 0 0 tHUB
71 0 0 0 25204 0 0 RF_INTERDEV_SCTP
9 0 0 0 25204 0 0 Crash writer
78 0 0 0 25204 0 0 CF_INTERDEV_SCTP
216 0 0 0 25204 0 0 VoIP AAA
224 0 9816 252 22768 0 0 Crypto IKMP
180 0 12592 252 19544 0 0 CIFS Proxy Proce
54 0 11688 0 18892 0 0 mxt5100
182 0 10556 504 17256 0 0 Crypto HW Proc
211 0 10052 0 17120 0 0 cpf_process_tpQ
217 0 31220 11892 16828 0 0 crypto engine pr
97 0 9380 252 16332 0 0 CDP Protocol
263 0 10020 912 16312 0 0 EEM Policy Direc
207 0 8244 0 15448 0 0 Skinny HW confer
264 0 11856 9888 15172 0 0 Syslog
39 0 24152 968 15024 0 0 Net Background
252 0 12388 4504 14848 0 0 VLAN Manager
218 0 2428 832 14800 0 0 Crypto CA
157 0 3972 0 14176 0 0 HTTP CORE
272 0 3212352 3211452 14172 0 0 IP SNMP
275 0 1152 252 14172 0 0 IP SNMPV6
277 0 25868 9248 13856 6804 0 SNMP Traps
40 0 1400 1096 13508 0 0 IDB Work
227 0 13872 12640 13372 0 0 IPSEC key engine
PID TTY Allocated Freed Holding Getbufs Retbufs Process
273 0 582992 0 13272 0 0 PDU DISPATCHER
274 0 16630556 17213548 13272 0 0 SNMP ENGINE
276 0 0 0 13256 0 0 SNMP ConfCopyPro
55 0 0 0 13204 0 0 sal_dpc_process
35 0 252 252 13204 0 0 XML Proxy Client
116 0 252 252 13204 0 0 SSM connection m
41 0 252 252 13204 0 0 Logger
221 0 0 0 13204 0 0 encrypt proc
33 0 252 252 13204 0 0 Dialer event
223 0 252 1188 13204 0 0 Crypto IKE Dispa
23 0 0 0 13204 0 0 Policy Manager
120 0 252 252 13204 0 0 KRB5 AAA
108 0 504 504 13204 0 0 PPP Hooks
167 0 0 0 13204 0 0 VPDN call manage
126 0 316 316 13204 0 0 PPP IPCP
142 0 0 0 13204 0 0 L2X Data Daemon
132 0 0 1060728 13204 0 0 TCP Timer
133 0 5281532 9604 13204 0 0 TCP Protocols
99 0 252 252 13204 0 0 Ethernet LMI
73 0 0 0 13204 0 0 AAL2CPS TIMER_CU
125 0 252 252 13204 0 0 PPP IP Route
111 0 0 0 13204 0 0 SSS Test Client
110 0 0 0 13204 0 0 SSS Manager
174 0 0 0 13204 0 0 IPS Process
8 0 0 0 13204 0 0 OIR Handler
155 0 252 252 13204 0 0 ATM OAM Input
86 0 252 252 13204 0 0 Ethernet Timer C
162 0 252 252 13204 0 0 X.25 Background
85 0 252 252 13204 0 0 Ethernet CFM
156 0 252 252 13204 0 0 ATM OAM TIMER
25 0 5336 0 12540 0 0 Entity MIB API
256 0 2664 0 12256 0 0 EEM ED Counter
79 0 5304 252 12256 0 0 Dot1x Mgr Proces
0 0 12052 7132656 12052 0 0 *Sched*
251 0 4660 0 11864 0 0 trunk conditioni
197 0 2260 568 10896 0 0 Crypto WUI
244 0 868 0 10460 0 0 EEM ED Resource
124 0 224 0 10428 0 0 IP RIB Update
5 0 3352 252 10364 0 0 Check heaps
123 0 76 0 10280 0 0 IP Background
235 0 0 0 10204 0 0 Crypto Device Up
245 0 612 0 10204 0 0 EEM ED Track
150 0 0 0 10204 0 0 CES Client SVC R
257 0 612 0 10204 0 0 EEM ED Interface
98 0 252 252 10204 0 0 Ethernet OAM Pro
91 0 252 252 10204 0 0 Dot11 auth Dot1x
90 0 0 0 10204 0 0 pmkid
89 0 252 252 10204 0 0 Dot11 aaa proces
43 0 0 0 10204 0 0 Per-Second Jobs
258 0 612 0 10204 0 0 EEM ED IOSWD
259 0 612 0 10204 0 0 EEM ED None
260 0 612 0 10204 0 0 EEM ED OIR
261 0 1224 0 10204 0 0 EEM ED SNMP
262 0 3060 0 10204 0 0 EEM ED Timer
153 0 252 252 10204 0 0 Adj Manager
255 0 612 0 10204 0 0 EEM ED CLI
119 0 252 252 10204 0 0 IP Host Track Pr
195 0 1422732 838072 9980 0 0 TPLUS
82 0 2500 252 9452 0 0 DTP Protocol
PID TTY Allocated Freed Holding Getbufs Retbufs Process
219 0 0 0 9204 0 0 Crypto PKI-CRL
231 0 0 0 9204 0 0 Key Proc
220 0 0 0 9204 0 0 Crypto SSL
3 0 2100 0 9168 0 0 gk dns process
100 0 1072 0 8276 0 0 IP ARP Adjacency
181 0 1192 252 8144 0 0 URL filter proc
16 0 744 0 7948 0 0 IPC Seat Manager
138 0 732 160 7936 0 0 L2MM
278 0 1044 504 7744 0 0 NTP
203 0 436 0 7640 0 0 CCVPM_R2
210 0 336 0 7540 0 0 CCSWVOICE
280 0 10256 1484 7492 0 0 AAA Accounting
268 0 224 0 7476 0 0 CEF Scanner
114 0 504 252 7456 0 0 Spanning Tree
135 0 504 252 7456 0 0 Dot1x Supplicant
137 0 504 252 7456 0 0 Dot1x Supplicant
136 0 504 252 7456 0 0 Dot1x Supplicant
242 0 224 0 7428 0 0 DATA Collector
241 0 224 0 7428 0 0 DATA Transfer Pr
176 0 388 252 7340 0 0 SDEE Management
36 0 252 0 7320 0 0 IGMP Input
213 0 228 0 7296 0 0 DHCPD Database
144 0 252 252 7276 0 0 PPPoE Flow Contr
204 0 0 0 7276 0 0 EPHONE MWI Refre
128 0 0 0 7220 0 0 SNMP Timers
105 0 252 252 7204 0 0 TurboACL chunk
106 0 0 0 7204 0 0 IPv6 RIB Redistr
103 0 0 0 7204 0 0 ICMP event handl
139 0 0 0 7204 0 0 MRD
140 0 0 0 7204 0 0 IGMPSN
141 0 252 252 7204 0 0 RLM groups Proce
96 0 252 252 7204 0 0 AAA Dictionary R
94 0 0 0 7204 0 0 AAA ACCT Proc
93 0 252 252 7204 0 0 AAA Server
146 0 0 0 7204 0 0 IUA Main Process
147 0 252 252 7204 0 0 RUDPV1 Main Proc
148 0 0 0 7204 0 0 bsm_timers
149 0 0 0 7204 0 0 bsm_xmt_proc
92 0 0 0 7204 0 0 Dot11 Mac Auth
151 0 252 252 7204 0 0 Dialer Forwarder
152 0 0 0 7204 0 0 IP Cache Ager
88 0 252 252 7204 0 0 Dot11 Mgmt & Ass
154 0 252 252 7204 0 0 Flow Exporter Ti
87 0 0 0 7204 0 0 linktest
84 0 252 252 7204 0 0 EtherChnl
83 0 252 252 7204 0 0 PI MATM Aging Pr
158 0 0 0 7204 0 0 RARP Input
159 0 0 0 7204 0 0 Transport Port A
160 0 0 0 7204 0 0 IPv6 Inspect Tim
161 0 0 0 7204 0 0 PAD InCall
131 0 0 0 7204 0 0 Socket Timers
163 0 252 252 7204 0 0 PPP Bind
164 0 252 252 7204 0 0 PPP SSS
166 0 252 252 7204 0 0 RBSCP Background
107 0 252 252 7204 0 0 MOP Protocols
129 0 0 0 7204 0 0 Asy FS Helper
104 0 252 252 7204 0 0 TurboACL
127 0 0 0 7204 0 0 IP Traceroute
171 0 0 0 7204 0 0 Auth-proxy AAA B
PID TTY Allocated Freed Holding Getbufs Retbufs Process
172 0 0 0 7204 0 0 CHKPT EXAMPLE
173 0 0 0 7204 0 0 CHKPT DevTest
77 0 0 0 7204 0 0 CES Line Conditi
76 0 252 252 7204 0 0 Call Management
112 0 0 0 7204 0 0 SSS Feature Mana
177 0 0 0 7204 0 0 Select Timers
75 0 0 0 7204 0 0 IGMP Snooping Re
179 0 252 252 7204 0 0 CIFS API Process
74 0 0 0 7204 0 0 IGMP Snooping Pr
72 0 252 252 7204 0 0 Ether-Switch RBC
70 0 0 0 7204 0 0 RF_INTERDEV_DELA
69 0 252 252 7204 0 0 VMI Background
184 0 7632 252 7204 0 0 FLEX DSPRM MAIN
185 0 0 0 7204 0 0 FLEX DSP KEEPALI
186 0 252 252 7204 0 0 CRM_CALL_UPDATE_
187 0 252 252 7204 0 0 HDA DSPRM MAIN
113 0 0 0 7204 0 0 SSS Feature Time
189 0 252 252 7204 0 0 AAA Cached Serve
190 0 252 252 7204 0 0 ENABLE AAA
191 0 0 0 7204 0 0 EM Background Pr
192 0 0 0 7204 0 0 Key chain liveke
193 0 252 252 7204 0 0 LINE AAA
194 0 252 252 7204 0 0 LOCAL AAA
67 0 252 252 7204 0 0 Bryce I2C CMD Qu
196 0 252 252 7204 0 0 VSP_MGR
66 0 0 0 7204 0 0 HDV background
198 0 252 252 7204 0 0 Crypto Support
199 0 0 0 7204 0 0 IPSECv6 PS Proc
115 0 0 0 7204 0 0 X.25 Encaps Mana
65 0 252 252 7204 0 0 FLEX DNLD MAIN
202 0 252 252 7204 0 0 VPM_MWI_BACKGROU
81 0 0 0 7204 0 0 EAP Framework
165 0 0 0 7204 0 0 MQC Flow Event B
205 0 0 0 7204 0 0 FB/KS Log HouseK
206 0 252 252 7204 0 0 EPHONE MWI BG Pr
64 0 0 0 7204 0 0 DSPFARM DSP READ
208 0 252 252 7204 0 0 VOICE REG BG Pro
209 0 0 0 7204 0 0 Presence Process
117 0 0 0 7204 0 0 AC Switch
62 0 252 252 7204 0 0 SM Monitor
61 0 252 252 7204 0 0 Netclock Backgro
80 0 0 0 7204 0 0 MAB Framework
168 0 252 252 7204 0 0 Inspect process
59 0 252 252 7204 0 0 ESWILPPM
170 0 0 0 7204 0 0 Authentication P
57 0 252 252 7204 0 0 ESWPPM
56 0 0 0 7204 0 0 ARL Table Manage
200 0 0 0 7204 0 0 EPM MAIN PROCESS
249 0 0 0 7204 0 0 Crypto cTCP proc
51 0 0 0 7204 0 0 Token Daemon
222 0 0 0 7204 0 0 Crypto INT
50 0 0 0 7204 0 0 AggMgr Process
49 0 0 600 7204 0 0 Per-minute Jobs
48 0 252 252 7204 0 0 Compute load avg
226 0 0 0 7204 0 0 IPSEC manual key
47 0 0 0 7204 0 0 Net Input
228 0 0 0 7204 0 0 CRYPTO QoS proce
45 0 0 0 7204 0 0 IPSEC HA Mgr
230 0 0 0 7204 0 0 Crypto PAS Proc
PID TTY Allocated Freed Holding Getbufs Retbufs Process
44 0 0 0 7204 0 0 IKE HA Mgr
232 0 0 0 7204 0 0 GDOI GM Process
233 0 0 0 7204 0 0 UNICAST REKEY
234 0 0 0 7204 0 0 UNICAST REKEY AC
42 0 252 1312 7204 0 0 TTY Background
236 0 0 0 7204 0 0 Multi-ISA Event
237 0 0 0 7204 0 0 Multi-ISA Cleanu
238 0 252 252 7204 0 0 Control-plane ho
239 0 0 0 7204 0 0 PM Callback
240 0 0 0 7204 0 0 Atheros LED Ctro
6 0 52076 262844 7204 10660 98404 Pool Manager
7 0 252 252 7204 0 0 Timers
243 0 51656 283184 7204 0 0 AAA SEND STOP EV
38 0 0 0 7204 0 0 Critical Bkgnd
37 0 0 0 7204 0 0 Inode Table Dest
246 0 252 252 7204 0 0 RMON Recycle Pro
247 0 252 252 7204 0 0 RMON Deferred Se
10 0 252 252 7204 0 0 Environmental mo
12 0 0 0 7204 0 0 IPC Dynamic Cach
250 0 0 0 7204 0 0 trunk conditioni
34 0 0 0 7204 0 0 SERIAL A'detect
32 0 252 252 7204 0 0 GraphIt
31 0 252 252 7204 0 0 SMART
13 0 0 0 7204 0 0 IPC Zone Manager
29 0 0 0 7204 0 0 RO Notify Timers
28 0 252 252 7204 0 0 Serial Backgroun
27 0 0 0 7204 0 0 HC Counter Timer
24 0 252 252 7204 0 0 DDR Timers
22 0 0 0 7204 0 0 AAA_SERVER_DEADT
21 0 252 252 7204 0 0 AAA high-capacit
20 0 252 252 7204 0 0 ATM Idle Timer
19 0 6348 2476 7204 780 0 ARP Background
17 0 0 0 7204 0 0 IPC BackPressure
265 0 0 0 7204 0 0 IP SLA MPLSLM Pr
266 0 0 0 7204 0 0 VPDN Test
15 0 0 0 7204 0 0 IPC Deferred Por
269 0 0 2500 7204 0 0 crypto sw pk pro
14 0 0 0 7204 0 0 IPC Periodic Tim
188 0 0 0 7068 0 0 cpf_process_msg_
109 0 32016 0 7068 0 0 TACACS+
11 0 0 0 7068 0 0 chkpt message ha
169 0 0 0 7068 0 0 DHCPD Timer
30 0 0 0 4204 0 0 RMI RM Notify Wa
60 0 252 252 4204 0 0 Eswilp Storm Con
58 0 252 252 4204 0 0 Eswilp Storm Con
2 0 252 252 4204 0 0 Load Meter
53 0 0 0 4204 0 0 dev_device_remov
52 0 0 0 4204 0 0 dev_device_inser
37611736 Total
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: