Rebooting GK monthly to avoid memory leak issue causing GK non functional

kumar-brajesh · ‎04-29-2010

We are running gatekeeper IOS version 12.3(11r)T2 where we have to reboot GK monthly to avoid memory leak issue causing GK to be nonfunctional after some time.

It was recommended to have monthly reboots to avoid the memory leaks. It has worked since it has avoided any additional memory leak. The reboots were recommended by Cisco to avoid the memory leak.

Does any one know if there is fix for this now or if any new IOS version has a fix.

Any help will be appreciated.

Regards,

Brajesh.

paolo bevilacqua · ‎05-03-2010

I think that this question should go to Cisco and one should not accept reload based workarounds.

Felipe Garrido · ‎05-17-2010

Brajesh,

12.3(11r)T2 sounds like the IOS bootloader version. Can you copy the output of "show version." I can do a quick search to determine if there are any know leaks. Do you know which process actually leaks? I agree that you may want to open a TAC SR to start profiling memory utilization to determine the cause of the leak.

To view the current memory allocation use,

show process memory sorted

-Felipe

kumar-brajesh · ‎05-17-2010

Felipe,

Here is show version.

vus-cvpgk1#sh ver
Cisco IOS Software, 3800 Software (C3825-ADVENTERPRISEK9_IVS-M), Version 12.4(15
)T7, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Wed 13-Aug-08 23:16 by prod_rel_team

ROM: System Bootstrap, Version 12.3(11r)T2, RELEASE SOFTWARE (fc1)

vus-cvpgk1 uptime is 2 weeks, 4 hours, 7 minutes
System returned to ROM by reload at 23:28:25 GMT Mon May 3 2010
System restarted at 23:30:02 GMT Mon May 3 2010
System image file is "flash:c3825-adventerprisek9_ivs-mz.124-15.T7.bin"

Cisco 3825 (revision 1.1) with 1011712K/36864K bytes of memory.
Processor board ID FTX1025A308
2 Gigabit Ethernet interfaces
1 Virtual Private Network (VPN) Module
DRAM configuration is 64 bits wide with parity enabled.
479K bytes of NVRAM.
1930760K bytes of ATA System CompactFlash (Read/Write)

Configuration register is 0x2102

=========================

The below bug seems to be close to our scenario. Please have a look.

CSCsw23397

1st Found-In
12.4T
12.4(15)T7
12.4(15)T8

Voice Gateway may leak memory for DSMP EVENT chu and DSMP DSP REQ chunk
Symptoms: A Cisco Router acting as a voice Gateway may leak memory for DSMP DSP REQ chunk and DSMP DSP REQ chunk

Conditions: The symptom appears to be triggered by calls that disconnect prematurely.

Fixed-In
12.4(22)MD1
12.4(20)T2
12.4(23.12)M
12.4(22)T1
12.4(23.12)T
12.4(23.12)PI10
12.4(23.15.1)PIX11
12.4(22)MDA
12.4(23a)M
12.4(23.15.4)PIC1
12.4(24.5.1)PIX11
12.4(15)T9
12.4(24.5.2)PIC1
12.4(22)XR
12.4(22)YE2

Felipe Garrido · ‎05-18-2010

I wouldn't expect that to be the bug unless the gatekeeper gateway also has TDM/analog interface handling calls. Do you have any information on the process that's leaking? Can you get the output of "show proc mem sorted" or provide the previous TAC SR number opened to originally address the issue?

kumar-brajesh · ‎05-18-2010

Earlier resolution what we found from Cisco is to reboot GK to fix this issue.

Please find the command output.

---------------------------

Processor Pool Total: 919148940 Used: 24673240 Free: 894475700

I/O Pool Total: 37748224 Used: 12959664 Free: 24788560

PID TTY Allocated Freed Holding Getbufs Retbufs Process

0 0 64213880 24742544 31780752 0 0 *Init*

68 0 659772 1328 640444 0 0 USB Startup

122 0 3760476856 3760018464 470240 0 0 gk process

118 0 365608 1052 367364 0 0 EAPoUDP Process

26 0 264500 0 274000 113400 0 EEM ED Syslog

214 0 247992 0 255196 0 0 QOS_MODULE_MAIN

254 0 161220 10404 167812 0 0 EEM Server

1 0 112392 13968 119596 0 0 Chunk Manager

0 0 61174996 63634536 104152 258728 0 *Dead*

4 0 65588 0 90792 0 0 EDDRI_MAIN

225 0 66852 0 80056 0 0 Crypto IKEv2

130 0 74304 0 74256 0 0 CEF process

248 0 75212 9024 73392 0 0 Syslog Traps

178 0 66416 252 73368 0 0 HTTP Process

229 0 45288 3204 67288 0 0 Crypto ACL

63 0 57904 252 64856 0 0 VNM DSPRM MAIN

212 0 940 0 62144 0 0 http client proc

95 0 49456 0 56660 0 0 ACCT Periodic Pr

101 0 49444 0 56648 0 0 IP ARP Retry Age

183 0 32136 252 39088 0 0 CCVPM_HDSPRM

143 0 39244 252 38060 0 0 DHCPD Receive

201 0 26200 0 36404 0 0 CCVPM_HTSP

134 0 10052 0 35256 0 0 COPS

102 0 234876 144492 34628 0 0 IP Input

271 0 17980 0 31252 0 0 IP SLAs Responde

46 0 45512 800 30712 0 0 rf task

175 0 16812 252 29764 0 0 IPS Auto Update

270 578 4164412 4144008 27816 0 0 Virtual Exec

18 0 22452 2400 27256 0 0 ARP Input

121 0 1068 252 26020 0 0 Border Element p

145 0 18944 252 25896 0 0 SCTP Main Proces

215 0 284 0 25488 0 0 RPMS_PROC_MAIN

267 0 252 0 25456 0 0 tHUB

71 0 0 0 25204 0 0 RF_INTERDEV_SCTP

9 0 0 0 25204 0 0 Crash writer

78 0 0 0 25204 0 0 CF_INTERDEV_SCTP

216 0 0 0 25204 0 0 VoIP AAA

224 0 9816 252 22768 0 0 Crypto IKMP

180 0 12592 252 19544 0 0 CIFS Proxy Proce

54 0 11688 0 18892 0 0 mxt5100

182 0 10556 504 17256 0 0 Crypto HW Proc

211 0 10052 0 17120 0 0 cpf_process_tpQ

217 0 31220 11892 16828 0 0 crypto engine pr

97 0 9380 252 16332 0 0 CDP Protocol

263 0 10020 912 16312 0 0 EEM Policy Direc

207 0 8244 0 15448 0 0 Skinny HW confer

264 0 11856 9888 15172 0 0 Syslog

39 0 24152 968 15024 0 0 Net Background

252 0 12388 4504 14848 0 0 VLAN Manager

218 0 2428 832 14800 0 0 Crypto CA

157 0 3972 0 14176 0 0 HTTP CORE

272 0 3212352 3211452 14172 0 0 IP SNMP

275 0 1152 252 14172 0 0 IP SNMPV6

277 0 25868 9248 13856 6804 0 SNMP Traps

40 0 1400 1096 13508 0 0 IDB Work

227 0 13872 12640 13372 0 0 IPSEC key engine

PID TTY Allocated Freed Holding Getbufs Retbufs Process

273 0 582992 0 13272 0 0 PDU DISPATCHER

274 0 16630556 17213548 13272 0 0 SNMP ENGINE

276 0 0 0 13256 0 0 SNMP ConfCopyPro

55 0 0 0 13204 0 0 sal_dpc_process

35 0 252 252 13204 0 0 XML Proxy Client

116 0 252 252 13204 0 0 SSM connection m

41 0 252 252 13204 0 0 Logger

221 0 0 0 13204 0 0 encrypt proc

33 0 252 252 13204 0 0 Dialer event

223 0 252 1188 13204 0 0 Crypto IKE Dispa

23 0 0 0 13204 0 0 Policy Manager

120 0 252 252 13204 0 0 KRB5 AAA

108 0 504 504 13204 0 0 PPP Hooks

167 0 0 0 13204 0 0 VPDN call manage

126 0 316 316 13204 0 0 PPP IPCP

142 0 0 0 13204 0 0 L2X Data Daemon

132 0 0 1060728 13204 0 0 TCP Timer

133 0 5281532 9604 13204 0 0 TCP Protocols

99 0 252 252 13204 0 0 Ethernet LMI

73 0 0 0 13204 0 0 AAL2CPS TIMER_CU

125 0 252 252 13204 0 0 PPP IP Route

111 0 0 0 13204 0 0 SSS Test Client

110 0 0 0 13204 0 0 SSS Manager

174 0 0 0 13204 0 0 IPS Process

8 0 0 0 13204 0 0 OIR Handler

155 0 252 252 13204 0 0 ATM OAM Input

86 0 252 252 13204 0 0 Ethernet Timer C

162 0 252 252 13204 0 0 X.25 Background

85 0 252 252 13204 0 0 Ethernet CFM

156 0 252 252 13204 0 0 ATM OAM TIMER

25 0 5336 0 12540 0 0 Entity MIB API

256 0 2664 0 12256 0 0 EEM ED Counter

79 0 5304 252 12256 0 0 Dot1x Mgr Proces

0 0 12052 7132656 12052 0 0 *Sched*

251 0 4660 0 11864 0 0 trunk conditioni

197 0 2260 568 10896 0 0 Crypto WUI

244 0 868 0 10460 0 0 EEM ED Resource

124 0 224 0 10428 0 0 IP RIB Update

5 0 3352 252 10364 0 0 Check heaps

123 0 76 0 10280 0 0 IP Background

235 0 0 0 10204 0 0 Crypto Device Up

245 0 612 0 10204 0 0 EEM ED Track

150 0 0 0 10204 0 0 CES Client SVC R

257 0 612 0 10204 0 0 EEM ED Interface

98 0 252 252 10204 0 0 Ethernet OAM Pro

91 0 252 252 10204 0 0 Dot11 auth Dot1x

90 0 0 0 10204 0 0 pmkid

89 0 252 252 10204 0 0 Dot11 aaa proces

43 0 0 0 10204 0 0 Per-Second Jobs

258 0 612 0 10204 0 0 EEM ED IOSWD

259 0 612 0 10204 0 0 EEM ED None

260 0 612 0 10204 0 0 EEM ED OIR

261 0 1224 0 10204 0 0 EEM ED SNMP

262 0 3060 0 10204 0 0 EEM ED Timer

153 0 252 252 10204 0 0 Adj Manager

255 0 612 0 10204 0 0 EEM ED CLI

119 0 252 252 10204 0 0 IP Host Track Pr

195 0 1422732 838072 9980 0 0 TPLUS

82 0 2500 252 9452 0 0 DTP Protocol

PID TTY Allocated Freed Holding Getbufs Retbufs Process

219 0 0 0 9204 0 0 Crypto PKI-CRL

231 0 0 0 9204 0 0 Key Proc

220 0 0 0 9204 0 0 Crypto SSL

3 0 2100 0 9168 0 0 gk dns process

100 0 1072 0 8276 0 0 IP ARP Adjacency

181 0 1192 252 8144 0 0 URL filter proc

16 0 744 0 7948 0 0 IPC Seat Manager

138 0 732 160 7936 0 0 L2MM

278 0 1044 504 7744 0 0 NTP

203 0 436 0 7640 0 0 CCVPM_R2

210 0 336 0 7540 0 0 CCSWVOICE

280 0 10256 1484 7492 0 0 AAA Accounting

268 0 224 0 7476 0 0 CEF Scanner

114 0 504 252 7456 0 0 Spanning Tree

135 0 504 252 7456 0 0 Dot1x Supplicant

137 0 504 252 7456 0 0 Dot1x Supplicant

136 0 504 252 7456 0 0 Dot1x Supplicant

242 0 224 0 7428 0 0 DATA Collector

241 0 224 0 7428 0 0 DATA Transfer Pr

176 0 388 252 7340 0 0 SDEE Management

36 0 252 0 7320 0 0 IGMP Input

213 0 228 0 7296 0 0 DHCPD Database

144 0 252 252 7276 0 0 PPPoE Flow Contr

204 0 0 0 7276 0 0 EPHONE MWI Refre

128 0 0 0 7220 0 0 SNMP Timers

105 0 252 252 7204 0 0 TurboACL chunk

106 0 0 0 7204 0 0 IPv6 RIB Redistr

103 0 0 0 7204 0 0 ICMP event handl

139 0 0 0 7204 0 0 MRD

140 0 0 0 7204 0 0 IGMPSN

141 0 252 252 7204 0 0 RLM groups Proce

96 0 252 252 7204 0 0 AAA Dictionary R

94 0 0 0 7204 0 0 AAA ACCT Proc

93 0 252 252 7204 0 0 AAA Server

146 0 0 0 7204 0 0 IUA Main Process

147 0 252 252 7204 0 0 RUDPV1 Main Proc

148 0 0 0 7204 0 0 bsm_timers

149 0 0 0 7204 0 0 bsm_xmt_proc

92 0 0 0 7204 0 0 Dot11 Mac Auth

151 0 252 252 7204 0 0 Dialer Forwarder

152 0 0 0 7204 0 0 IP Cache Ager

88 0 252 252 7204 0 0 Dot11 Mgmt & Ass

154 0 252 252 7204 0 0 Flow Exporter Ti

87 0 0 0 7204 0 0 linktest

84 0 252 252 7204 0 0 EtherChnl

83 0 252 252 7204 0 0 PI MATM Aging Pr

158 0 0 0 7204 0 0 RARP Input

159 0 0 0 7204 0 0 Transport Port A

160 0 0 0 7204 0 0 IPv6 Inspect Tim

161 0 0 0 7204 0 0 PAD InCall

131 0 0 0 7204 0 0 Socket Timers

163 0 252 252 7204 0 0 PPP Bind

164 0 252 252 7204 0 0 PPP SSS

166 0 252 252 7204 0 0 RBSCP Background

107 0 252 252 7204 0 0 MOP Protocols

129 0 0 0 7204 0 0 Asy FS Helper

104 0 252 252 7204 0 0 TurboACL

127 0 0 0 7204 0 0 IP Traceroute

171 0 0 0 7204 0 0 Auth-proxy AAA B

PID TTY Allocated Freed Holding Getbufs Retbufs Process

172 0 0 0 7204 0 0 CHKPT EXAMPLE

173 0 0 0 7204 0 0 CHKPT DevTest

77 0 0 0 7204 0 0 CES Line Conditi

76 0 252 252 7204 0 0 Call Management

112 0 0 0 7204 0 0 SSS Feature Mana

177 0 0 0 7204 0 0 Select Timers

75 0 0 0 7204 0 0 IGMP Snooping Re

179 0 252 252 7204 0 0 CIFS API Process

74 0 0 0 7204 0 0 IGMP Snooping Pr

72 0 252 252 7204 0 0 Ether-Switch RBC

70 0 0 0 7204 0 0 RF_INTERDEV_DELA

69 0 252 252 7204 0 0 VMI Background

184 0 7632 252 7204 0 0 FLEX DSPRM MAIN

185 0 0 0 7204 0 0 FLEX DSP KEEPALI

186 0 252 252 7204 0 0 CRM_CALL_UPDATE_

187 0 252 252 7204 0 0 HDA DSPRM MAIN

113 0 0 0 7204 0 0 SSS Feature Time

189 0 252 252 7204 0 0 AAA Cached Serve

190 0 252 252 7204 0 0 ENABLE AAA

191 0 0 0 7204 0 0 EM Background Pr

192 0 0 0 7204 0 0 Key chain liveke

193 0 252 252 7204 0 0 LINE AAA

194 0 252 252 7204 0 0 LOCAL AAA

67 0 252 252 7204 0 0 Bryce I2C CMD Qu

196 0 252 252 7204 0 0 VSP_MGR

66 0 0 0 7204 0 0 HDV background

198 0 252 252 7204 0 0 Crypto Support

199 0 0 0 7204 0 0 IPSECv6 PS Proc

115 0 0 0 7204 0 0 X.25 Encaps Mana

65 0 252 252 7204 0 0 FLEX DNLD MAIN

202 0 252 252 7204 0 0 VPM_MWI_BACKGROU

81 0 0 0 7204 0 0 EAP Framework

165 0 0 0 7204 0 0 MQC Flow Event B

205 0 0 0 7204 0 0 FB/KS Log HouseK

206 0 252 252 7204 0 0 EPHONE MWI BG Pr

64 0 0 0 7204 0 0 DSPFARM DSP READ

208 0 252 252 7204 0 0 VOICE REG BG Pro

209 0 0 0 7204 0 0 Presence Process

117 0 0 0 7204 0 0 AC Switch

62 0 252 252 7204 0 0 SM Monitor

61 0 252 252 7204 0 0 Netclock Backgro

80 0 0 0 7204 0 0 MAB Framework

168 0 252 252 7204 0 0 Inspect process

59 0 252 252 7204 0 0 ESWILPPM

170 0 0 0 7204 0 0 Authentication P

57 0 252 252 7204 0 0 ESWPPM

56 0 0 0 7204 0 0 ARL Table Manage

200 0 0 0 7204 0 0 EPM MAIN PROCESS

249 0 0 0 7204 0 0 Crypto cTCP proc

51 0 0 0 7204 0 0 Token Daemon

222 0 0 0 7204 0 0 Crypto INT

50 0 0 0 7204 0 0 AggMgr Process

49 0 0 600 7204 0 0 Per-minute Jobs

48 0 252 252 7204 0 0 Compute load avg

226 0 0 0 7204 0 0 IPSEC manual key

47 0 0 0 7204 0 0 Net Input

228 0 0 0 7204 0 0 CRYPTO QoS proce

45 0 0 0 7204 0 0 IPSEC HA Mgr

230 0 0 0 7204 0 0 Crypto PAS Proc

PID TTY Allocated Freed Holding Getbufs Retbufs Process

44 0 0 0 7204 0 0 IKE HA Mgr

232 0 0 0 7204 0 0 GDOI GM Process

233 0 0 0 7204 0 0 UNICAST REKEY

234 0 0 0 7204 0 0 UNICAST REKEY AC

42 0 252 1312 7204 0 0 TTY Background

236 0 0 0 7204 0 0 Multi-ISA Event

237 0 0 0 7204 0 0 Multi-ISA Cleanu

238 0 252 252 7204 0 0 Control-plane ho

239 0 0 0 7204 0 0 PM Callback

240 0 0 0 7204 0 0 Atheros LED Ctro

6 0 52076 262844 7204 10660 98404 Pool Manager

7 0 252 252 7204 0 0 Timers

243 0 51656 283184 7204 0 0 AAA SEND STOP EV

38 0 0 0 7204 0 0 Critical Bkgnd

37 0 0 0 7204 0 0 Inode Table Dest

246 0 252 252 7204 0 0 RMON Recycle Pro

247 0 252 252 7204 0 0 RMON Deferred Se

10 0 252 252 7204 0 0 Environmental mo

12 0 0 0 7204 0 0 IPC Dynamic Cach

250 0 0 0 7204 0 0 trunk conditioni

34 0 0 0 7204 0 0 SERIAL A'detect

32 0 252 252 7204 0 0 GraphIt

31 0 252 252 7204 0 0 SMART

13 0 0 0 7204 0 0 IPC Zone Manager

29 0 0 0 7204 0 0 RO Notify Timers

28 0 252 252 7204 0 0 Serial Backgroun

27 0 0 0 7204 0 0 HC Counter Timer

24 0 252 252 7204 0 0 DDR Timers

22 0 0 0 7204 0 0 AAA_SERVER_DEADT

21 0 252 252 7204 0 0 AAA high-capacit

20 0 252 252 7204 0 0 ATM Idle Timer

19 0 6348 2476 7204 780 0 ARP Background

17 0 0 0 7204 0 0 IPC BackPressure

265 0 0 0 7204 0 0 IP SLA MPLSLM Pr

266 0 0 0 7204 0 0 VPDN Test

15 0 0 0 7204 0 0 IPC Deferred Por

269 0 0 2500 7204 0 0 crypto sw pk pro

14 0 0 0 7204 0 0 IPC Periodic Tim

188 0 0 0 7068 0 0 cpf_process_msg_

109 0 32016 0 7068 0 0 TACACS+

11 0 0 0 7068 0 0 chkpt message ha

169 0 0 0 7068 0 0 DHCPD Timer

30 0 0 0 4204 0 0 RMI RM Notify Wa

60 0 252 252 4204 0 0 Eswilp Storm Con

58 0 252 252 4204 0 0 Eswilp Storm Con

2 0 252 252 4204 0 0 Load Meter

53 0 0 0 4204 0 0 dev_device_remov

52 0 0 0 4204 0 0 dev_device_inser

37611736 Total