08-14-2012 11:43 PM - edited 03-04-2019 05:16 PM
Hi There,
We're hitting about 95% memory utilization and it appears most of it is being consume by *Dead* processes (see below).
Is there some way to recover the memory being used up by the *Dead* processes or will we have to reboot the router?
RTR-01#sh proc memory sorted
Processor Pool Total: 133318240 Used: 127600908 Free: 5717332
I/O Pool Total: 40893952 Used: 13781152 Free: 27112800
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 151244900 52146784 99991348 554104 0 *Dead*
0 0 63691228 22433984 36948272 0 0 *Init*
139 0 129059308 128081536 932496 24948 0 HTTP CORE
192 0 304768 34984 275612 0 0 IPSEC key engine
188 0 115776 3764 137384 0 0 Crypto ACL
133 0 136472 196 135296 0 0 DHCPD Receive
170 0 118884 472 127392 0 0 Crypto WUI
181 0 88248 0 95228 0 0 QOS_MODULE_MAIN
4 0 65588 0 90568 0 0 EDDRI_MAIN
113 0 67384 0 77364 0 0 IP RIB Update
116 0 73340 0 73340 0 0 CEF process
179 0 940 0 61920 0 0 http client proc
157 0 44876 196 51660 0 0 CCVPM_HDSPRM
74 450 116092 82560 46124 0 0 SSH Process
195 0 24912 0 37892 0 0 Crypto Delete Ma
172 0 26180 0 36160 0 0 CCVPM_HTSP
36 0 3799060 780 35744 5388 0 Net Background
6 0 32526052 46627708 35684 25629576 32805804 Pool Manager
60 0 27448 196 34232 0 0 VNM DSPRM MAIN
RTR-01#sh ver
Cisco IOS Software, 3800 Software (C3845-ADVIPSERVICESK9-M), Version 12.4(3e), RELEASE SOFTWARE (fc2)
Technical Support:
http://www.cisco.com/techsupport
Copyright (c) 1986-2006 by Cisco Systems, Inc.
Compiled Wed 14-Jun-06 01:49 by alnguyen
ROM: System Bootstrap, Version 12.3(11r)T2, RELEASE SOFTWARE (fc1)
RTR-01 uptime is 20 weeks, 1 day, 18 hours, 26 minutes
System returned to ROM by power-on
System restarted at 17:17:23 AEDT Mon Mar 26 2012
System image file is "flash:c3845-advipservicesk9-mz.124-3e.bin"
Thanks.
Andy
08-14-2012 11:53 PM
It appears to be a series of dead SSH sessions.
RTR-01#show memory dead
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 659DB9A0 133318240 127629396 5688844 5112068 5543780
I/O 2D900000 40893952 13801756 27092196 26974320 27053116
Processor memory
Address Bytes Prev Next Ref PrevF NextF Alloc PC what
659DB9A0 0000020004 00000000 659E07F4 001 -------- -------- 60D87F30 Acct Req chunk
659F47BC 0000004100 659F2CDC 659F57F0 001 -------- -------- 6173BB94 SSH Process
659F57F0 0000004100 659F47BC 659F6824 001 -------- -------- 6173BB94 SSH Process
659F6824 0000004100 659F57F0 659F7858 001 -------- -------- 6173BB94 SSH Process
659F7858 0000004100 659F6824 659F888C 001 -------- -------- 6173BB94 SSH Process
659F888C 0000004100 659F7858 659F98C0 001 -------- -------- 6173BB94 SSH Process
659F98C0 0000004100 659F888C 659FA8F4 001 -------- -------- 6173BB94 SSH Process
659FA8F4 0000004100 659F98C0 659FB928 001 -------- -------- 6173BB94 SSH Process
659FB928 0000004100 659FA8F4 659FC95C 001 -------- -------- 6173BB94 SSH Process
659FC95C 0000004100 659FB928 659FD990 001 -------- -------- 6173BB94 SSH Process
659FD990 0000004100 659FC95C 659FE9C4 001 -------- -------- 6173BB94 SSH Process
659FE9C4 0000004100 659FD990 659FF9F8 001 -------- -------- 6173BB94 SSH Process
659FF9F8 0000004100 659FE9C4 65A00A2C 001 -------- -------- 6173BB94 SSH Process
65A00A2C 0000004100 659FF9F8 65A01A60 001 -------- -------- 6173BB94 SSH Process
65A01A60 0000004100 65A00A2C 65A02A94 001 -------- -------- 6173BB94 SSH Process
65A037BC 0000004100 65A02A94 65A047F0 001 -------- -------- 6173BB94 SSH Process
65A047F0 0000004100 65A037BC 65A05824 001 -------- -------- 6173BB94 SSH Process
65A05824 0000004100 65A047F0 65A06858 001 -------- -------- 6173BB94 SSH Process
65A06BD4 0000004100 65A06858 65A07C08 001 -------- -------- 6173BB94 SSH Process
65A07C08 0000004100 65A06BD4 65A08C3C 001 -------- -------- 6173BB94 SSH Process
65A08C3C 0000004100 65A07C08 65A09C70 001 -------- -------- 6173BB94 SSH Process
65A09C70 0000004100 65A08C3C 65A0ACA4 001 -------- -------- 6173BB94 SSH Process
65A0ACA4 0000004100 65A09C70 65A0BCD8 001 -------- -------- 6173BB94 SSH Process
65A0C74C 0000004100 65A0BCD8 65A0D780 001 -------- -------- 6173BB94 SSH Process
65A0D780 0000004100 65A0C74C 65A0E7B4 001 -------- -------- 6173BB94 SSH Process
65A0E7B4 0000004100 65A0D780 65A0F7E8 001 -------- -------- 6173BB94 SSH Process
65A0F7E8 0000004100 65A0E7B4 65A1081C 001 -------- -------- 6173BB94 SSH Process
65A1081C 0000004100 65A0F7E8 65A11850 001 -------- -------- 6173BB94 SSH Process
65A11850 0000004100 65A1081C 65A12884 001 -------- -------- 6173BB94 SSH Process
65A12884 0000004100 65A11850 65A138B8 001 -------- -------- 6173BB94 SSH Process
65A138B8 0000004100 65A12884 65A148EC 001 -------- -------- 6173BB94 SSH Process
65A148EC 0000004100 65A138B8 65A15920 001 -------- -------- 6173BB94 SSH Process
65A15920 0000004100 65A148EC 65A16954 001 -------- -------- 6173BB94 SSH Process
65A16BF4 0000000172 65A16954 65A16CD0 001 -------- -------- 612F21FC SSH Process
08-14-2012 11:55 PM
You're running very old software, update.
08-15-2012 02:06 AM
Hello Andy,
Paolo is completely correct, your IOS is strongly outdated. You should consider an upgrade. Regarding the taken memory, I am afraid the only way to reclaim it is to reload the device.
Best regards,
Peter
08-16-2012 04:25 PM
Thanks for the replies guy.
Will look at the IOS upgrade option.
10-12-2012 01:18 AM
Hi Andy,
We have got same problem.
The device is also 3800.
We tried it iwith following IOS.
c3845-advipservicesk9-mz.124-24.T3.bin
c3845-advipservicesk9-mz.151-4.M4.bin
In both case the memory is decreasing continuously and there are many SSH dead process.
Our questions is have you found any "good" IOS which does not consume the memory?
Thanks in advance!
10-12-2012 02:00 AM
There is always the possibility of bugs. With these, you have to contact the TAC.
Check if someone isn't attempting many ssh accesses without you knowing.
10-12-2012 02:24 AM
Hi,
You right. It may be a bug but the question is where is the bug.
But i thought other CISCO costumer had bumbed into this problem also and had already opened TAC.
The SSH access is restricted by ACL.
Only the LMS manages the router via SSH periodically.
I think the LMS SSH process causes the DEAD memory problems.
I have not got any clue for this because the router does not show what kind of SSH source address were there.
I have same problem on ACE module. There the LMS SSH session are still exists and not disconnected.
Regards,
05-31-2017 08:44 AM
Could you be guarding (via ACL) against SSH from the Internet, but allowing access to a wider group of internal addresses than intended. Could somebody be half-opening many SSH sessions programmatically, doing a sort of SSH DDOS from an internal host?
In general, could this be memory that IOS is not properly reclaiming when an SSH process times out, rather than when the user quits normally?
I see that Paolo had already posted a comment somewhat along that DDOS angle.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: