Re: SNMP engine process holding high memory in catalyst 3750

vinod.mali · ‎07-11-2013

Hi Guys,

I am facing this issue of high memory util in one of my cat 3750 switch, when going through the show proc memory I could see the snmp engine holding a lot of memory about 98 Mb.

------------------ show version ------------------

Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.0(1)SE3, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 30-May-12 13:41 by prod_rel_team

ROM: Bootstrap program is C3750E boot loader

BOOTLDR: C3750E Boot Loader (C3750X-HBOOT-M) Version 12.2(53r)SE2, RELEASE SOFTWARE (fc1)

uptime is 20 weeks, 1 day, 21 hours, 20 minutes

System returned to ROM by power-on

System image file is "flash:/c3750e-universalk9-mz.150-1.SE3/c3750e-uni

------------------ show memory statistics ------------------

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)

Processor 46E5A34 179092576 161806080 17286496 14954752 12574356

I/O E000000 16777216 13033408 3743808 2746280 3363372

Driver te 2C00000 4194304 44 4194260 4194260 4194260

------------------ show process memory ------------------

Processor Pool Total: 179092576 Used: 161788632 Free: 17303944

I/O Pool Total: 16777216 Used: 13033240 Free: 3743976

Driver te Pool Total: 4194304 Used: 40 Free: 4194264

PID TTY Allocated Freed Holding Getbufs Retbufs Process

0 0 93390008 31864848 56590176 0 0 *Init*

0 0 12424 3259937860 12424 0 0 *Sched*

0 0 2627251548 3709118784 4041676 14898831 570680 *Dead*

0 0 0 0 1181480 0 0 *MallocLite*

385 0 3658315444 3906459100 98144192 96444 0 SNMP ENGINE

configuration related to snmp on switch

snmp-server group snmpuser v3 auth notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF7F

snmp-server group network-admin v3 auth write v1default notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF0F access SNMP-ACL

snmp-server group network-admin v3 auth context vlan-10 notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF0F

snmp-server group network-admin v3 auth context vlan-20 notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF0F

snmp-server group network-admin v3 auth context vlan-30 notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF0F

snmp-server group network-admin v3 auth context vlan-40 notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF0F

snmp-server group network-admin v3 auth context vlan-50 notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF0F

snmp-server group network-admin v3 auth context vlan-60 notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF0F

snmp-server group network-admin v3 auth context vlan-70notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF0F

snmp-server group network-admin v3 auth context vlan-80 notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF0F

snmp-server group network-admin v3 auth context vlan-90 notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF0F

snmp-server group network-admin v3 auth context vlan-20 notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF0F

snmp-server trap-source Vlan10

snmp-server source-interface informs Vlan10

snmp-server enable traps energywise

snmp-server enable traps entity

snmp-server enable traps cpu threshold

snmp-server enable traps stpx inconsistency root-inconsistency loop-inconsistency

snmp-server enable traps vlancreate

snmp-server enable traps vlandelete

snmp-server enable traps envmon fan shutdown supply temperature status

snmp-server enable traps stackwise

snmp-server enable traps errdisable

snmp-server enable traps mac-notification change move threshold

snmp-server host 10.1.1.3 version 3 auth <removed>

For other switches of same model with the similar IOS and configuration I am not facing this issue.

Am not geeting to the root of this issue.......

Leo Laohoo · ‎07-11-2013

Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.0(1)SE3, RELEASE SOFTWARE (fc1)

If 15.0 is what you require, use either 15.0(2)SE2 or 15.0(2)SE4.

If you need to do SNMP monitoring, use the ones you only need. Do not enable ALL the SNMP traps.

sameermunj · ‎07-12-2013

Hi Leo

Do you mean that the version mentioned above (15.0(1)SE3) has any bug related to SNMP? Is it the reason that another stack with same configuration and same number of switches is utilizing 6MB memory for SNMP engine process wherein the stack in question is using 98Mb for same process.

If we disable the all the traps and stop Polling from the Management Software,will the utilization of this SNMP enginer process will go down drastically ? If still the utilization remains the same,can it be considered as Memory leak issue?

Regards

Sameer

Leo Laohoo · ‎07-12-2013

When you carelessly enable ALL SNMP traps, CPU spikes. Period. This gets worst in 15.0 IOS.

I am NOT saying that you disable all SNMP traps. Pick the ones you really need and disable the rest.

sameermunj · ‎07-15-2013

Hi Leo

Can you please clarify the same

If we stop polling or disable all the traps,will the SNMP enginer process will release the memory its holding currently.
If the process still holds the memory,can it be considered as Memory leak?

Thanks for your support

Regards

Sameer

Leo Laohoo · ‎07-15-2013

If we stop polling or disable all the traps,will the SNMP enginer process will release the memory its holding currently.

I'm not saying disabling ALL traps. What I am saying is don't use the command of "snmp-server enable traps ". The command "snmp-server enable traps " enables ALL SNMP traps.

You choose which SNMP traps are logically valid to your network. Take my network, these are the ones I have:

snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart

snmp-server enable traps transceiver all

snmp-server enable traps tty

snmp-server enable traps cpu threshold

snmp-server enable traps power-ethernet police

snmp-server enable traps envmon fan shutdown supply temperature status

snmp-server enable traps license

snmp-server enable traps energywise

sameermunj · ‎07-15-2013

Hi Leo

I understood your point here but my 2 questions in the previous posts are still unanswered ..

1-If we stop polling or disable all the traps,will the SNMP enginer process will release the memory its holding currently.

2-If the process still holds the memory,can it be considered as Memory leak?

Would appriciate your reply on the 2 queries mentioed above.

Regards

Sameer

Leo Laohoo · ‎07-15-2013

1-If we stop polling or disable all the traps,will the SNMP enginer process will release the memory its holding currently.

Mostly, if you set up the SNMP correctly the IOS will "SLOWLY" release the CPU/Memory. If not, reboot the switch and observe.

2-If the process still holds the memory,can it be considered as Memory leak?

If the memory is not released gracefully, yes, it's a memory leak in my book.

sameermunj · ‎07-15-2013

Hi Leo

Thanks for the quick reply..

we stopped polling and disable all the traps also,but still the memory holding for the process SNMP enginer is not going down at all after 3-4 days.This definately looks like memory leak issue.

Leo Laohoo · ‎07-15-2013

Try reboot the appliance.

And what IOS are you running?

sameermunj · ‎07-15-2013

Hi Leo

We rebotted one more stack 3 month back..(Actually it rebotted automatically after the memory utilization went high) and after reboot the utilization went down sharply.The challenge is it takes around 2 months for the problem to reappear.Looks like process keeps on holding memory and over the period it collects big block of memory which its not releasing at all..

Typical Memory leak scenario..we have checked memory leak related bugs in 15.0(1)SE3 but couldnt found any matching one.Memory leak related bugs are available in SE2 &SE4 Releases .

Leo Laohoo · ‎07-22-2013

Use 15.0(2)SE4 instead.

sameermunj · ‎07-22-2013

Hi Leo

Thanks for your suggession.

15.0.2 se4 is the latest and released recently..Is it good to go with it.

we were planning for 15.0.2 se1..(We tried with Software advisor and the option suggested was 15.0.2 se1)

Regards

Sameer