Solved: 3750 memory heavily utilized - nothing obvious visible!

chaminda_t · ‎05-17-2010

I have 3750 poe (IOS ver 12.2 (25r)SEE4) switch that is responding very slowly. Utilization is as follows: CPU @ 6%, Memory 1@29%, Memory 2@ 66%. Apart from the switch complaining of a power supply failure and a Eigrp neighbour not being on a common subnet, I cannot see anything obvious why the switch’s memory is heavily utilised @ 66% and slowing down the response time of the switch. Please advice. Thanks

Calin C. · ‎05-18-2010

I saw that you are using aprox. 25M from aprox.90M available. That's not such high utilisation if you have a decent amount of traffic. I have checked some of my 3750 in the production environment and I have about 30M utilisation.

And why you are saying that CEF is not working?

To check if CEF is indeed disabled on a interface do:

show cef interface Fa x/0/y

and check for the filed IP CEF

I think you have nothing to worry about.

Let me know if this helps and please rate if it does!

View solution in original post

pciaccio · ‎05-18-2010

Check to see if you are running IDS / IPS If you are you will see high memory useage. It is normal when using it... You may need to scale back on some of the signatures...

View solution in original post

Calin C. · ‎05-17-2010

This behavior is quite hard to solve. Try to answer the following questions and maybe it will help you.

If you reload the switch the problems reappear after a while?

I see you have routing enabled there...everything OK? Check the EIGRP. Try to shut it down and see if you still have high mem usage.

Broadcast, multicast, unicast storm?

High number of mac-addresses (can be an attack)?

What SDM mode are you using on the switch?

If the switch is in productive environment, take care with troubleshooting as you may interrupt network.

chaminda_t · ‎05-17-2010

Hi,

Many thanks for the quick response. Routing is enabled (eigrp). I could only see no more than apprx 15 dynamic addresses in the mac-address-table. SDM (if you meant: Security Device Manager) –not used on this site.

I am not able to reload the switch for now as it is in a strategic production environment. Would this be the next practical step?

Calin C. · ‎05-17-2010

SDM - Switch database management

enable

show sdm prefer
The current template is "desktop default" template.
The selected template optimizes the resources in
the switch to support this level of features for
8 routed interfaces and 1024 VLANs.

number of unicast mac addresses:                  6K
   number of IPv4 IGMP groups + multicast routes:    1K
   number of IPv4 unicast routes:                    8K
     number of directly-connected IPv4 hosts:        6K
     number of indirect IPv4 routes:                 2K
   number of IPv4 policy based routing aces:         0
   number of IPv4/MAC qos aces:                      0.5K
   number of IPv4/MAC security aces:                 1K

This is default one

If you need to change it (e.g. more than 8 routed interfaces or 1024 VLANs)

enable

conf t:

sdm ?

prefer Config TCAM and Forwarding RAM sizes. Warning: need to reset switch
for configuration to take effect.

sdm prefer ?
access              Access bias
default             Default bias
dual-ipv4-and-ipv6 Support both IPv4 and IPv6
ipe                 IPe bias
routing             Unicast bias
vlan                VLAN bias

can you send here the output of show processes memory sorted ?

Br

Calin

chaminda_t · ‎05-17-2010

Hi,

SDM profile is the default. I have attached the 'sh proc mem sorted' FYI. Also I noticed that, cef does not seem to work. Please look at the attached file 'sh interfaces stats'. please advice

Calin C. · ‎05-18-2010

I saw that you are using aprox. 25M from aprox.90M available. That's not such high utilisation if you have a decent amount of traffic. I have checked some of my 3750 in the production environment and I have about 30M utilisation.

And why you are saying that CEF is not working?

To check if CEF is indeed disabled on a interface do:

show cef interface Fa x/0/y

and check for the filed IP CEF

I think you have nothing to worry about.

Let me know if this helps and please rate if it does!

pciaccio · ‎05-18-2010

Check to see if you are running IDS / IPS If you are you will see high memory useage. It is normal when using it... You may need to scale back on some of the signatures...

chaminda_t · ‎05-21-2010

Gents, Many thanks for all the advice -

Issue seemed to have been the following:

CSCtg39752 Bug

3750 : Ingress IP Traffic gets punted to CPU

Symptom: All ingress routed IP traffic gets punted to CPU due to incorrect hw programming.

Conditions: Without any network topology, hw/sw, cfgs or traffic pattern change it is observed 3750 in StackWise mode starts punting traffic to CPU due to incorrect hw programming. This causes heavy pkt loss and network instability

Workaround:Reboot does not work. Erase entire cfg, and reapply same cfg. Still reported issue will get trigger with this temporary workaround