6509 high memory usage, any need to be concerned?

mastrboy_ · ‎05-14-2010

I have been reading trough http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/best/practices/recommendations.html before deploying a 6509 VSS, after checking the command "sh platform hardware capacity cpu" i see almost all the memory is in use, while the switch basicly does nothing, it has som switches connected but there is not any traffic.

The documentation states:

Issue the show platform hardware capacity command periodically to monitor system resource usage. Any system resource with usage of 90% or greater should be addressed.

Output from sh platform hardware capacity cpu:

CPU Resources
  CPU utilization: Sw/Mod             5 seconds       1 minute       5 minutes
                    1/1                0% /  6%             0%              1%
                    1/5  RP            2% /  3%             1%              1%
                    1/5  SP            3% /  3%             4%              4%
                    1/7                0% /  1%             0%              0%
                    2/1                0% /  6%             0%              1%
                    2/5  RP            0% /  0%             0%              0%
                    2/5  SP            2% /  2%             4%              4%
                    2/7                0% /  1%             0%              0%
  Processor memory: Sw/Mod   Bytes:       Total           Used           %Used
                     1/1               25165648       24653960             98%
                     1/5  RP           88799456       83598736             94%
                     1/5  SP           94394032       90762536             96%
                     1/7               29359864       25274104             86%
                     2/1               25165648       24672040             98%
                     2/5  RP           84605240       82950992             98%
                     2/5  SP           90199816       89629440             99%
                     2/7               29359864       25216104             86%
  I/O memory: Sw/Mod         Bytes:       Total           Used           %Used
               1/1                      8126344        7886544             97%
               1/5  RP                 12058384       10714264             89%
               1/5  SP                  8388608        8104304             97%
               1/7                      8126344        7886544             97%
               2/1                      8126344        7886544             97%
               2/5  RP                 12058384       10714264             89%
               2/5  SP                  8388608        8104304             97%
               2/7                      8126344        7886544             97%

Output from sh proc mem:

System Memory : 1048576K total, 365989K used, 682587K free, 1000K kernel reserved
Lowest(b) : 698920960

Could there be a memory leak or is this normal?

IOS Version: Cisco IOS Software, s72033_rp Software (s72033_rp-IPSERVICESK9_WAN-VM), Version 12.2(33)SXI3, RELEASE SOFTWARE (fc2)

added sh proc mem info

Giuseppe Larosa · ‎05-14-2010

Hello Joakim,

>> Could there be a memory leak or is this normal?

memory leak would mean increasing use of memory over time caused by a process not releasing memory resources

but we see that:

your IOS image 12.2(33)SXI3 is considered good for VSS as reported by other colleagues like Reza in other threads

sh proc mem shows

System Memory : 1048576K total, 365989K used, 682587K free, 1000K kernel reserved

682587 KB free memory

so I would say that the sh platform hardware capacity cpu may show memory usage of CEF tables rather then main cpu memory usage

I would look at PFC memory and DFC memory on each linecard

how many IP prefixes are present in the system?

sh ip route summary

can tell you this

Hope to help

Giuseppe

mastrboy_ · ‎05-14-2010

Not much:

# sh ip route summary

IP routing table name is Default-IP-Routing-Table(0)

IP routing table maximum-paths is 32

Route Source Networks Subnets Overhead Memory (bytes)

connected 0 6 432 864

static 1 1 144 288

internal 5 10940

Total 6 7 576 12092

Removing Queue Size 0

usman ali dar · ‎11-05-2014

hey folks

we have a 6509 and its been constantly working on from 85-96% daily bases can you see the below output and kindly help me understand this out and how can i get this down

regards

sh hardware capacity

Module Ttlent QoSent RBLent Lbl LOU TCP Dstbl Ethcam ACTtbl V6ext
1 2% 6% 0% 1% 1% 0% 2% 0% 0% 0%
6 2% 6% 0% 1% 1% 0% 2% 0% 0% 0%
8 2% 6% 0% 1% 1% 0% 2% 0% 0% 0%
BA-C154-COR01#sh platform hardware capacity ?
acl Show QoS/Security ACL capacity
cpu Show CPU resources capacity
eobc Show EOBC resources capacity
fabric Show Fabric resources capacity
flash Show Flash/NVRAM resources capacity
forwarding Show forwarding engine capacity
ibc show IBC resources
interface Show Interface resources capacity
monitor Show SPAN resources capacity
multicast Show L3 and LTL Multicast resources
netflow Show Netflow capacity
pfc Show PFC resources capacity
power Show Power resources capacity
qos Show QoS resources capacity
rate-limit Show CPU Rate Limiters capacity
rewrite-engine Show rewrite-engine capacity
system Show System resources capacity
vlan Show VLAN resources capacity
| Output modifiers
<cr>

BA-C154-COR01#sh platform hardware capacity vlan
BA-C154-COR01#sh platform hardware capacity vlan
VLAN Resources
VLANs: 4094 total, 158 VTP, 153 extended VTP, 71 internal, 3712 free
BA-C154-COR01#sh platform hardware capacity
System Resources
PFC operating mode: PFC4
Supervisor redundancy mode: administratively sso, operationally sso
Switching resources: Module Part number Series CEF mode
1 WS-X6824-SFP CEF720 dCEF
2 WS-SVC-WISM2-K9 CEF720 CEF
3 WS-X6704-10GE CEF720 CEF
6 VS-SUP2T-10G supervisor CEF
7 WS-SVC-NAM-2-250S classic CEF
8 WS-X6848-GE-TX CEF720 dCEF

Power Resources
Power supply redundancy mode: administratively redundant
operationally redundant
System power: 3795W, 0W (0%) inline, 2650W (70%) total allocated
Powered devices: 0 total, 0 Class4, 0 Class3, 0 Class2, 0 Class1, 0 Class0, 0 Cisco

Flash/NVRAM Resources
Usage: Module Device Bytes: Total Used %Used
1 dfc#1-bootflash: 15990784 0 0%
2 cfc#2-bootflash: 15990784 3610808 23%
3 cfc#3-bootflash: 32768000 0 0%
6 RP nvram: 2552192 117856 5%
6 RP const_nvram: 1048556 16564 2%
6 RP bootdisk: 1024557056 339574784 33%
8 dfc#8-bootflash: 15990784 0 0%

CPU Resources
CPU utilization: Module 5 seconds 1 minute 5 minutes
1 11% / 1% 11% 11%
2 4% / 3% 2% 1%
3 5% / 4% 1% 1%
6 RP 96% / 81% 94% 94%
8 9% / 0% 12% 12%
Processor memory: Module Bytes: Total Used %Used
1 1002567200 202036204 20%
2 1002567200 85504916 9%
3 197260960 57244324 29%
6 RP 1563624544 263755848 17%
8 1002567200 202550648 20%
I/O memory: Module Bytes: Total Used %Used
6 RP 268435456 110348952 41%

EOBC Resources
Module Packets/sec Total packets Dropped packets
1 Rx: 31 3189904054 0
Tx: 26 425151314 3
2 Rx: 8 3190197720 0
Tx: 2 203243440 3
3 Rx: 10 3190201752 0
Tx: 5 225185037 3
6 RP Rx: 139 1573214168 0
Tx: 142 1612182916 0
8 Rx: 49 3189834076 0
Tx: 44 475351872 3

VLAN Resources
VLANs: 4094 total, 158 VTP, 153 extended VTP, 71 internal, 3712 free

L2 Forwarding Resources
MAC Table usage: Module Collisions Total Used %Used
1 0 131072 10723 8%
6 0 131072 10901 8%
8 0 131072 10960 8%

L3 Forwarding Resources
FIB TCAM usage: Total Used %Used
72 bits (IPv4, MPLS, EoM) 262144 5254 2%
144 bits (IP mcast, IPv6) 131072 348 1%
288 bits (IPv6 mcast) 65536 46 1%

detail: Protocol Used %Used
IPv4 5253 2%
MPLS 0 0%
EoM 1 1%

IPv6 4 1%
IPv4 mcast 344 1%
IPv6 mcast 46 1%

Adjacency usage: Total Used %Used
1048576 37189 4%

Forwarding engine load:
Module pps peak-pps peak-time
1 163793 788843 13:52:42 EDT Tue Sep 9 2014
6 745649 1216821 16:51:37 EDT Sun Aug 17 2014
8 22802 198500 11:09:10 EDT Wed Sep 3 2014

Netflow resources:
Netflow table size: 515032 entries total

Netflow table usage: Module/Instance Input flows Output flows
1 0% 0%
6 0% 0%
8 0% 0%

Profile/Flowmask allocation:
IPv4: total 32, used/reserved 5, available 27
ID Feature
0 FM
1 FM
29 NDE
30 NDE
31 FM
IPv6: total 32, used/reserved 4, available 28
ID Feature
0 FM
1 FM
30 NDE
31 NDE
MPLS: total 8, used/reserved 4, available 4
ID Feature
0 FM
5 QOS
6 QOS
7 QOS
L2: total 8, used/reserved 3, available 5
ID Feature
0 FM
1 FM
7 FM

Sampler allocation: total 1024, used/reserved 1, available 1023

Copy sampler allocation: total 8, used/reserved 1, available 7

CPU Rate Limiters Resources
Rate limiters: Total Used Reserved %Used
Layer 3 31 2 0 6%
Layer 2 Input 20 9 5 45%
Layer 2 Output 6 0 0 0%

Classification Mgr Tcam Resources
Key: Ttlent - Total TCAM entries, QoSent - QoS TCAM entries, LOU - LOUs,
RBLent - RBACL TCAM entries, Lbl - Labels, TCP - TCP Flags,
Dstbl - Destinfo Table, Ethcam - Ethertype Cam Table,
ACTtbl - Accounting Table, V6ext - V6 Extn Hdr Table

Module Ttlent QoSent RBLent Lbl LOU TCP Dstbl Ethcam ACTtbl V6ext
1 2% 6% 0% 1% 1% 0% 2% 0% 0% 0%
6 2% 6% 0% 1% 1% 0% 2% 0% 0% 0%
8 2% 6% 0% 1% 1% 0% 2% 0% 0% 0%

L3 Multicast Resources
Replication mode: egress
Bi-directional PIM Designated Forwarder Table Capacity: 8 Per Vrf
Bi-directional PIM Designated Forwarder Table usage:
Vrf IPV4 used IPV6 used Total used

Default 1 0 1 (12 %)
Replication capability: Module Capability
1 egress
2 egress
3 egress
6 egress
7 ingress
8 egress

MET table Entries: Module Total Used %Used
1 65518 358 1%
6 32752 110 1%
8 65518 248 1%

QoS Policer Resources
Aggregate policers: Module Total Used %Used
1 16384 2 1%
6 16384 2 1%
8 16384 2 1%
Microflow policer configurations: Module Total Used %Used
1 128 1 1%
6 128 1 1%
8 128 1 1%
Netflow policer configurations: Module Total Used %Used
1 384 0 0%
6 384 0 0%
8 384 0 0%
Aggregate policer configurations: Module Total Used %Used
1 1024 2 1%
6 1024 2 1%
8 1024 2 1%
Distributed policers: Total Used %Used
4096 1 1%
QoS Tcam Entries: Module Total Used %Used
1 16384 1034 6%
6 16384 1034 6%
8 16384 1034 6%

Switch Fabric Resources
Bus utilization: current: 38%, peak was 74% at 16:59:07 EDT Sun Aug 17 2014
Fabric utilization: Ingress Egress
Module Chanl Speed rate peak rate peak
1 0 20G 7% 22% @10:42 28Oct14 3% 27% @04:00 13Aug14
2 0 20G 2% 10% @12:17 29Oct14 0% 5% @09:06 27Oct14
3 0 20G 0% 15% @04:40 17Aug14 0% 23% @16:02 17Aug14
3 1 20G 7% 22% @14:31 14Aug14 0% 10% @23:13 11Oct14
6 0 20G 7% 57% @16:02 17Aug14 7% 17% @14:09 29Oct14
6 1 20G 2% 5% @10:24 04Nov14 18% 48% @16:02 17Aug14
8 0 20G 0% 7% @08:32 23Sep14 0% 3% @09:28 05Nov14
8 1 20G 0% 2% @11:06 26Sep14 0% 3% @09:28 05Nov14
Switching mode: Module Switching mode
1 dcef
2 acef
3 acef
6 bus
8 dcef

Interface Resources
Interface drops:
Module Total drops: Tx Rx Highest drop port: Tx Rx
7 2866334516 0 7 0
8 10252753 0 15 0

Interface buffer sizes:
Mod/Port Bytes: Tx buffer Rx buffer
1/1 1221184 173952
1/2 1221184 173952
1/3 1221184 173952
1/4 1221184 173952
1/5 1221184 173952
1/6 1221184 173952
1/7 1221184 173952
1/8 1221184 173952
1/9 1221184 173952
1/10 1221184 173952
1/11 1221184 173952
1/12 1221184 173952
1/13 1221184 173952
1/14 1221184 173952
1/15 1221184 173952
1/16 1221184 173952
1/17 1221184 173952
1/18 1221184 173952
1/19 1221184 173952
1/20 1221184 173952
1/21 1221184 173952
1/22 1221184 173952
1/23 1221184 173952
1/24 1221184 173952
3/1 14622592 2064768
3/2 14622592 2064768
3/3 14622592 2064768
3/4 14622592 2064768
6/1 8640576 10127232
6/2 8640576 10127232
6/3 8640576 10127232
6/4 91887104 109293504
6/5 91887104 109293504
8/1 1221184 173952
8/2 1221184 173952
8/3 1221184 173952
8/4 1221184 173952
8/5 1221184 173952
8/6 1221184 173952
8/7 1221184 173952
8/8 1221184 173952
8/9 1221184 173952
8/10 1221184 173952
8/11 1221184 173952
8/12 1221184 173952
8/13 1221184 173952
8/14 1221184 173952
8/15 1221184 173952
8/16 1221184 173952
8/17 1221184 173952
8/18 1221184 173952
8/19 1221184 173952
8/20 1221184 173952
8/21 1221184 173952
8/22 1221184 173952
8/23 1221184 173952
8/24 1221184 173952
8/25 1221184 173952
8/26 1221184 173952
8/27 1221184 173952
8/28 1221184 173952
8/29 1221184 173952
8/30 1221184 173952
8/31 1221184 173952
8/32 1221184 173952
8/33 1221184 173952
8/34 1221184 173952
8/35 1221184 173952
8/36 1221184 173952
8/37 1221184 173952
8/38 1221184 173952
8/39 1221184 173952
8/40 1221184 173952
8/41 1221184 173952
8/42 1221184 173952
8/43 1221184 173952
8/44 1221184 173952
8/45 1221184 173952
8/46 1221184 173952
8/47 1221184 173952
8/48 1221184 173952
IBC Resources
Module Packets/sec Total packets Dropped packets
6 RP Rx: 90948 160807019625 1
Tx: 92221 178588757499 0

SPAN Resources
Source sessions: 16 maximum, 2 used
Type Max Used
Local 2(*) 2
Local-tx 14 0
RSPAN source 2(*) 0
ERSPAN source 2(*) 0
Capture 1(*) 0
Service module 1(*) 0
OAM loopback 1(*) 0
Reflector 1(*) 0
* - shared source sessions and the total can not exceed 2
Destination sessions: 64 maximum, 0 used
Type Max Used
RSPAN destination 64(*) 0
ERSPAN destination 23(*) 0
* - shared destination sessions and the total can not exceed 64

Multicast LTL Resources
Usage: 38848 Total, 835 Used

Leo Laohoo · ‎11-05-2014

Post the output to the command "sh proc cpu sorted | ex 0.00".