Nexus Memory issues

Document

Mon, 11/07/2011 - 22:31
Nov 7th, 2011
User Badges:
  • Cisco Employee,

For general assessment of memory available, following two commands are used. From these command output you will be able to tell whether the platform memory utilization is normal or un-expected:


N7K-1-VDC2# show system resources


Load average: 1 minute: 0.11 5 minutes: 0.09 15 minutes: 0.14


Processes : 1241 total, 2 running


CPU states : 2.0% user, 3.4% kernel, 94.6% idle


Memory usage: 4115232K total, 3606556K used, 508676K free


Note: The “show system resources” command is used to display platform memory statistics.


For memory information per process for current VDC , following command is used:


N7k-3-VDC3# show processes memory


PID MemAlloc MemLimit MemUsed StackBase/Ptr Process


----- -------- ---------- ---------- ----------------- ----------------


4662 52756480 562929945 150167552 bfffdf00/bfffd970 netstack


MemAlloc: Data Segement size


MemLimit: Maximum memory process can use


MemUsed: Virtual Memory


For a detailed assessment of platform memory utilization, use:


N7k-3-VDC3# show system internal kernel meminfo


MemTotal: 4115232 kB -------This is the total amount of memory in the system (4 GB in Sup 1)


MemFree: 263684 kB


Buffers: 82400 kB


Cached: 1817788 kB -------The memory used by the page cache


ShmFS: 1533324 kB


Allowed: 1028808 Pages


Free: 65921 Pages


Available: 164026 Pages -------Amount of free memory in pages


SwapCached: 0 kB


Active: 2080320 kB


Inactive: 1433752 kB


HighTotal: 3338960 kB


HighFree: 4092 kB


LowTotal: 776272 kB


LowFree: 259592 kB


SwapTotal: 0 kB


SwapFree: 0 kB


Dirty: 0 kB


Writeback: 0 kB


AnonPages: 1613748 kB


Mapped: 456088 kB  ------Memory mapped into page tables


Slab: 142884 kB --------Rough indication of kernel memory consumption


SReclaimable: 25556 kB


SUnreclaim: 117328 kB


PageTables: 32756 kB


Note: One page of memory is equivalent of 4 KB of memory.


Using above commands you can determine if the utilization is high due to page cache, process holding memory or Kernel.


For further troubleshooting:


For Page cache:


If Cached is high, you should check the file system utilization and determine what kind of files are using the file system:


N7K-1-VDC2# show system internal flash


Mount-on                  1K-blocks      Used   Available   Use%  Filesystem


/                            409600     43008      367616     11   /dev/root


/proc                             0         0           0      0   proc


/sys                              0         0           0      0   none


/isan                        409600    269312      140288     66   none


/var/tmp                     307200       876      306324      1   none


/var/sysmgr                 1048576    999424       49152      96   none


/var/sysmgr/ftp              307200     24576      282624      8   none


/dev/shm                    1048576    412672      635904     40   none


/volatile                    204800         0      204800      0   none


/debug                         2048        16        2032      1   none


/dev/mqueue                       0         0           0      0   none


/mnt/cfg/0                    76099      5674       66496      8   /dev/hda5


/mnt/cfg/1                    75605      5674       66027      8   /dev/hda6


/bootflash                  1796768    629784     1075712     37   /dev/hda3


/var/sysmgr/startup-cfg      409600     27536      382064      7   none


/mnt/plog                     56192      3064       53128      6   /dev/mtdblock2


/dev/pts                          0         0           0      0   devpts


/mnt/pss                      38554      6682       29882     19   /dev/hda4


/slot0                      2026608         4     2026604      1   /dev/hdc1


/logflash                   7997912    219408     7372232      3   /dev/hde1


/bootflash_sup-remote       1767480   1121784      555912     67   127.1.1.6:/mnt/bootflash/


/logflash_sup-remote        7953616    554976     6994608      8   127.1.1.6:/mnt/logflash/


For displaying contents of these sub-folders, use:


show system internal dir full directory path


for example: show system internal dir /bootflash


And, to delete a specific file use:


filesys delete full file path


For Kernel:


Contact Cisco TAC


For user processes:


N7K# show system internal processes memory


PID TTY STAT TIME MAJFLT TRS RSS VSZ %MEM COMMAND


4727 ? Ss 00:00:00 0 1549 123248 132832 2.9 /isan/bin/pixm


4728 ? Ssl 00:00:00 0 408 78388 143104 1.8 /isan/bin/routing-sw/mrib -m 4


6662 ? Ssl 00:00:05 0 2762 64024 144396 1.5 /isan/bin/routing-sw/netstack /isan/etc/routing-sw/pm.cfg


4538 ? Ssl 00:00:00 0 2762 60448 211664 1.4 /isan/bin/routing-sw/netstack /isan/etc/routing-sw/pm.cfg


5865 ? Ssl 00:00:01 0 2762 60416 113320 1.4 /isan/bin/routing-sw/netstack /isan/etc/routing-sw/pm.cfg


6395 ? Ssl 00:00:00 0 2762 52008 105552 1.2 /isan/bin/routing-sw/netstack /isan/etc/routing-sw/pm.cfg


4271 ? Ssl 00:00:00 0 609 49812 61420 1.2 /isan/bin/routing-sw/urib


7879 ? Ssl 00:00:00 0 1909 44800 90508 1.0 /isan/bin/routing-sw/bgp -t 64000


5696 ? Ssl 00:00:17 0 337 44696 55252 1.0 /isan/bin/routing-sw/clis -cli /isan/etc/routing-sw/cli


5333 ? Ssl 00:00:14 0 337 44652 55208 1.0 /isan/bin/routing-sw/clis -cli /isan/etc/routing-sw/cli


4182 ? Ssl 00:00:15 0 337 44648 55204 1.0 /isan/bin/routing-sw/clis -cli /isan/etc/routing-sw/cli


6076 ? Ssl 00:00:14 0 337 44624 55284 1.0 /isan/bin/routing-sw/clis -cli /isan/etc/routing-sw/cli


6825 ? Ssl 00:00:00 0 1402 44576 84020 1.0 /isan/bin/routing-sw/pim -t


4268 ? Ssl 00:00:00 0 363 27132 38896 0.6 /isan/bin/routing-sw/u6rib


4732 ? Ssl 00:00:00 0 404 25220 65360 0.6 /isan/bin/routing-sw/m6rib


4726 ? S<s 00:00:00 0 144 25208 30188 0.6 /isan/bin/pixmc


Increase in the utilization for a specific process will indicate a problem. Troubleshooting process afterward depends on the culprit process.


Note: The above output when taken from any VDC will show content of all VDCs.


Memory threshold for releases prior to 4.2(4):


  • 70% Minor
  • 80% Critical
  • 90% Major


For 4.2(4) and later releases:


  • 85% Minor
  • 90% Critical
  • 95% Major


These thresholds can be configured by using command:


system memory-thresholds minor percentage severe percentage critical percentage


The command that can be used to check memory alert info is:


Show system internal memory-alert-log


Loading.

Actions

This Document

Related Content