Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 

Facebook Forum - Troubleshooting Memory and high CPU related issues on CISCO Routers (ISR, ISRG2) , 7200 (NPE-G1, NPE-G2)

Live chat with Cisco expert Srikanth Babu on October 18, 2012

Learn and ask about common symptoms, causes of Memory and high CPU related issues and how to troubleshoot the same on Cisco Integrated Services Routers (ISR-G2) , 7200 Network Processing Engine(NPE)-G1, NPE-G2.

Reference document: http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml

Srikanth Babu is a customer support engineer in the Technical Assistance Center in India, where he has nearly 3 years of experience. His areas of expertise include core architecture of Cisco routers, troubleshooting high CPU, memory issues on routers, packet drops/latency,Facebook_Forum_Template_067_small.png interface errors and more. He holds a bachelor's degree in electronics and communication as well as CCNA and CCNP certifications.

Where:

Please go to Cisco Support Facebook Page on the event day: http://www.facebook.com/CiscoSupportCommunity

When:

October 18  9:00 AM PDT (San Francisco; UTC -7 hrs)

This corresponds to:

3:00 PM CEST(Paris; UTC +1 hr)

9:00 PM PKT (Pakistan, UTC +5 hrs)

9:30 PM IST (India; UTC +5:30 hrs)

11:00 PM (Indonesia; UTC +7 hrs)

What is Facebook Forum?

Facebook forums are online conversations, held at a pre-arranged time on our Facebook page. It gives you an opportunity to interact with a live Cisco expert and get more information about a particular technology, service or product.

How do I participate?

On the day of the event, go to our Facebook page http://www.facebook.com/CiscoSupportCommunity.

To RSVP Click Here

Version history
Revision #:
1 of 1
Last update:
‎10-10-2012 01:51 PM
Updated by:
 
Labels (1)
Comments

Here's an archive of the October, 2012 Facebook Forum.

Why do we have memories in routers? Do memories in Cisco Routers have the same size and shape as we see in computers?

In layman terms, routers have a few memory components most of which is similar and comparable to computer systems. We have NVRAM where the start-up configurations is saved.Flash (PCMCI card) where we store the .bin (binary format) of the IOS and the DRAM where the binary image is decompressed. Also DRAM is used by processes (BGP, Crypto etc) to run and to store packets when they are being processed.

So does an interface of a router have an interface controller?

Yes every interface has an interface controller which would send an interrupt to the CPU when it receives a packet.

How can we check the memory status in a Cisco Router?

To check the Memory status of the device we can use any of the following command: "show version" where you would see x/y. Add both of these and you would get the total DRAM."show version" would also have the size of the flash installed on the router.

Example 1 : Cisco 7206VXR (NPE-G2) processor (revision A) with 917504K/65536K bytes of memory. Here the total DRAM is 1 GB. Example 2: 250880K bytes of ATA PCMCIA card at slot 2 (Sector size 512 bytes). The PCMCIA installed would be around 256 MB. Example 3: 65536K bytes of Flash internal SIMM (Sector size 512K). Here the flash installed would be 64 MB..Other useful memory commands would be "show memory statistics".

I heard in a training video that even if you enable 'debug all' it only uses 20% of CPU util - is this true as I was taught not to use too many debugs on a live network?

Running "debug all" would be really CPU intensive. The amount of debugs running would depend on the feature configured on the device and would vary from one device to another..Yes, you are correct , we should use debugs with caution. @ Tim, it is best to if we can run only the necessary debugs. e.g.: if you want to debug ip packets..use an ACL which matches the IPs.

I have a question regarding CPU in Cisco routers, when a router's CPU is processing data, is the priority of this data the same as configured or assigned by what so called QoS or it has different mechanism? as we know computer's CPU has its own priority of data processing for running applications. I just wanted to know if it's the same in Cisco routers or not.

How OQS would play a part packets being processes: every-time a packets hits an interface the interface controller sends an interrupt to the CPU and Interrupt Service Routine (ISR) would be processed to take care of this packet. Regarding your questions on what would be difference if it is normal packet or a packet which is marked by QOS, these packets would be placed in different queues depending on their priority marked on them and from there on it would be the normal operation of CPU handling a packet.

What should you look for when you see a MALLOCFAIL error on a router and what commands should you collect after that?

Lets take an example here: %SYS-2-MALLOCFAIL: Memory allocation of 1028 bytes failed from 0x601617A4,

pool Processor. Here the pool in where the memory allocation is occurring is the processor Pool. We should collect the

below outputs : show memory statistics, show processor memory sorted, show memory all totals. From these commands we can identify the process which is holding memory.. Once we identify the process run a "show proc mem <PID>".

If the Pool is IO..then the memory allocation failure is happening in the IO Pool, IO Pool is used for keeping packets as they come in. collect : show buffer , show buffer usage etc..Once we identify the pool (small, medium, large etc) .. collect : show buffer pool <small> packet/header..

How to intepret CPU output on a Cisco router?

XYZ;#sh processes cpu history

Load for five secs: 69%/68%; one minute: 70%; five minutes: 71%

Here the totla CPU is 69% and the CPU due to interrupt is 68%.

An interrupt is sent to the CPU when a packets hits an interface..higher the traffic/packet higher would be the interrupt sent to the CPU and higher would be the CPU cycles used..So when we see the interrupt part of this output to be high we should validate if the traffic hitting the device is high..the traffic handling capability of a device would come down significantly if we have processor intensive configuration on the device (eg : NAT, QOS, WCCP, Netflow etc)

Also interrupt driven CPU could be due to false interrupts, where we would need to perform profiling:

##### Steps to Perform Profiling #####

1. show region

2. profile <main:text start> <main:text end> 4

3. Wait for 10 mins.

4. profile stop

5. term len 0

6. show profile terse or show profile detail

7. unprofile all

e.g. 2: Load for five secs: 69%/10%; one minute: 70%; five minutes: 71%. Here the CPU is not due to interrupt/traffic.In this case we will need to collect "show proc cpu sorted" and identify the process causing high CPU and then execute : "show stack <pid>"

Is there any memory threshold that Cisco recommends as a safe figure? For example say, if there is 15 MB free in the processor pool is it normal? Looking forward to your response.

XYZ#sh mem stat

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)Processor 67C00900 98563840 45141600 53422240 53389604 52697988

I/O 2DA00000 39845376 19772268 20073108 20073108 20058204

There isn't a threshold which Cisco recommends. Again the memory utilized on a router would depend on the configuration etc.What we should see is the free count in the above output.. Also we should check if the largest is close to free. if this is not the case then this could be a case of memory fragmentation which would need a closer look...

Also don't forget to upgrade the DRAM on the device if you are adding more features/configuration.

To see the Facebook forum on Facebook visit :

http://www.facebook.com/CiscoSupportCommunity/posts/10151283364391412

http://www.facebook.com/events/296401127139851/

To see the archive on Facebook Notes, visit:

http://www.facebook.com/CiscoSupportCommunity?sk=notes#!/notes/cisco-online-support-community-netpro/facebook-forum-summary-troubleshooting-memory-and-high-cpu-related-issues-on-cis/463815946990347