7200VXR with NPE-G1 crash

craig.potter · ‎04-02-2009

Hi all,

I have had a Cisco 7206VXR with NPE-G1 router running 12.4(22)T AdvIP Serv. crash twice in the last week. My syslog indicates a memory allocation failure / no memory error. The router runs a full BGP table. I was hoping somebody could point out quickly if there was a known issue with this configuration or not. I am investigating Bug reports and using output interpreter as well as egaging TAC but if anyone can offer some rapid thoughts, i would appreciate it. Many thanks in advance.

Giuseppe Larosa · ‎04-03-2009

Hello Craig,

the router should be fine for this job.

is it learning a single full table from a single ISP ?

Do you see free memory reducing over time (some process not releasing memory) ?

This could lead to a a software bug.

or it might be hardware related there is any indication in the crash info about the pointers values at the moment of crash?

Hope to help

Giuseppe

craig.potter · ‎04-03-2009

Thanks for your response. My error seems to revolve around memory fragmentation which causes an out of memory issue and stops everything. I have pasted the syslog messages below for reference. I cannot find any known bugs so it may be new. I will be watching the memory over the next 24 hours to see if I can find why or what is happening to the memory.

Apr 1 09:57:11 203.41.44.1 311: .Mar 31 22:57:10.217: %SYS-2-MALLOCFAIL: Memory allocation of 780 bytes failed from 0x60243A38, alignment 32

Apr 1 09:57:11 203.41.44.1 312: Pool: I/O Free: 50800 Cause: Memory fragmentation

Apr 1 09:57:11 203.41.44.1 313: Alternate Pool: None Free: 0 Cause: No Alternate pool

Apr 1 09:57:11 203.41.44.1 314: -Process= "Pool Manager", ipl= 0, pid= 6, -Traceback= 0x612F58E8 0x6000A944 0x60010ABC 0x60011324 0x60243A40 0x60021790 0x60021A2C

Apr 3 14:34:05 203.xx.xx.x 400: Apr 3 03:34:04.386: %SYS-2-MALLOCFAIL: Memory allocation of 920 bytes failed from 0x602439EC, alignment 0

Apr 3 14:34:05 203.xx.xx.x 401: Pool: Processor Free: 309184 Cause: Memory fragmentation

Apr 3 14:34:05 203.xx.xx.x 402: Alternate Pool: None Free: 0 Cause: No Alternate pool

Apr 3 14:34:05 203.xx.xx.x 403: -Process= "ADJ resolve process", ipl= 0, pid= 82, -Traceback= 0x612F58E8 0x6000A944 0x60010ABC 0x600111F0 0x63832F7C 0x602439F4 0x60244080 0x60244E10 0x61A281B4 0x61A28918 0x61A28EFC 0x61A290A8 0x61A2A800 0x638425B8 0x6171680C 0x616849D0

Apr 3 14:34:05 203.xx.xx.x 404: Apr 3 03:34:04.946: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for CCE dp subbloc. No memory available -Process= "Chunk Manager", ipl= 3, pid= 1, -Traceback= 0x612F58E8 0x6001D1B0

Apr 3 14:34:31 203.xx.xx.x 406: Apr 3 03:34:28.702: %BGP-5-ADJCHANGE: neighbor 139.x.x.x Down No memory

Apr 3 14:34:31 203.xx.xx.x 407: Apr 3 03:34:28.738: %BGP-5-ADJCHANGE: neighbor 203.x.x.x Down No memory

Apr 3 14:34:31 203.xx.xx.x 408: Apr 3 03:34:28.738: %BGP-3-NOTIFICATION: sent to neighbor 203.62.252.39 3/1 (update malformed) 0 bytes

Apr 3 14:34:58 203.xx.xx.x 409: Apr 3 03:34:57.615: %BGP-5-ADJCHANGE: neighbor 139.x.x.x Up

Apr 3 14:35:12 203.xx.xx.x 410: Apr 3 03:35:11.363: %BGP-5-ADJCHANGE: neighbor 203.x.x.x Up

Apr 3 14:36:30 203.xx.xx.x 411: Apr 3 03:36:29.871: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x6001DCDC, alignment 0

Apr 3 14:36:30 203.xx.xx.x 412: Pool: Processor Free: 872832 Cause: Memory fragmentation

Apr 3 14:36:30 203.xx.xx.x 413: Alternate Pool: None Free: 0 Cause: No Alternate pool

Apr 3 14:36:30 203.xx.xx.x 414: -Process= "BGP Router", ipl= 0, pid= 289, -Traceback= 0x612F58E8 0x6000A944 0x60010ABC 0x60011158 0x6001DCE4 0x6001EDA0 0x61C25E90 0x61C26294 0x61C31658 0x60EA67F8 0x60EA71C0 0x60EA83E0 0x63806678 0x60EA8498 0x60E7B390 0x60E7D15C

Apr 3 14:36:30 203.xx.xx.x 415: Apr 3 03:36:29.979: %COMMON_FIB-3-NOMEM: Memory allocation failure for prefix in IPv4 CEF [0x6166F9B8] (fatal) (76 subsequent failures).

Apr 3 14:36:35 203.xx.xx.x 416: Apr 3 03:36:29.979: %COMMON_FIB-4-DISABLING: IPv4 CEF is being disabled due to a fatal error.

Apr 3 14:36:35 203.xx.xx.x 417: Apr 3 03:36:34.663: %BGP-5-ADJCHANGE: neighbor 139.x.x.x Down No memory

Apr 3 14:36:35 203.xx.xx.x 418: Apr 3 03:36:34.695: %BGP-5-ADJCHANGE: neighbor 203.x.x.x Down No memory

Apr 3 14:36:36 203.xx.xx.x 419: Apr 3 03:36:35.303: %BGP-3-NOTIFICATION: sent to neighbor 203.x.x.x 3/1 (update malformed) 0 bytes

Thanks again.

Craig

Leo Laohoo · ‎04-03-2009

Craig,

Can you post your crashinfo file please?

Troubleshooting Memory Problems

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6f3a.shtml