7200VXR with NPE-G1 crash

Unanswered Question
Apr 2nd, 2009
User Badges:

Hi all,


I have had a Cisco 7206VXR with NPE-G1 router running 12.4(22)T AdvIP Serv. crash twice in the last week. My syslog indicates a memory allocation failure / no memory error. The router runs a full BGP table. I was hoping somebody could point out quickly if there was a known issue with this configuration or not. I am investigating Bug reports and using output interpreter as well as egaging TAC but if anyone can offer some rapid thoughts, i would appreciate it. Many thanks in advance.



  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Giuseppe Larosa Fri, 04/03/2009 - 00:20
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Craig,

the router should be fine for this job.

is it learning a single full table from a single ISP ?


Do you see free memory reducing over time (some process not releasing memory) ?

This could lead to a a software bug.


or it might be hardware related there is any indication in the crash info about the pointers values at the moment of crash?


Hope to help

Giuseppe


craig.potter Fri, 04/03/2009 - 03:24
User Badges:

Thanks for your response. My error seems to revolve around memory fragmentation which causes an out of memory issue and stops everything. I have pasted the syslog messages below for reference. I cannot find any known bugs so it may be new. I will be watching the memory over the next 24 hours to see if I can find why or what is happening to the memory.


Apr 1 09:57:11 203.41.44.1 311: .Mar 31 22:57:10.217: %SYS-2-MALLOCFAIL: Memory allocation of 780 bytes failed from 0x60243A38, alignment 32

Apr 1 09:57:11 203.41.44.1 312: Pool: I/O Free: 50800 Cause: Memory fragmentation

Apr 1 09:57:11 203.41.44.1 313: Alternate Pool: None Free: 0 Cause: No Alternate pool

Apr 1 09:57:11 203.41.44.1 314: -Process= "Pool Manager", ipl= 0, pid= 6, -Traceback= 0x612F58E8 0x6000A944 0x60010ABC 0x60011324 0x60243A40 0x60021790 0x60021A2C



Apr 3 14:34:05 203.xx.xx.x 400: Apr 3 03:34:04.386: %SYS-2-MALLOCFAIL: Memory allocation of 920 bytes failed from 0x602439EC, alignment 0

Apr 3 14:34:05 203.xx.xx.x 401: Pool: Processor Free: 309184 Cause: Memory fragmentation

Apr 3 14:34:05 203.xx.xx.x 402: Alternate Pool: None Free: 0 Cause: No Alternate pool

Apr 3 14:34:05 203.xx.xx.x 403: -Process= "ADJ resolve process", ipl= 0, pid= 82, -Traceback= 0x612F58E8 0x6000A944 0x60010ABC 0x600111F0 0x63832F7C 0x602439F4 0x60244080 0x60244E10 0x61A281B4 0x61A28918 0x61A28EFC 0x61A290A8 0x61A2A800 0x638425B8 0x6171680C 0x616849D0

Apr 3 14:34:05 203.xx.xx.x 404: Apr 3 03:34:04.946: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for CCE dp subbloc. No memory available -Process= "Chunk Manager", ipl= 3, pid= 1, -Traceback= 0x612F58E8 0x6001D1B0

Apr 3 14:34:31 203.xx.xx.x 406: Apr 3 03:34:28.702: %BGP-5-ADJCHANGE: neighbor 139.x.x.x Down No memory

Apr 3 14:34:31 203.xx.xx.x 407: Apr 3 03:34:28.738: %BGP-5-ADJCHANGE: neighbor 203.x.x.x Down No memory

Apr 3 14:34:31 203.xx.xx.x 408: Apr 3 03:34:28.738: %BGP-3-NOTIFICATION: sent to neighbor 203.62.252.39 3/1 (update malformed) 0 bytes

Apr 3 14:34:58 203.xx.xx.x 409: Apr 3 03:34:57.615: %BGP-5-ADJCHANGE: neighbor 139.x.x.x Up

Apr 3 14:35:12 203.xx.xx.x 410: Apr 3 03:35:11.363: %BGP-5-ADJCHANGE: neighbor 203.x.x.x Up

Apr 3 14:36:30 203.xx.xx.x 411: Apr 3 03:36:29.871: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x6001DCDC, alignment 0

Apr 3 14:36:30 203.xx.xx.x 412: Pool: Processor Free: 872832 Cause: Memory fragmentation

Apr 3 14:36:30 203.xx.xx.x 413: Alternate Pool: None Free: 0 Cause: No Alternate pool

Apr 3 14:36:30 203.xx.xx.x 414: -Process= "BGP Router", ipl= 0, pid= 289, -Traceback= 0x612F58E8 0x6000A944 0x60010ABC 0x60011158 0x6001DCE4 0x6001EDA0 0x61C25E90 0x61C26294 0x61C31658 0x60EA67F8 0x60EA71C0 0x60EA83E0 0x63806678 0x60EA8498 0x60E7B390 0x60E7D15C

Apr 3 14:36:30 203.xx.xx.x 415: Apr 3 03:36:29.979: %COMMON_FIB-3-NOMEM: Memory allocation failure for prefix in IPv4 CEF [0x6166F9B8] (fatal) (76 subsequent failures).

Apr 3 14:36:35 203.xx.xx.x 416: Apr 3 03:36:29.979: %COMMON_FIB-4-DISABLING: IPv4 CEF is being disabled due to a fatal error.

Apr 3 14:36:35 203.xx.xx.x 417: Apr 3 03:36:34.663: %BGP-5-ADJCHANGE: neighbor 139.x.x.x Down No memory

Apr 3 14:36:35 203.xx.xx.x 418: Apr 3 03:36:34.695: %BGP-5-ADJCHANGE: neighbor 203.x.x.x Down No memory

Apr 3 14:36:36 203.xx.xx.x 419: Apr 3 03:36:35.303: %BGP-3-NOTIFICATION: sent to neighbor 203.x.x.x 3/1 (update malformed) 0 bytes



Thanks again.


Craig



Leo Laohoo Fri, 04/03/2009 - 04:14
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Craig,

Can you post your crashinfo file please?


Troubleshooting Memory Problems

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6f3a.shtml

Actions

This Discussion