Thanks for the explanation.

kgtnewmedia · ‎09-18-2014

Hi,

any advices what to do if all RAM of the RP is consumed?

------------------------------------------------------------------------------------------

r1-core#show log

Sep 17 03:38:13.999 CET: %OSPF-5-ADJCHG: Process 8877, Nbr 10.100.205.246 on Vlan60 from FULL to DOWN, Neighbor Down: Dead timer expired

Sep 17 03:38:20.715 CET: %OSPF-5-ADJCHG: Process 8877, Nbr 10.100.205.246 on Vlan60 from LOADING to FULL, Loading Done

Sep 17 03:38:48.451 CET: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x434488CC, alignment 0

Pool: Processor Free: 1107328 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "BGP Task", ipl= 0, pid= 626

-Traceback= 4340F154z 43428FE4z 4343FDD4z 41C77F70z 41CCB3C4z 41CCD7C0z 41CD0604z 41D3C380z 41D3CB48z 41D24B20z 433FAAC0z 433FAAACz

Sep 17 03:38:54.683 CET: %SYS-2-CFORKMEM: Process creation of BGP Open failed (no memory). -Process= "BGP Router", ipl= 0, pid= 558

-Traceback= 43444E34z 43444FDCz 41C8E09Cz 433F52D4z 433F5408z 41C903D0z 41CA22F8z

------------------------------------------------------------------------------------------

r1-core#show version

Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9-M), Version 15.1(1)SY1, RELEASE SOFTWARE (fc5)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 01-May-13 13:16 by prod_rel_team

ROM: System Bootstrap, Version 12.2(17r)SX7, RELEASE SOFTWARE (fc1)

BOOTLDR: Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9-M), Version 15.1(1)SY1, RELEASE SOFTWARE (fc5)

r1-core-inn uptime is 16 weeks, 4 days, 4 hours, 16 minutes

Uptime for this control processor is 16 weeks, 4 days, 4 hours, 6 minutes

System returned to ROM by reload at 04:14:47 CET Wed Jan 12 2000 (SP by reload)

System restarted at 18:34:19 CET Sun May 25 2014

System image file is "disk0:s72033-adventerprisek9-mz.151-1.SY1.bin"

Last reload reason: Reload Command

...

cisco WS-C6509-E (R7000) processor (revision 1.3) with 983008K/65536K bytes of memory.

Processor board ID SMC11140074

SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache

Last reset from s/w reset

16 Virtual Ethernet interfaces

122 Gigabit Ethernet interfaces

1917K bytes of non-volatile configuration memory.

65536K bytes of Flash internal SIMM (Sector size 512K).

Configuration register is 0x2102

------------------------------------------------------------------------------------------

r1-core#show module

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

1 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1301F7T8

3 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1108HB7S

5 2 Supervisor Engine 720 (Active) WS-SUP720-BASE SAD08340D87

8 24 CEF720 24 port 1000mb SFP WS-X6724-SFP SAL1338Z0SV

Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------ ------------ -------

1 0024.1469.ccb0 to 0024.1469.ccdf 3.0 12.2(18r)S1 15.1(1)SY1 Ok

3 001a.e27a.3840 to 001a.e27a.386f 2.5 12.2(18r)S1 15.1(1)SY1 Ok

5 0011.21b9.9928 to 0011.21b9.992b 3.2 8.5(4) 15.1(1)SY1 Ok

8 0027.0d34.ed50 to 0027.0d34.ed67 4.1 12.2(18r)S1 15.1(1)SY1 Ok

Mod Sub-Module Model Serial Hw Status

---- --------------------------- ------------------ ----------- ------- -------

1 Distributed Forwarding Card WS-F6700-DFC3BXL SAL11370Q8C 5.3 Ok

3 Distributed Forwarding Card WS-F6700-DFC3BXL SAL1333W4V1 5.6 Ok

5 Policy Feature Card 3 WS-F6K-PFC3BXL SAL1040X3AU 1.8 Ok

5 MSFC3 Daughterboard WS-SUP720 SAL1040Z0JL 2.2 Ok

8 Distributed Forwarding Card WS-F6700-DFC3BXL SAL1439UQVP 5.6 Ok

Mod Online Diag Status

---- -------------------

1 Pass

3 Pass

5 Pass

8 Pass

------------------------------------------------------------------------------------------

r1-core#show memory summary

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)

Processor 4732E520 885824224 873473292 12350932 1107328 556184

I/O 8000000 67108864 21641732 45467132 45467132 45465404

Critical 537DCDE0 475940 44 475896 475896 475896

Critical 949A940 36084 44 36040 36040 36040

Processor memory

Address Bytes Prev Next Ref PrevF NextF Alloc PC what

4732E520 0016853976 00000000 48341124 001 -------- -------- 433EB834 *Init*

48341124 0000012800 4732E520 48344350 001 -------- -------- 43FAE89C *Init*

48344350 0000005000 48341124 48345704 001 -------- -------- 433F9578 List Headers

48345704 0000000160 48344350 483457D0 001 -------- -------- 43FAAFA0 *Init*

483457D0 0000000160 48345704 4834589C 001 -------- -------- 43FAAFA0 *Init*

4834589C 0000000160 483457D0 48345968 001 -------- -------- 43FAAFA0 *Init*

48345968 0000000160 4834589C 48345A34 001 -------- -------- 43FAAFA0 *Init*

48345A34 0000000160 48345968 48345B00 001 -------- -------- 43FAAFA0 *Init*

48345B00 0000001500 48345A34 48346108 001 -------- -------- 43452288 messages

48346108 0000010000 48345B00 48348844 001 -------- -------- 434522E4 Watched Queue

48348844 0000010000 48346108 4834AF80 001 -------- -------- 4345233C Watched Bitfield

4834AF80 0000005000 48348844 4834C334 001 -------- -------- 43452394 Watcher Info

4834C334 0000001500 4834AF80 4834C93C 001 -------- -------- 434523EC Watcher Message Queue

4834C93C 0000004288 4834C334 4834DA28 001 -------- -------- 4073CD0C TTY data

4834DA28 0000000512 4834C93C 4834DC54 001 -------- -------- 4073E668 TTY Output Buf

4834DC54 0000000160 4834DA28 4834DD20 001 -------- -------- 43FAAFA0 *Init*

4834DD20 0000026000 4834DC54 483542DC 001 -------- -------- 41481DFC Process Stack

483542DC 0000000184 4834DD20 483543C0 001 -------- -------- 4344FDC8 Process Events

483543C0 0000000160 483542DC 4835448C 001 -------- -------- 43FAAFA0 *Init*

4835448C 0000000648 483543C0 48354740 001 -------- -------- 4344D754 Process

48354740 0000000124 4835448C 483547E8 001 -------- -------- 41485478 *Init*

483547E8 0000010000 48354740 48356F24 001 -------- -------- 433F82E0 List Elements

48356F24 0000010000 483547E8 48359660 001 -------- -------- 433F82E0 List Elements

48359660 0000010000 48356F24 4835BD9C 001 -------- -------- 433F82E0 List Elements

4835BD9C 0000010000 48359660 4835E4D8 001 -------- -------- 433F82E0 List Elements

4835E4D8 0000010000 4835BD9C 48360C14 001 -------- -------- 433F82E0 List Elements

48360C14 0000000220 4835E4D8 48360D1C 001 -------- -------- 408658C8 Env action

48360D1C 0000000184 48360C14 48360E00 001 -------- -------- 408658C8 Env action

48360E00 0000000184 48360D1C 48360EE4 001 -------- -------- 408658C8 Env action

48360EE4 0000000148 48360E00 48360FA4 001 -------- -------- 43782A4C Crypto CA

48360FA4 0000000184 48360EE4 48361088 001 -------- -------- 408657C4 Env action

48361088 0000000184 48360FA4 4836116C 001 -------- -------- 408658C8 Env action

4836116C 0000000184 48361088 48361250 001 -------- -------- 408657C4 Env action

48361250 0000000184 4836116C 48361334 001 -------- -------- 408658C8 Env action

48361334 0000000832 48361250 483616A0 001 -------- -------- 40D9BF10 FM_HA_RDT_INDEX_LIST

483616A0 0000000832 48361334 48361A0C 001 -------- -------- 40D9BF10 FM_HA_RDT_INDEX_LIST

48361A0C 0000000128 483616A0 48361AB8 001 -------- -------- 40D29F90 FM core

48361AB8 0000000240 48361A0C 48361BD4 001 -------- -------- 437BAABC Crypto CA

48361BD4 0000000028 48361AB8 48361C1C 001 -------- -------- 437A3F88 Crypto CA

48361C1C 0000000184 48361BD4 48361D00 001 -------- -------- 408657C4 Env action

48361D00 0000000184 48361C1C 48361DE4 001 -------- -------- 408657C4 Env action

48361DE4 0000000184 48361D00 48361EC8 001 -------- -------- 408658C8 Env action

48361EC8 0000000032 48361DE4 48361F14 001 -------- -------- 40C106E8 draco-oir-process:slot 1

48361F14 0000000032 48361EC8 48361F60 001 -------- -------- 42269CFC Entity MIB C6k Private Process

48361F60 0000000032 48361F14 48361FAC 001 -------- -------- 420D4CB4 Bitfield pool

48361FAC 0000000512 48361F60 483621D8 001 -------- -------- 425779B4 IPC Rx Win Table

483621D8 0000000512 48361FAC 48362404 001 -------- -------- 425779B4 IPC Rx Win Table

48362404 0000000648 483621D8 483626B8 001 -------- -------- 407B4094 XDR: Peer cblk

483626B8 0000000320 48362404 48362824 001 -------- -------- 407BA6C8 XDR: stats array

48362824 0000000300 483626B8 4836297C 001 -------- -------- 407BA6C8 XDR: stats array

4836297C 0000000280 48362824 48362AC0 001 -------- -------- 407BA6C8 XDR: stats array

48362AC0 0000002420 4836297C 48363460 001 -------- -------- 407BA6C8 XDR: stats array

48363460 0000000460 48362AC0 48363658 001 -------- -------- 407BA6C8 XDR: stats array

48363658 0000000560 48363460 483638B4 001 -------- -------- 407BA6C8 XDR: stats array

483638B4 0000000320 48363658 48363A20 001 -------- -------- 407BA6C8 XDR: stats array

48363A20 0000000280 483638B4 48363B64 001 -------- -------- 407BA6C8 XDR: stats array

48363B64 0000000120 48363A20 48363C08 001 -------- -------- 407BA6C8 XDR: stats array

48363C08 0000000156 48363B64 48363CD0 001 -------- -------- 407BA6C8 XDR: stats array

48363CD0 0000001500 48363C08 483642D8 001 -------- -------- 434040C4 Reg Function Seq List

483642D8 0000001500 48363CD0 483648E0 001 -------- -------- 4340411C Reg Function iList

483648E0 0000001500 483642D8 48364EE8 001 -------- -------- 43404174 Reg Function Caselist

48364EE8 0000000032 483648E0 48364F34 001 -------- -------- 42075A40 Init

48364F34 0000000184 48364EE8 48365018 001 -------- -------- 408658C8 Env action

48365018 0000000184 48364F34 483650FC 001 -------- -------- 4124C56C QM-SB

483650FC 0000000060 48365018 48365164 001 -------- -------- 40B237E0 IPC Seat Manager

48365164 0000000088 483650FC 483651E8 001 -------- -------- 4133A234 SIP10GQM-SB

483651E8 0000000032 48365164 48365234 001 -------- -------- 41D08250 BGP Router

48365234 0000000048 483651E8 48365290 001 -------- -------- 43222694 Entity MIB API

48365290 0000000112 48365234 4836532C 001 -------- -------- 4227D18C *Init*

4836532C 0000000032 48365290 48365378 001 -------- -------- 4203E7F0 Parser Linkage

48365378 0000000032 4836532C 483653C4 001 -------- -------- 4203E7F0 Parser Linkage

483653C4 0000000032 48365378 48365410 001 -------- -------- 4203E7F0 Parser Linkage

48365410 0000000032 483653C4 4836545C 001 -------- -------- 42075A40 Init

4836545C 0000000072 48365410 483654D0 001 -------- -------- 43404C54 *Init*

483654D0 0000000072 4836545C 48365544 001 -------- -------- 43404C54 *Init*

48365544 0000000052 483654D0 483655A4 001 -------- -------- 43404C54 *Init*

483655A4 0000000204 48365544 4836569C 001 -------- -------- 43404C54 *Init*

4836569C 0000000112 483655A4 48365738 001 -------- -------- 43404C54 *Init*

48365738 0000000024 4836569C 4836577C 001 -------- -------- 43404C54 *Init*

4836577C 0000001552 48365738 48365DB8 001 -------- -------- 404F4FB0 *Init*

48365DB8 0000003000 4836577C 4836699C 001 -------- -------- 40500E7C Transport ERP id cb chunk

4836699C 0000000032 48365DB8 483669E8 001 -------- -------- 42075A40 Init

483669E8 0000000032 4836699C 48366A34 001 -------- -------- 4203E7F0 Parser Linkage

48366A34 0000000032 483669E8 48366A80 001 -------- -------- 4203E7F0 Parser Linkage

48366A80 0000000032 48366A34 48366ACC 001 -------- -------- 42075A40 Init

48366ACC 0000000024 48366A80 48366B10 001 -------- -------- 43F76708 *Init*

48366B10 0000000024 48366ACC 48366B54 001 -------- -------- 43F766F4 *Init*

48366B54 0000000024 48366B10 48366B98 001 -------- -------- 43F76728 *Init*

48366B98 0000000024 48366B54 48366BDC 001 -------- -------- 43F76708 *Init*

48366BDC 0000000024 48366B98 48366C20 001 -------- -------- 43F766F4 *Init*

48366C20 0000000024 48366BDC 48366C64 001 -------- -------- 43F76728 *Init*

48366C64 0000000024 48366C20 48366CA8 001 -------- -------- 43F76708 *Init*

48366CA8 0000000024 48366C64 48366CEC 001 -------- -------- 43F766F4 *Init*

48366CEC 0000000024 48366CA8 48366D30 001 -------- -------- 43F76728 *Init*

48366D30 0000000024 48366CEC 48366D74 001 -------- -------- 43F76708 *Init*

48366D74 0000000024 48366D30 48366DB8 001 -------- -------- 43F766F4 *Init*

48366DB8 0000000024 48366D74 48366DFC 001 -------- -------- 43F76728 *Init*

48366DFC 0000000024 48366DB8 48366E40 001 -------- -------- 43F76708 *Init*

48366E40 0000000024 48366DFC 48366E84 001 -------- -------- 43F766F4 *Init*

48366E84 0000000024 48366E40 48366EC8 001 -------- -------- 43F76728 *Init*

48366EC8 0000000024 48366E84 48366F0C 001 -------- -------- 43F76708 *Init*

48366F0C 0000000024 48366EC8 48366F50 001 -------- -------- 43F766F4 *Init*

48366F50 0000000024 48366F0C 48366F94 001 -------- -------- 43F76728 *Init*

48366F94 0000000024 48366F50 48366FD8 001 -------- -------- 43F76708 *Init*

48366FD8 0000000024 48366F94 4836701C 001 -------- -------- 43F766F4 *Init*

4836701C 0000000024 48366FD8 48367060 001 -------- -------- 43F76728 *Init*

48367060 0000000024 4836701C 483670A4 001 -------- -------- 43F76708 *Init*

483670A4 0000000024 48367060 483670E8 001 -------- -------- 43F766F4 *Init*

483670E8 0000000024 483670A4 4836712C 001 -------- -------- 43F76728 *Init*

4836712C 0000000024 483670E8 48367170 001 -------- -------- 43F76708 *Init*

48367170 0000000024 4836712C 483671B4 001 -------- -------- 43F766F4 *Init*

483671B4 0000000024 48367170 483671F8 001 -------- -------- 43F76728 *Init*

483671F8 0000000024 483671B4 4836723C 001 -------- -------- 43F76708 *Init*

4836723C 0000000024 483671F8 48367280 001 -------- -------- 43F766F4 *Init*

48367280 0000000024 4836723C 483672C4 001 -------- -------- 43F76728 *Init*

483672C4 0000000024 48367280 48367308 001 -------- -------- 43F76708 *Init*

48367308 0000000024 483672C4 4836734C 001 -------- -------- 43F766F4 *Init*

4836734C 0000000024 48367308 48367390 001 -------- -------- 43F76728 *Init*

48367390 0000000024 4836734C 483673D4 001 -------- -------- 43F76708 *Init*

483673D4 0000000024 48367390 48367418 001 -------- -------- 43F766F4 *Init*

48367418 0000000024 483673D4 4836745C 001 -------- -------- 43F76728 *Init*

4836745C 0000000024 48367418 483674A0 001 -------- -------- 43F76708 *Init*

483674A0 0000000024 4836745C 483674E4 001 -------- -------- 43F766F4 *Init*

483674E4 0000000024 483674A0 48367528 001 -------- -------- 43F76728 *Init*

48367528 0000000024 483674E4 4836756C 001 -------- -------- 43F76708 *Init*

4836756C 0000000024 48367528 483675B0 001 -------- -------- 43F766F4 *Init*

483675B0 0000000024 4836756C 483675F4 001 -------- -------- 43F76728 *Init*

483675F4 0000000024 483675B0 48367638 001 -------- -------- 43F76708 *Init*

48367638 0000000024 483675F4 4836767C 001 -------- -------- 43F766F4 *Init*

4836767C 0000000024 48367638 483676C0 001 -------- -------- 43F76728 *Init*

483676C0 0000000024 4836767C 48367704 001 -------- -------- 43F76708 *Init*

48367704 0000000024 483676C0 48367748 001 -------- -------- 43F766F4 *Init*

48367748 0000000024 48367704 4836778C 001 -------- -------- 43F76728 *Init*

4836778C 0000000024 48367748 483677D0 001 -------- -------- 43F76708 *Init*

483677D0 0000000024 4836778C 48367814 001 -------- -------- 43F766F4 *Init*

48367814 0000000024 483677D0 48367858 001 -------- -------- 43F76728 *Init*

48367858 0000000024 48367814 4836789C 001 -------- -------- 43F76708 *Init*

4836789C 0000000024 48367858 483678E0 001 -------- -------- 43F766F4 *Init*

483678E0 0000000024 4836789C 48367924 001 -------- -------- 43F76728 *Init*

48367924 0000000024 483678E0 48367968 001 -------- -------- 43F76708 *Init*

48367968 0000000024 48367924 483679AC 001 -------- -------- 43F766F4 *Init*

483679AC 0000000024 48367968 483679F0 001 -------- -------- 43F76728 *Init*

483679F0 0000000024 483679AC 48367A34 001 -------- -------- 43F76708 *Init*

48367A34 0000000024 483679F0 48367A78 001 -------- -------- 43F766F4 *Init*

48367A78 0000000024 48367A34 48367ABC 001 -------- -------- 43F76728 *Init*

48367ABC 0000000024 48367A78 48367B00 001 -------- -------- 43F76708 *Init*

48367B00 0000000024 48367ABC 48367B44 001 -------- -------- 43F766F4 *Init*

48367B44 0000000024 48367B00 48367B88 001 -------- -------- 43F76728 *Init*

48367B88 0000000024 48367B44 48367BCC 001 -------- -------- 43F76708 *Init*

48367BCC 0000000024 48367B88 48367C10 001 -------- -------- 43F766F4 *Init*

48367C10 0000000024 48367BCC 48367C54 001 -------- -------- 43F76728 *Init*

48367C54 0000000024 48367C10 48367C98 001 -------- -------- 43F76708 *Init*

48367C98 0000000024 48367C54 48367CDC 001 -------- -------- 43F766F4 *Init*

48367CDC 0000000024 48367C98 48367D20 001 -------- -------- 43F76728 *Init*

48367D20 0000000024 48367CDC 48367D64 001 -------- -------- 43F76708 *Init*

48367D64 0000000024 48367D20 48367DA8 001 -------- -------- 43F766F4 *Init*

48367DA8 0000000024 48367D64 48367DEC 001 -------- -------- 43F76728 *Init*

48367DEC 0000000024 48367DA8 48367E30 001 -------- -------- 43F76708 *Init*

48367E30 0000000024 48367DEC 48367E74 001 -------- -------- 43F766F4 *Init*

48367E74 0000000024 48367E30 48367EB8 001 -------- -------- 43F76728 *Init*

48367EB8 0000000024 48367E74 48367EFC 001 -------- -------- 43F76708 *Init*

48367EFC 0000000024 48367EB8 48367F40 001 -------- -------- 43F766F4 *Init*

48367F40 0000000024 48367EFC 48367F84 001 -------- -------- 43F76728 *Init*

48367F84 0000000024 48367F40 48367FC8 001 -------- -------- 43F76708 *Init*

48367FC8 0000000024 48367F84 4836800C 001 -------- -------- 43F766F4 *Init*

4836800C 0000000024 48367FC8 48368050 001 -------- -------- 43F76728 *Init*

48368050 0000000024 4836800C 48368094 001 -------- -------- 43F76708 *Init*

48368094 0000000024 48368050 483680D8 001 -------- -------- 43F766F4 *Init*

483680D8 0000000024 48368094 4836811C 001 -------- -------- 43F76728 *Init*

4836811C 0000000024 483680D8 48368160 001 -------- -------- 43F76708 *Init*

48368160 0000000024 4836811C 483681A4 001 -------- -------- 43F766F4 *Init*

483681A4 0000000024 48368160 483681E8 001 -------- -------- 43F76728 *Init*

...

------------------------------------------------------------------------------------------

Bug in IOS Version? More RAM is not possible in SUP720-3BXL.

Thanks,

Thomas

Leo Laohoo · ‎09-18-2014

You've got Traceback errors. It's a bug. You may need to reload your chassis.

Raise a TAC Case using this thread so TAC can determine what process or configuration is chewing up your memory.

kgtnewmedia · ‎09-18-2014

Thanks for the explanation. Unfortunately there is no TAC access. Would it help to upgrade to the latest image? (15.1(2) SY3)

Thanks,

Thomas

Leo Laohoo · ‎09-18-2014

You could try upgrading the IOS.

kgtnewmedia · ‎09-24-2014

No luck with upgrading the IOS version to the latest one. (15.1(2) SY3) After booting the new image some of the BGP sessions was flapping and only 10MB RAM was available. Any hintes?

Regards,

Thomas

Leo Laohoo · ‎09-25-2014

Tracebacks and BGP flapping are not necessarily related.

After the IOS upgrade, did you see any more Tracebacks?

kgtnewmedia · ‎09-25-2014

> Tracebacks and BGP flapping are not necessarily related.

BGP flapps because there was no more RAM available. (according to the log "ip cef was disabled" because of low memory) After reactivating "ip cef distributed" the BGP sessions becomes stable.

> After the IOS upgrade, did you see any more Tracebacks?

I did not see any tracebacks in the log. Because of having much lower memory available with the new image (only 10-15MB RAM after booting) I decide to downgrade and boot with the previous version.

InayathUlla Sharieff · ‎09-19-2014

Thomas,

Yes you can thats the latest ios on 15 release.

Before that you can give a try to reload the box which would resolve this memory issue.

HTH

kgtnewmedia · ‎09-19-2014

Personally I would like to "reload and upgrade", because I don't know if the bug affects the system again after only reloading the chassis.

Regards,

Thomas

InayathUlla Sharieff · ‎09-19-2014

Yes as I said earlier reload the box so that the issue is resolved and you are good iwth your routing table.

HTH

******Plz dont forget to rate all usefull posts*******

kgtnewmedia · ‎09-19-2014

Thanks. Will perfom an upgrade. (hopefully that the bug is fixed in the current code)

Regards,

Thomas

InayathUlla Sharieff · ‎09-19-2014

Bugs are there but to disclose that we need to work more on the traceback hence open the tac case.

If no contrct with cisco then go ahead and reload the box and upgrade the ios to the latest version.

HTH

No memory available on a SUP720-3BXL with 1GB RAM