Re: VPN rebooting every 2 days

senthil.natarajan · ‎09-16-2002

Hi

VPN 3015 is rebooting every 2 days. Here is the dump log. Have you people come across any bug relating to this issue. Thanks in advance.

Senthil

CRASHDUMP OF 'Cisco_3015'

PLATFORM: CXX (board rev 3)

TIME:

Crash Time: 09/12/2002 20:25:29 (up 2d:0h:21m:51s)

Start Time: 09/10/2002 20:03:41

VERSION:

Bootcode:

Cisco Systems, Inc./VPN 3000 Concentrator Series Version 2.5.Rel Jun 21 2000 18:57:52

Software:

Cisco Systems, Inc./VPN 3000 Concentrator Version 3.5.Rel built by vmurphy on Nov 27 2001 11:22:31

Loader:

Cisco Systems, Inc./VPN 3000 Concentrator Series Version 3.0.Rel Feb 26 2001 11:12:30

CALL STACK:

Exception Type: 0x00000700/Program was taken from 0x0044EBFC SP = 010BFC60

Callers PC level 1 = 0xFFFFFFFC SP = 010BFC78

Callers PC level 2 = 0x0044EDF8 SP = 010BFC90

Callers PC level 3 = 0x003FBAB8 SP = 010BFCB8

Callers PC level 4 = 0x003EBDA0 SP = 010BFCF0

Callers PC level 5 = 0x0038E1E4 SP = 010BFD00

Callers PC level 6 = 0x0040753C SP = 010BFD50

Callers PC level 7 = 0x0040885C SP = 010BFD70

End of stack frame(s)

REGISTERS:

CR =40000200 XER =00000000 LR =0044EDFC

CTR =0040753C MSR =00083930

R0 =0044EDFC 010BFC60 006A7040 00000000 0077D510 01FC8B18 0077D510 00000000

R8 =00000008 03EA9A2C 00000000 00000000 007E7895 0077986C 00000000 00000000

R16 =00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

R24 =00000000 00000000 00000000 00000000 007280AC 006E9310 000006A1 01FC3E88

IP =0044EBFC-0044EBFC: 919F001C

HEAP MEMORY:

Memory corruption found!

THE FOLLOWING INFORMATION MAY BE INVALID!

Memory Buffer = [0x01FC3E88]

Previous = [0x00000000] Next = [0x01FC8B18]

TID = [0x0000003D] CPC1 = [0x003FBAB8] CPC2 = [0x003EBDA0]

Size = [4176568] SIG1 = [0xfacedbad] SIG2 = [0xfacedbad]

Corruption found: skipping remaining memory tests

Dumping corrupted allocated block

Dumping memory header

0000: FACEDBAD 01FC8B18 00000000 0000003D ...............=

0010: 003FBAB8 003EBDA0 003FBAB8 00803C30 .?...>...?....<0

Dumping first 68 bytes of corrupt block

0000: 0008000E 07374708 00004002 00000000 .....7G...@.....

0010: 00FD3D20 000FE260 07D2090C 00000000 ..= ...`........

0020: 00FD3CA0 00000000 00000019 05312452 ..<..........1$R

0030: 00000000 00000000 00000000 00000000 ................

0040: 05312452 .1$R

BUFFERS:

Pool BufSize NBufs Free Alloc UseCnt Fail LW Low Index

POOL 0x0800 05000 03797 01203 00572579 00000000 00010 03781

POOL 0x2800 00025 00025 00000 00000007 00000000 00010 00024

Allocated buffer histogram

Buffer pool size = 2048:

01) CPC: 00015f2c, flags: 00000010, count: 1200

02) CPC: 00015f2c, flags: 80000080, count: 1

03) CPC: 00015f2c, flags: 00000084, count: 1

04) CPC: 00015f2c, flags: 00000080, count: 1

Buffer pool size = 10240:

TIMERS:

Active Timers: [17] Inactive Timers: [10]

SYSTEM Qs:

System Queues: Active: 12 Free: 60788

ALTIGA SEMAPHORES:

The current number of created semaphores is: 29

The total number of semaphore creates is: 29

The total number of semaphore frees is: 0

The number of deletes while a task was waiting is: 0

Name State Type Tid # GetCPC1 FreeCPC1 GetCPC2 FreeCPC2 Waiting

-----------------------------------------------------------------------------

L2TP_HELLO Avail ~Del 00540000 0 00000000 00000000 00000000 00000000 None

XML_MIB Avail ~Del 005B0000 0 00000000 00000000 00000000 00000000 None

SSH_SESS Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

FW_SEM Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

ASDB Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

HTTP_SESS Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

IKE_CONN Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

NWLIST_SEM1 Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

ACTN Avail ~Del 00020000 0 00000000 00000000 00000000 00000000 None

ENRO Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

CERT Avail ~Del 00020000 0 00000000 00000000 00000000 00000000 None

ENRL_R_LOCK Avail ~Del 00020000 0 00000000 00000000 00000000 00000000 None

ENRL_W_LOCK Avail ~Del 00020000 0 00000000 00000000 00000000 00000000 None

CA_WRITE Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

CA_READ Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

CLIENT Avail ~Del 00020000 0 00000000 00000000 00000000 00000000 None

ASS1 Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

KEY_SEM2 Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

KEY_SEM1 Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

FRAG Avail ~Del 00020000 0 00000000 00000000 00000000 00000000 None

NATF Avail ~Del 00020000 0 00000000 00000000 00000000 00000000 None

NATS Avail ~Del 00020000 0 00000000 00000000 00000000 00000000 None

NATL Avail ~Del 00020000 0 00000000 00000000 00000000 00000000 None

CTCP_PORT Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

CTCP_CON Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

FILTER_SEM1 Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

WINBOND Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

CMPS Avail ~Del 00000000 0 00000000 00000000 00000000 00000000 None

CGXS Avail ~Del 00000000 0 00000000 00000000 00535610 00000000 None

PSOS TASKS:

Running task: IPRT (003d0000)

pSOS+ TASKS

NAME TID PRIO MODE STATUS SUSP TICKS PARAMS

IDLE 00010000 000 2000 Ready NO

ROOT 00020000 230 2000 Ready YES

pINP 00040000 246 2001 Ready YES

pOUT 00050000 245 2001 Ready YES

pMNG 00030000 247 2001 EvWait NO NA Event: 0x00000003

pROC 00060000 244 2001 Ready YES

RM 00330000 210 2002 QWait NO NA MgQ ID: 0x00110000

DMST 00340000 110 2002 Ready NO

DBGT 00350000 250 2002 QWait NO NA MgQ ID: 0x00150000

PERH 00360000 254 2002 WkAfter NO 35

CAPI 00370000 200 2002 QWait NO NA MgQ ID: 0x00210000

IPRT 003d0000 220 2003 Ready NO

EVNT 003e0000 200 2002 QWait NO NA MgQ ID: 0x00170000

SDIN 003f0000 200 2002 EvWait NO NA Event: 0x00000003

HTTP 00400000 200 2002 EvWait NO NA Event: 0x0000003F

PPP_ 00410000 150 2002 QWait NO NA MgQ ID: 0x00180000

IKED 00420000 220 2002 QWait NO NA MgQ ID: 0x001C0000

IPAD 00430000 204 2002 QWait NO NA MgQ ID: 0x00200000

SLOG 00440000 120 2002 QWait NO NA MgQ ID: 0x00230000

PERL 00450000 120 2002 WkAfter NO 69

CFG_ 00460000 220 2002 EvWait NO NA Event: 0x0000001F

WAHA 00470000 200 2002 QWait NO NA MgQ ID: 0x00260000

SESS 00480000 150 2002 QWait NO NA MgQ ID: 0x00270000

DHCP 00490000 150 2002 QWait NO NA MgQ ID: 0x00220000

AUPD 004a0000 210 2002 QWait NO NA MgQ ID: 0x002A0000

PSHD 004b0000 200 2002 QWait NO NA MgQ ID: 0x001A0000

PSHC 004c0000 200 2006 QWait NO NA MgQ ID: 0x00090000

TFTP 004f0000 120 2002 Ready YES

FTPD 00500000 120 2002 EvWait NO NA Event: 0x20000000

TELD 00510000 200 2002 QWait NO NA MgQ ID: 0x001B0000

SNMP 00520000 110 2002 QWait NO NA MgQ ID: 0x001E0000

PPTP 00530000 150 2002 QWait NO NA MgQ ID: 0x00160000

L2TP 00540000 150 2002 QWait NO NA MgQ ID: 0x001D0000

AUTH 00550000 200 2002 QWait NO NA MgQ ID: 0x00120000

SMTP 00560000 200 2002 QWait NO NA MgQ ID: 0x001F0000

CERT 00570000 200 2002 QWait NO NA MgQ ID: 0x00240000

CLNT 00580000 200 2002 QWait NO NA MgQ ID: 0x00250000

SSL 00590000 200 2002 EvWait NO NA Event: 0x00000003

SSH1 005a0000 200 2002 EvWait NO NA Event: 0x0000000F

XMLI 005b0000 200 2002 WkAfter NO 2177

FWT 005c0000 230 2002 EvWait NO NA Event: 0x20000000

RADP 005d0000 200 2002 QWait NO NA MgQ ID: 0x002E0000

VCA 005f0000 210 2002 EvWait NO NA Event: 0x20000000

---- Attributes for Task ID: 003d0000 ----

Name: IPRT

Attributes: LOCAL

Start Addr: 0x00408788

Waiting Task ID: 0x00000000

Status: Ready

Priority: 220

Base Priority: 220

Initial Priority: 220

Event Wait Condition: 2

Time Slice Quantum: 10

Time Slice Remaining: 10

Task Blocked Object: 0x00130000

Event Tasks Waiting: 0x00000000

Events Caught: 1073741824

Events Received: 0

Supervisor Stack Size: 0x00004000

Current SSP: 0x010BFC98

Initial SSP: 0x010BFD80

TID Name SSPBegin SSPEnd SSP%

00010000 IDLE 07FFDD80 07FFBD80 05%

00020000 ROOT 07FF9D80 07FF7D80 29%

00040000 pINP 07FF4D80 07FF3D80 04%

00050000 pOUT 07FF2D80 07FF1D80 04%

00030000 pMNG 07FF6D80 07FF5D80 11%

00060000 pROC 07FF1980 07FEFD80 02%

00330000 RM 010E7280 010E5D80 21%

00340000 DMST 010E3D80 010DFD80 17%

00350000 DBGT 010DED80 010DDD80 32%

00360000 PERH 010DCD80 010DBD80 10%

00370000 CAPI 010D9D80 010D7D80 28%

003D0000 IPRT 010BFD80 010BBD80 20%

003E0000 EVNT 010B9D80 010B5D80 08%

003F0000 SDIN 010B4D80 010B3D80 27%

00400000 HTTP 010B1D80 010ADD80 25%

00410000 PPP_ 010ABD80 010A9D80 14%

00420000 IKED 010A7D80 010A3D80 25%

00430000 IPAD 010A1D80 0109FD80 10%

00440000 SLOG 0109DD80 0109BD80 05%

00450000 PERL 01099D80 01095D80 11%

00460000 CFG_ 01093D80 01091D80 16%

00470000 WAHA 0108FD80 0108BD80 08%

00480000 SESS 01089D80 01087D80 10%

00490000 DHCP 01085D80 0107FD80 26%

004A0000 AUPD 0107DD80 0107BD80 13%

004B0000 PSHD 0107A780 01079D80 19%

004C0000 PSHC 01077D80 01073D80 11%

004F0000 TFTP 00FD6D80 00FD5D80 29%

00500000 FTPD 00FD3D80 00FD1D80 15%

00510000 TELD 00FD0D80 00FCFD80 31%

00520000 SNMP 00FCDD80 00FCBD80 27%

00530000 PPTP 00FC9D80 00FC7D80 15%

00540000 L2TP 00FC5D80 00FC3D80 16%

00550000 AUTH 00FC1D80 00FBDD80 28%

00560000 SMTP 00FBBD80 00FB9D80 13%

00570000 CERT 00FB7D80 00FB5D80 40%

00580000 CLNT 00FB3D80 00FB1D80 05%

00590000 SSL 00FAFD80 00FADD80 47%

005A0000 SSH1 00FABD80 00FA7D80 12%

005B0000 XMLI 00FA5D80 00FA1D80 06%

005C0000 FWT 00F9FD80 00F9BD80 08%

005D0000 RADP 00F99D80 00F97D80 16%

005F0000 VCA 00F91D80 00F8DD80 03%

PSOS Q:

pSOS+ MESSAGE QUEUES

Name QID TkWait TQLn MQLn MQMx Total Fails nomgb qfull other

RMQ1 110000 330000 1 0 1388 25 0 0 0 0

ASQ1 120000 550000 1 0 1388 48 0 0 0 0

IPRQ 130000 000000 0 0 1D4C 1611535 0 0 0 0

HTDQ 140000 000000 0 0 3E8 6 0 0 0 0

DBGQ 150000 350000 1 0 C8 9 0 0 0 0

PPTQ 160000 530000 1 0 1388 2 0 0 0 0

EMQ1 170000 3E0000 1 0 200 18356 0 0 0 0

PPPQ 180000 410000 1 0 1388 0 0 0 0 0

ASQ2 190000 000000 0 0 1388 21 0 0 0 0

PSHQ 1A0000 4B0000 1 0 64 0 0 0 0 0

TNDQ 1B0000 510000 1 0 64 0 0 0 0 0

IKEQ 1C0000 420000 1 0 186A 20023 0 0 0 0

L2TQ 1D0000 540000 1 0 64 17409 0 0 0 0

SNMP 1E0000 520000 1 0 64 1 0 0 0 0

SMTP 1F0000 560000 1 0 64 0 0 0 0 0

IPAQ 200000 430000 1 0 64 20 0 0 0 0

CAPI 210000 370000 1 0 64 0 0 0 0 0

DHCP 220000 490000 1 0 1388 11663 0 0 0 0

SLOG 230000 440000 1 0 64 0 0 0 0 0

CERT 240000 570000 1 0 64 2 0 0 0 0

CLNT 250000 580000 1 0 64 0 0 0 0 0

WHQ1 260000 470000 1 0 1388 8 0 0 0 0

SESS 270000 480000 1 0 3E8 5803 0 0 0 0

SSHQ 280000 000000 0 0 3E8 0 0 0 0 0

CFGQ 290000 000000 0 0 64 0 0 0 0 0

AUPQ 2A0000 4A0000 1 0 64 7 0 0 0 0

POEQ 2B0000 000000 0 0 64 0 0 0 0 0

XMLI 2C0000 000000 0 0 64 0 0 0 0 0

FWQ1 2D0000 000000 0 0 64 0 0 0 0 0

RADQ 2E0000 5D0000 1 0 64 0 0 0 0 0

PSOS REGIONS:

pSOS+ REGIONS

Name I.D. QType Address Length UnitSize FreeByte LrgContig TQLen DO?

RN#0 0000 FIFO 0096DD7C 0767a000 00002000 00570000 00568000 00000000 No

EVENTS:

Dumping queued event strings ...

Critical section info:

Isr: 0

Critical section: 1

OS critical section: 0

Critical section debug info unavailable in this build.

REGULAR REGS

R01: 010bfc60

010BFC60: 010BFC78 00000000 007280AC 003FBAB8 ...x.....r...?..

010BFC70: 0000002C 01FC3E88 010BFC90 0044EDFC ...,..>......D..

010BFC80: 010BFCC0 07374708 0000002C 010F7750 .....7G....,..wP

010BFC90: 010BFCB8 003FBABC 010BFD50 0008DDBC .....?.....P....

R02: 006a7040

006A7040: 02000000 00046FC0 00000000 00046FE8 ......o.......o.

006A7050: 02000000 00047014 02000000 00047040 ......p.......p@

006A7060: 02000000 00047068 00000000 00047094 ......ph......p.

006A7070: 04000000 000470BC 02000000 000470EC ......p.......p.

R04: 0077d510

0077D510: 000006A1 0000073E 00000000 01FC3E88 .......>......>.

0077D520: 000000B8 000000D6 00000000 02F1C878 ...............x

0077D530: 0000004E 00000665 00000000 037F6054 ...N...e......`T

0077D540: 00000072 000000A1 00000000 03EAEF88 ...r............

R05: 01fc8b18

01FC8B18: FACEDBAD 01FC8730 01FC3E88 00000042 .......0..>....B

01FC8B28: 004A9FDC 004AA0B0 0000001E 009540EC .J...J........@.

01FC8B38: 3231362E 3235352E 3231322E 31323100 216.255.212.121.

01FC8B48: 00000000 00000000 00000000 00000531 ...............1

R06: 0077d510

0077D510: 000006A1 0000073E 00000000 01FC3E88 .......>......>.

0077D520: 000000B8 000000D6 00000000 02F1C878 ...............x

0077D530: 0000004E 00000665 00000000 037F6054 ...N...e......`T

0077D540: 00000072 000000A1 00000000 03EAEF88 ...r............

R09: 03ea9a2c

03EA9A2C: 00000000 00000000 003EBD64 03EA9A08 .........>.d....

03EA9A3C: 00000000 00000005 00000001 00000040 ...............@

03EA9A4C: 00000000 037A20CC 00000000 00000000 .....z .........

03EA9A5C: 00000000 00000000 00000000 00000000 ................

R12: 007e7895

007E7895: 00000000 00000000 00000000 00000000 ................

007E78A5: 00000000 00000000 00000000 00000000 ................

007E78B5: 00000000 00000000 00000000 00000000 ................

007E78C5: 00000000 00000000 00000000 00000000 ................

R13: 0077986c

0077986C: 00000000 00000000 00000000 00000000 ................

0077987C: 00000000 00000000 00000000 00000000 ................

0077988C: 00000000 00000000 00000000 00000000 ................

0077989C: 00000000 00000000 00000000 00000000 ................

R28: 007280ac

007280AC: 00000000 01FC3E88 00000000 00000001 ......>.........

007280BC: 00000000 00000000 00000000 00000000 ................

007280CC: 00000000 00000000 00000000 00000000 ................

007280DC: 00000000 00454594 004534C0 00452800 .....EE..E4..E(.

R29: 006e9310

006E9310: 00000000 00000000 000329AD 00000001 ..........).....

006E9320: 000329A1 00000001 BB000000 00000000 ..).............

006E9330: 00000000 00000000 00000002 00000000 ................

006E9340: 00000064 01B95DA4 00000000 00000000 ...d..].........

R31: 01fc3e88

01FC3E88: FACEDBAD 01FC8B18 00000000 0000003D ...............=

01FC3E98: 003FBAB8 003EBDA0 003FBAB8 00803C30 .?...>...?....<0

01FC3EA8: 0008000E 07374708 00004002 00000000 .....7G...@.....

01FC3EB8: 00FD3D20 000FE260 07D2090C 00000000 ..= ...`........

STACK FRAME:

Frame 0:

010BFC60: 010BFC78 00000000 007280AC 003FBAB8 ...x.....r...?..

010BFC70: 0000002C 01FC3E88 ...,..>.

Frame 1:

010BFC78: 010BFC90 0044EDFC 010BFCC0 07374708 .....D.......7G.

010BFC88: 0000002C 010F7750 ...,..wP

Frame 2:

010BFC90: 010BFCB8 003FBABC 010BFD50 0008DDBC .....?.....P....

010BFCA0: 0008DF4C 001189D0 00000000 00000000 ...L............

010BFCB0: 003EBD64 010F7750 .>.d..wP

Frame 3:

010BFCB8: 010BFCF0 003EBDA4 0008000E 00000000 .....>..........

010BFCC8: 00000000 007207B8 00000000 03EA0878 .....r.........x

010BFCD8: 010BFCE8 0077986C 00000000 00000000 .....w.l........

010BFCE8: 010BFD00 00000000 ........

Frame 4:

010BFCF0: 010BFD00 0038E1E8 00000000 010BFD58 .....8.........X

Frame 5:

010BFD00: 010BFD50 00407540 00000000 00000000 ...P.@u@........

010BFD10: 00000000 00000000 00000000 00000000 ................

010BFD20: 00000000 0079C4C4 20000200 007D4AA8 .....y.. ....}J.

010BFD30: 00000000 00408850 0006ED74 00000000 .....@.P...t....

010BFD40: 00000000 00000000 00000000 0079C4C4 .............y..

Frame 6:

010BFD50: 010BFD70 00408860 00000000 00000000 ...p.@.`........

010BFD60: 00000000 00000000 00000000 00000000 ................

Frame 7:

010BFD70: 00000000 DEADDEAD 00000000 00000000 ................

010BFD80: 00000001 00000000 00000000 00000000 ................

010BFD90: 012BFDF2 00558A77 FFFFF7FE E0A8D21C .+...U.w........

010BFDA0: 058E7CFB 00010000 00000000 00000000 ..|.............

010BFDB0: 00000000 00000000 00000000 00000000 ................

010BFDC0: 00000000 00000000 00000000 00000000 ................

010BFDD0: 00000000 00000000 00000000 00000000 ................

010BFDE0: 00000000 00000000 00000000 00000000 ................

awaheed · ‎09-16-2002

Hi,

The issue has been fixed already in the latest version of the Concentrator code, use either v3.5.5 or v3.6.1 and you should not see this problem again. In case you do go ahead and open a TAC case on this. And this issue occured only in some software states and has only been seen in rare cases.

Hope this helps,

Regards,

Aamir

-=-=-