Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Community Member

ME3400 crash unexpectedly

Hi,

has anybody experience this, Our ME3400 crashed unexpectedly and got the below errors:

meQCY0290A#sh ver

Cisco IOS Software, ME340x Software (ME340x-METROACCESSK9-M), Version 12.2(55)SE, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2010 by Cisco Systems, Inc.

Compiled Sun 08-Aug-10 00:21 by prod_rel_team

Image text-base: 0x01000000, data-base: 0x02600000

ROM: Bootstrap program is ME340x boot loader

BOOTLDR: ME340x Boot Loader (ME340x-HBOOT-M) Version 12.2(44r)EY, RELEASE SOFTWARE (fc1)

meQCY0290A uptime is 7 minutes

System returned to ROM by address error at PC 0x22797A8, address 0x0

System restarted at 05:47:52 PH Thu Mar 15 2012

System image file is "flash:me340x-metroaccessk9-mz.122-55.SE.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco ME-3400E-24TS-M (PowerPC405) processor (revision A0) with 131072K bytes of memory.

Processor board ID FOC1339X122

Last reset from address error

2 Virtual Ethernet interfaces

25 FastEthernet interfaces

2 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

meQCY0290A#sh log | beg Mar 15

000037: .Mar 15 05:48:59.607 PH: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed state to up

000038: .Mar 15 05:48:59.607 PH: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up

000039: .Mar 15 05:48:59.607 PH: %LINK-3-UPDOWN: Interface FastEthernet0/2, changed state to up

000040: .Mar 15 05:48:59.607 PH: %LINK-3-UPDOWN: Interface FastEthernet0/3, changed state to up

000041: .Mar 15 05:48:59.607 PH: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up

000042: Mar 15 05:49:00.614 PH: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/2, changed state to up

000043: Mar 15 05:49:00.631 PH: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up

000044: Mar 15 05:49:00.647 PH: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to up

000045: Mar 15 05:49:00.664 PH: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/3, changed state to up

000046: Mar 15 05:49:02.157 PH: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.61.11.13 Port 514 started - reconnection

000047: Mar 15 05:49:07.165 PH: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.61.15.12 Port 514 started - reconnection

000048: Mar 15 05:49:08.407 PH: %REP-4-LINKSTATUS: GigabitEthernet0/2 (segment 5) is operational

000049: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: System previously crashed with the following message:

000050: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Cisco IOS Software, ME340x Software (ME340x-METROACCESSK9-M), Version 12.2(55)SE, RELEASE SOFTWARE (fc2)

000051: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Technical Support: http://www.cisco.com/techsupport

000052: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Copyright (c) 1986-2010 by Cisco Systems, Inc.

000053: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Compiled Sun 08-Aug-10 00:21 by prod_rel_team

000054: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: 

000055: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

000056: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: 

000057: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: SRR0 = 0x01516AF4  SRR1 = 0x00029230  SRR2 = 0x022797A8  SRR3 = 0x00029230

000058: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: ESR = 0x00000000  DEAR = 0x00000000  TSR = 0x84000000  DBSR = 0x10000000

000059: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: 

000060: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: CPU Register Context:

000061: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Vector = 0x00002000  PC = 0x022797A8  MSR = 0x00029230  CR = 0x22004042

000062: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: LR = 0x0227976C  CTR = 0x01F9E470  XER = 0x4000007E

000063: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R0 = 0x0227976C  R1 = 0x0442F860  R2 = 0x00000000  R3 = 0x03F7565C

000064: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R4 = 0xFFFFFFFE  R5 = 0x00000000  R6 = 0x0442F848  R7 = 0x00000000

000065: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R8 = 0x00000000  R9 = 0x00000000  R10 = 0x02CE0000  R11 = 0x03F75658

000066: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R12 = 0xC9C25312  R13 = 0x00110000  R14 = 0x0241C768  R15 = 0x00000000

000067: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R16 = 0x0413D854  R17 = 0x00000000  R18 = 0x00000000  R19 = 0x00000000

000068: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R20 = 0x00000000  R21 = 0xFFFFFFFF  R22 = 0x00000000  R23 = 0x00000012

000069: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R24 = 0x000000EA  R25 = 0x00000000  R26 = 0x0442F914  R27 = 0x00000000

000070: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: R28 = 0x00000000  R29 = 0x031A0000  R30 = 0x00000000  R31 = 0x00000000

000071: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: 

000072: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Stack trace:

000073: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: PC = 0x022797A8, SP = 0x0442F860

000074: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Frame 00: SP = 0x0442F870    PC = 0x0227976C

000075: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Frame 01: SP = 0x0442F888    PC = 0x02275434

000076: Mar 15 05:49:41.449 PH: %PLATFORM-1-CRASHED: Frame 02: SP = 0x0442F8A8    PC = 0x02275930

000077: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 03: SP = 0x0442F8C8    PC = 0x0222862C

000078: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 04: SP = 0x0442F908    PC = 0x02416D18

000079: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 05: SP = 0x0442F940    PC = 0x02417378

000080: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 06: SP = 0x0442F950    PC = 0x02423888

000081: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 07: SP = 0x0442F968    PC = 0x02424FEC

000082: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 08: SP = 0x0442F980    PC = 0x02424524

000083: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 09: SP = 0x0442F990    PC = 0x02424CD4

000084: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 10: SP = 0x0442F9B0    PC = 0x02424D90

000085: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 11: SP = 0x0442F9E8    PC = 0x02417368

000086: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 12: SP = 0x0442F9F8    PC = 0x02423888

000087: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 13: SP = 0x0442FA10    PC = 0x02424FEC

000088: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 14: SP = 0x0442FA28    PC = 0x02424524

000089: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: Frame 15: SP = 0x0442FA38    PC = 0x02424CD4

000090: Mar 15 05:49:41.457 PH: %PLATFORM-1-CRASHED: 

meQCY0290A#     

Please also take note of the start of the Crashinfo file:

011771: Mar 15 05:46:41.544 PH: %SYS-6-STACKLOW: Stack for process SSH Process running low, 0/12000

=== Start of Crashinfo Collection (05:46:41 PH Thu Mar 15 2012) ===

For image:

Cisco IOS Software, ME340x Software (ME340x-METROACCESSK9-M), Version 12.2(55)SE, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2010 by Cisco Systems, Inc.

Compiled Sun 08-Aug-10 00:21 by prod_rel_team

  42D6B20  41D99F4  41D9A2C

442FA10:  442FA28  2424524        0        1  42D6B20       5D  442FA38  2424CD4

442FA30:  42D6B20       5D  442FA58  2424D90  442FA98  3F75749      B63  40F44EC

442FA50:  42D6B20        0  442FA90  2417368        6        1        0        1

442FA70:        1        1  442FB40       5E  41D9A2C  41D99F4  42D6B20        1

Thanks,

Bryan

1 ACCEPTED SOLUTION

Accepted Solutions
Cisco Employee

Re: ME3400 crash unexpectedly

Hello Bryan,

this is really not a hardware issue, it is a software issue, according to the extended crashinfo file, you are hitting bug

CSCth27817 (duplicated by CSCto60047), explained in the link below:-

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCto60047

CSCth27817

multiple crashes due to SSH Stack Low

Symptom:

A device running IOS may reload unexpectedly with a SYS-6-STACKLOW error message seen int eh crashinfo file.

Conditions:

There must be an active ssh session to or from the router

Workaround:

None

CSCto60047

Chunk corruption crash on trying to abort "show tech" over SSH

Symptoms:

A crash occurs either due to a chunk corruption or at ssh_send_queue_data.

Conditions:

This symptom occurs under the following conditions: 

- An SSH session exists between two routers.

- The show tech command is issued and then aborted.

Workaround:

There is no workaround.

This bug is fixed in 12.2(58)SE1 and later images, I would suggest upgrading the IOS image to 12.2(58)SE2 as it is more stable.

it is not fixed in 12.2(55)SE4, but i am not sure about 12.2(55)SE5.

Have a great day,

Zaid

5 REPLIES
Hall of Fame Super Gold

ME3400 crash unexpectedly

Cisco IOS Software, ME340x Software (ME340x-METROACCESSK9-M), Version 12.2(55)SE, RELEASE SOFTWARE (fc2)

It's an IOS bug alright.

The IOS version sounds like the same type as the 3750.  If this is the case, I'd recommend you upgrade to either 12.2(55)SE4 or SE5.

Community Member

ME3400 crash unexpectedly

Hi Leolaohoo,

I have check the bug tool kit and also the output interpreter tool and it does not lead me to a Cisco Bug.

Thanks,

Bryan

Hall of Fame Super Gold

Re: ME3400 crash unexpectedly

I have check the bug tool kit and also the output interpreter tool and it does not lead me to a Cisco Bug.

I doubt if you'll be able to find it particularly with the hex codes in the output.  You'll need TAC assistance to nail down the exact bug code.

I don't see any signs of hardware failure in your output.

Besides, look at your IOS version.  It's an "SE" or as I'd like to call it "SE0".  It's the first train of the code.  So like any typical software, the first version will always contain bug.  Cisco has made some bug fixes and released them through SE1, SE2, SE3, SE4 and now SE5, hence, I recommended you upgrade to either SE4 or SE5 as one of these two versions may have fixed your crashing issue.

Cisco Employee

Re: ME3400 crash unexpectedly

Hello Bryan,

this is really not a hardware issue, it is a software issue, according to the extended crashinfo file, you are hitting bug

CSCth27817 (duplicated by CSCto60047), explained in the link below:-

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCto60047

CSCth27817

multiple crashes due to SSH Stack Low

Symptom:

A device running IOS may reload unexpectedly with a SYS-6-STACKLOW error message seen int eh crashinfo file.

Conditions:

There must be an active ssh session to or from the router

Workaround:

None

CSCto60047

Chunk corruption crash on trying to abort "show tech" over SSH

Symptoms:

A crash occurs either due to a chunk corruption or at ssh_send_queue_data.

Conditions:

This symptom occurs under the following conditions: 

- An SSH session exists between two routers.

- The show tech command is issued and then aborted.

Workaround:

There is no workaround.

This bug is fixed in 12.2(58)SE1 and later images, I would suggest upgrading the IOS image to 12.2(58)SE2 as it is more stable.

it is not fixed in 12.2(55)SE4, but i am not sure about 12.2(55)SE5.

Have a great day,

Zaid

Community Member

Re: ME3400 crash unexpectedly

Hi Leolaohoo,

Yup I already opened a Cisco TAC for this, It's also good that Zaid Abu also sent the bug details above same as the findings by our Cisco TAC. We willl be trying to upgrade all the ME3400 switches, unfortunately the switches are to many to be upgraded immeadiately. I will give you feedback once all are upgraded.

Thanks for all your comments and all of it are helpful.

Regards,

Bryan

1324
Views
0
Helpful
5
Replies
CreatePlease to create content