Unable to boot 6500 Sup 2

craig.golding · ‎11-27-2005

Hello,

I have a 6506 with a Sup 2 and MSFC2. My new offsider was trying to upgrade the Sup from CatOS to IOS (very unwisely - without the documentation). He deleted the CatOS before copying the IOS to the Sup. On bootup we have the following output:

System Bootstrap, Version 7.1(1)

c6k_sup2 processor with 262144 Kbytes of main memory

Autoboot executing command: "boot bootflash:"

boot: cannot determine first file name on device "bootflash:"

System Bootstrap, Version 7.1(1)

c6k_sup2 processor with 262144 Kbytes of main memory

rommon 1 >

In the Bootflash we have the following:

rommon 2 > dir bootflash:

File size Checksum File name

6238236 bytes (0x5f301c) 0xc3d90a08 cat6000-sup2k8.7-4-3.bin (deleted)

As you can see from the following available commands (sp rommon) there is no undelete:

rommon 3 > ?

alias set and display aliases command

boot boot up an external process

break set/show/clear the breakpoint

confreg configuration register utility

cont continue executing a downloaded image

context display the context of a loaded image

dev list the device table

dir list files in file system

dis disassemble instruction stream

frame print out a selected stack frame

help monitor builtin command help

history monitor command history

meminfo main memory information

repeat repeat a monitor command

reset system reset

set show all monitor variables

stack produce a stack trace

sync write monitor environment to NVRAM

sysret print out info from last system return

unalias unset an alias

xmodem x/y modem download

I have tried booting the new IOS from a flash card but end up with "Unexpected exceptions". Lastly, and perhaps worst, I do not have another copy of the CatOS image other than the deleted image on the bootflash.

Any ideas anyone?

CG

Prashanth Krishnappa · ‎11-27-2005

If you cannot manage to get a flash card with a CAT OS image, XMODEM is the recovery option you have

http://www.cisco.com/warp/public/473/26.html#cat6k_rec

Hopefully your colleague has not deleted the MSFC2 IOS images on the MSFC bootflash. If he has, you will need to XMODEM the MSFC2 bootldr image after you recover the Supervisor using above steps.

craig.golding · ‎11-27-2005

Hi again,

I have tried to xmodem about 6 times this last week and again after your reply but it always fails at 16777334 bytes (about 65%). I thought that bootflash has no room left but the only file in there is the deleted CatOS.

Any other ideas?

amit-singh · ‎11-27-2005

Craig,

Which CatOS you are trying to Xodem.The bytes above shows arnd 16 MB file.Try to Xmodem a small cat OS image, I guess there is a file for 5.1 Mb.

Try to xmodem that small image and it should comeup.

regards,

-amit singh

craig.golding · ‎11-27-2005

Hi Amit,

I don't have a CatOS image at all. The only image I did have was on the sup in bootflash, see below:

rommon 3 > dir bootflash:

File size Checksum File name

6238236 bytes (0x5f301c) 0xc3d90a08 cat6000-sup2k8.7-4-3.bin (deleted)

I was trying to xmodem c6sup22-jsv-mz.121-20.E2.bin which is nearly 26MB (this is the only 6500 IOS image I have). Bootflash size is 32MB. If I had access to a correct CatOS image I would probably be ok. Otherwise I need some sort of work around.

Thanks for your suggestion.

Craig

bbaillie · ‎11-28-2005

If your intent is to migrate the Supervisor to IOS then here is the plan. If the supervisor is currently in a redundant configuration, on the active supervisor disable High availability.

At the ROMMON of the supervisor you want to load IOS type "set" to confirm the current environment variables. The important ones are "CONFIG_FILE=slot0:switch.cfg" and "BOOT=bootflash:cat6000-sup.6-1-2.bin,1"

To change this configuration type "CONFIG_FILE=" and hit carriage return. Type "BOOT=slot0:" and hit carriage return.

Type "sync", Type "reset".

Now the supervisor will boot the first file it finds on the flash card in slot0 without using a configuration file. When the supervisor is up you will need to perform the rest of the migration from CATOS to IOS but ensure you remain cognizant of the difference between "sup-bootflash" (supervisor flash bootable for rommon) and "bootflash" (MSFC flash only addressable when IOS is running and pretty much just for temporary storage in native IOS) and the need to reformat the flash on the supervisor if you intend to use it for storage(different filesystems between CAT and IOS). Your configuration for the switch will now have to be redone from scratch, you will learn to love the "interface range" command during this time.

Start with the commands "boot system flash slot0:" and "boot system flash bootflash:" to ensure the supervisor will boot again but prefer slot0:(which is now sup-slot0: in IOS speak.

Cheers,

Brian

craig.golding · ‎11-28-2005

Hi Brian,

Thanks for your help. I am migrating to IOS to standardise my network - this is the last device and luckily is a spare. I changed the values for 'CONFIG_FILE=' and 'BOOT=' and this manages to boot the IOS from slot0: automatically. However, this is where my next problem starts. The image decompresses ok but then a 'Software-forced reload' occurs and 'Unexpected exceptions' galore. Have a look at the output below - this continues until power is cycled as no breaks are accepted :

rommon 5 > reset

System Bootstrap, Version 7.1(1)

c6k_sup2 processor with 262144 Kbytes of main memory

Autoboot executing command: "boot slot0:"

Self decompressing the image : #################################################

################################################################################

############################################################################## [

OK]

%Software-forced reload

Unexpected exception, CPU signal 23, PC = 0x401252FC

-Traceback= 401252FC 40123478 403F13C4 40196F10 40129F98 4002104C

$0 : 00000000, AT : 42260000, v0 : 42880000, v1 : 00000001

a0 : 42871BD4, a1 : 0000C100, a2 : 00000000, a3 : 40980000

t0 : 0000FD00, t1 : 3401FD01, t2 : 40129908, t3 : FFFF00FF

t4 : 40129908, t5 : 00000007, t6 : 44212800, t7 : 34018000

s0 : 00000000, s1 : 00000000, s2 : 00000020, s3 : 42870000

s4 : 40020F90, s5 : 00000030, s6 : FFFFFFFE, s7 : FFFFFFFD

t8 : 34018000, t9 : FFFFFFFF, k0 : 30419001, k1 : 30410000

gp : 422647C2, sp : 8001FE58, s8 : FFFFFFFB, ra : 40123478

EPC : 401252FC, ErrorEPC : FFFFFFFF, SREG : 3401FD03

MDLO : 0000007D, MDHI : 00000000, BadVaddr : 980B3F2F

Cause 00000024 (Code 0x9): Breakpoint exception

This then changes to :

Unexpected exception, CPU signal 10, PC = 0x0

-Traceback= 0 40124D64 40124ECC 40131820 4013083C 40131504 4013083C 40127FD0

$0 : 00000000, AT : 42260000, v0 : 00000000, v1 : 00000027

a0 : 00000002, a1 : 00008000, a2 : 00000000, a3 : 40980000

t0 : 00000038, t1 : 34018001, t2 : 34018000, t3 : FFFF00FF

t4 : 40129908, t5 : 00000008, t6 : 00000000, t7 : 980B3F2F

s0 : 00000008, s1 : 0000000A, s2 : 0000000A, s3 : 00000008

s4 : 00000038, s5 : 7FFFFFFF, s6 : 00000000, s7 : 40985468

t8 : 34018000, t9 : FFFFFFFF, k0 : 42875C70, k1 : 00000038

gp : 422647C2, sp : 40985258, s8 : FFFFFFFF, ra : 40124D64

EPC : 00000000, ErrorEPC : FFFFFFFF, SREG : 34018003

MDLO : 00000000, MDHI : 00000002, BadVaddr : 00000000

Cause 00000008 (Code 0x2): TLB (load or instruction fetch) exception

And then changes to the following, which continues indefinitely:

-Traceback= 40066F44 40065CBC 40125E78 40125F28 40124BF4 40124EC0 40131820 40130

83C 40131504 4013083C 40127FD0

F2F3B089)@iX : 00000000,

Unexpected exception, CPU signal 10, PC = 0x40066F44

Any ideas? I can't find any help searching the Cisco website.

CG

bbaillie · ‎11-29-2005

The supervisor may still be attempting to use the old configuration from the CATOS and crashing. To make the Supervisor ignore the configuration file and just boot the IOS, from the ROMMON type "confreg 0x2142". Type reset and the supervisor will reboot and load the IOS again. If the problem still persists confirm memory requirements for the IOS version are met, remove all blades from the chassis except the supervisor and try again. Attempt to have only the supervisor and a single power supply in the chassis to ensure the problem can only be one of three, supervisor, chassis, or power supply. The error appears to be very cryptic but narrowing it down to one of three will help pinpoint the problem. The message "software-forced reload" could be memory or the CATOS configuration file so going to a "skinny" system may provide the best method of problem isolation.

Cheers,

Brian

craig.golding · ‎11-29-2005

Hi Brian,

I tried changing confreg to 0x2142 but to no avail - same symptoms. I pulled the other blades from the chassis leaving only the sup, and power supply. The memory sizes, firmware etc of the sup/msfc2 are exactly the same as our other six live supervisors (as well as the IOS on the pcmcia card).

Can you explain what you mean by a "skinny" system? Any more ideas are also appreciated.

Regards,

Craig

bbaillie · ‎11-29-2005

In the old mainframe days a "skinny" system was removing all peripherals and having only enough hardware (CPU RAM and disk) involved to boot the operating system, then add components until the bad item crashes the system (Boy am I old, heard of Multics before I worked with it).

Anyway on to business. You now have three parts, Supervisor blade, chassis, and power supply, one of them is bad. The good news is if you have a standby supervisor in a good switch, insert the known good standby supervisor into the chassis you are working with to rule out the chassis and power supply. If you feel like an adventure after ruling out the chassis and power supply, insert the troublesome supervisor in as the standby in a production switch to confirm its continued disfunction. Personally I suspect bad RAM in the supervisor you are troubleshooting, why you ask? IOS uses more RAM than CATOS so if a bad chunk of RAM is in the upper memory area then ka blammo after the IOS decompresses. Only a suspicion but if it walks like a duck, its a duck, then a call to the TAC for replacement is in order for the supervisor if this pans out.

Cheers,

Brian

craig.golding · ‎11-29-2005

Hey Brian,

I figured that 'skinny' was similar to 'barebones', now I know. I pulled a good sup from our live system and it works fine by itself in the spare chassis. Setting up a good sup and the bad one redundantly doesn't help. The good sup doesn't recognise bad at all and so I cant force a redundancy switchover. I'm starting to think along the same lines as you - RAM probs. Let me know if some miracle process pops up...

Thanks for your help.

Craig

blakelyholman · ‎11-29-2005

This looks like you may have a problem with the BOOTLDR env var in the rommon... Just follow the conversion doc from beginning to end; if you've got a native IOS image on a flashcard you should be OK.

amike25 · ‎11-29-2005

I had the same problem you have with converting a sup2-msfc2 to native IOS, The errors you receive are caused by the PCMCIA card (slot0) because it was not properly formated or it was formated under a different kind of OS.

In case you have another Catalyst 6500 switch available that runs native IOS try to format the PCMCIA card on that switch and copy the IOS image file again. If you receive the same errors, the next thing I advise you to do is to copy the IOS image file from the flash card to bootflash and than try the boot bootflash: command (if there is a way to copy the file from one device to another from rommon).

If you look in the instructions on how to convert a Cat6500 switch from CatOS to IOS you will see that one of the steps is to re-format the flash card and the bootflash device.

In my case, I tried booting from the flash card (slot0) about 5 times even after formating the card and got the same error messages. The moment I copied the file in bootflash I was able to boot the switch correctly from the first try.

Hope it helps.

craig.golding · ‎11-29-2005

Hi amike25,

The IOS I'm using is the same as our live version. I formatted the flash card in a live 6500 Sup2 of the same type and version (firmware and IOS) first time around but to no avail.

I'm unable to copy the IOS from slot0: to bootflash: as the only commands available are:

rommon 1 > ?

alias set and display aliases command

boot boot up an external process

break set/show/clear the breakpoint

confreg configuration register utility

cont continue executing a downloaded image

context display the context of a loaded image

dev list the device table

dir list files in file system

dis disassemble instruction stream

frame print out a selected stack frame

help monitor builtin command help

history monitor command history

meminfo main memory information

repeat repeat a monitor command

reset system reset

set show all monitor variables

stack produce a stack trace

sync write monitor environment to NVRAM

sysret print out info from last system return

unalias unset an alias

xmodem x/y modem download

rommon 2 >

Thanks for your reply. If Cisco had a utility I could run from slot0: to format bootflash and then copy the IOS to is I would probably be ok. Other than that I'm out of ideas.

Craig

craig.golding · ‎11-29-2005

Hi blakelyholman,

Thanks for your suggestion. The conversion doc is no help as it assumes that you are able to boot the IOS. My problem starts at Step 15 - the IOS decompresses and then the errors start to flow (see my previos posts). Besides that - I don't have the old Catos from step 2.

Thanks though.

Craig