Solved: [3750x] Switch failed to start on the new image

Loic Zambon · ‎04-07-2015

Good morning,

I updated the firmware of my switches 3750 x (19 switches in 8 stack group) with the new version. But 2 switches of a stack of 3 cannot take the last version. They are the same model, I don't understand why they cannot boot on the new image.

New version : 15.0(2)SE7

Old version : 12.2(58)SE2

Switches informations

Of the 17 Switches that works

WS-C3750X-24 15.0(2)SE7 C3750E-UNIVERSALK9-M

Model revision number           : A0
Motherboard revision number    : A0
Model number                    : WS-C3750X-24T-S

Of the 2Switches that not works

WS-C3750X-24 12.2(58)SE2 C3750E-UNIVERSALK9-M

Model revision number             : A0
Motherboard revision number        : C0
Model number                      : WS-C3750X-24T-S

This is the only different I found between the 2 switches with all other switches,

Image loading

In 1st step he load the image but he cannot boot and at the end he load the old image and boot on it.

Loading "flash:c3750e-universalk9-mz.150-2.SE7.bin"...@@@@@@@@@@@@@@@@@@

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@@@@@@@@@@...

File "flash:c3750e-universalk9-mz.150-2.SE7.bin" uncompressed and installed, entry point: 0x3000
executing...

Loading "flash:/c3750e-universalk9-mz.122-58.SE2.bin"...@@@@@@@@@@@@@@@@@@

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@@@@@@@@@@...

The switch that has the last version can not see the 2 others stacks. At the moment the only solution that I found it's the downgrade of the switch with version 15.

Regards,

mendixbeevee · ‎04-08-2015

The latest version that successfully boots on this device is 15.0(1)SE3

Any of the 15.0(2)* or 15.2.* images fail to boot and result in the device hanging during the @@@@@, as shown above.

So it seems Cisco already broke this in 2012. I almost cannot believe this.

View solution in original post

Leo Laohoo · ‎04-07-2015

Loading "flash:/c3750e-universalk9-mz.122-58.SE2.bin"

A lot of people run into problems like this because during the so-called "IOS upgrade" they used the very old, old method of only COPYING the IOS BIN file and the main cause of the issue is the BIN file they copied into the appliance(s) are corrupt. The output to the boot-up process confirms the IOS is corrupt.

I would recommend to use the TAR file and upgrade the IOS of the stack using the tried-and-tested command of "archive download-sw /destination 1 /destination 2 tftp://<TFTP IP address>/filename.tar". The command switch "/destination 1" and "/destination 2" means the IOS will be loaded into switch stack members 1 and 2. If you have more than two (or more) switch stack members then you add "/destination N" where N are the switch stack members.

Loic Zambon · ‎04-07-2015

I used the .bin only to have an exemple. At the first update I used this command (I used this command for all my stacks) :

archive download-sw tftp://AddressIP/c3750e-universalk9-mz.150-2.SE7.bin

In the stack only the 3rd take the update the 1st and 2nd not. I tried with this command :

archive tar /x tftp://AddressIP/c3750e-universalk9-tar.150-2.SE7.tar flash1:
archive tar /x tftp://AddressIP/c3750e-universalk9-tar.150-2.SE7.tar flash2:
conf t
boot system switch all flash:/c3750e-universalk9-mz.150-2.SE7/c3750e-universalk9-mz.150-2.SE7.bin

But they will not boot on the image.

Leo Laohoo · ‎04-07-2015

In the stack only the 3rd take the update the 1st and 2nd not.

And what error message was presented? Was it because there was no more available flash memory space?

Loic Zambon · ‎04-08-2015

I do not remember the error message (I'm not on site at the moment). But there was available flash memory space, I have the same result with the /overwrite parameter. He try to load the image v15 and he try again

Loading "flash:c3750e-universalk9-mz.150-2.SE7.bin"...@@@@@@@@@@@@@@@@@@

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@@@@@@@@@@...

File "flash:c3750e-universalk9-mz.150-2.SE7.bin" uncompressed and installed, entry point: 0x3000
executing...

Loading "flash:/c3750e-universalk9-mz.150-2.SE7.bin"...@@@@@@@@@@@@@@@@@@

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@@@@@@@@@@...

(here I don't remember the error message, and the rommon start)

>

Leo Laohoo · ‎04-08-2015

here I don't remember the error message, and the rommon start

Ok, I don't know about the others but that error message is key to your issue.

Let us know if you have or remember what the error message is because there are heaps of possibilities why your appliance won't boot the right IOS.

Loic Zambon · ‎04-12-2015

Hi,

I tried again friday with the last image version

...done Initializing Flash.
done.
Loading "flash:/c3750e-universalk9-mz.150-2.SE7.bin"...@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "flash:/c3750e-universalk9-mz.150-2.SE7.bin" uncompressed and installed, entry point: 0x3000
executing...
Loading "flash:/c3750e-universalk9-mz.150-2.SE7.bin"...@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "flash:/c3750e-universalk9-mz.150-2.SE7.bin" uncompressed and installed, entry point: 0x3000
executing...
Boot process failed...

The system is unable to boot automatically. The BOOT
environment variable needs to be set to a bootable
image

I was able to install the image 15.0 (1) SE3. Hoping that the bootloader is updated I tried to put the 15.0 (2) SE7 again. it does not work.

Regards,

Leo Laohoo · ‎04-13-2015

I was able to install the image 15.0 (1) SE3. Hoping that the bootloader is updated I tried to put the 15.0 (2) SE7 again. it does not work.

Loic,

You mentioned earlier that when you tried to upgrade the IOS there was an error message. I want to know what this error message is.

Loic Zambon · ‎04-13-2015

The system is unable to boot automatically. The BOOT
environment variable needs to be set to a bootable
image

It's just this. All other 3750X (with the same model) considers that this image are bootable, but not the switch with the C0 revision.

Leo Laohoo · ‎04-13-2015

Ok, post the output to the command "sh bootvar".

Loic Zambon · ‎04-13-2015

"sh bootvar" isn't available on 3750x. I have the "sh boot" command. When I'm on place I take the result of the command.

Loic Zambon · ‎04-14-2015

#sh boot

BOOT path-list      : flash:/c3750e-universalk9-mz.150-1.SE3/c3750e-universalk9-mz.150-1.SE3.bin
Config file         : flash:/config.text
Private Config file : flash:/private-config.text
Enable Break        : no
Manual Boot         : no
Allow Dev Key         : yes
HELPER path-list    :
Auto upgrade        : yes
Auto upgrade path   :
NVRAM/Config file
      buffer size:   524288
Timeout for Config
          Download:    0 seconds
Config Download
       via DHCP:       disabled (next boot: disabled)

Leo Laohoo · ‎04-14-2015

Nope. No idea. Your issue has got me "stumped".

Loic Zambon · ‎04-14-2015

No problem. With older v15 image all works.

Thank you for your time

mendixbeevee · ‎04-08-2015

Hi,

I'm trying to upgrade a 3750-X that I have here and run into a similar issue.

The switch refuses to boot either 15.0.2-SE7(MD) (c3750e-universalk9-mz.150-2.SE7.bin) or 15.2.3E(ED) (c3750e-universalk9-mz.152-3.E.bin)

When I try to load them, the output shows:

Loading "flash:/c3750e-universalk9-mz.150-2.SE7.bin"...@@@@@@@@@@@@@@@@@@@ [a total of 4989 x @ here] @@@@@@@@@@@@@@@@@@

or

Loading "flash:/c3750e-universalk9-mz.152-3.E.bin"...@@@@@@@@@@@@@@@@@@@ [a total of 6445 x @ here] @@@@@@@@@@@@@@@@@@

and then the @@@ stop appearing and the switch just hangs indefinitely.

The old image on the switch is 15.0(1)SE2, which boots fine.

This is a 3750-X from 2013, and sh ver also lists Motherboard revision number : C0

Switch#sh ver
Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.0(1)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Thu 22-Dec-11 00:05 by prod_rel_team

ROM: Bootstrap program is C3750E boot loader
BOOTLDR: C3750E Boot Loader (C3750X-HBOOT-M) Version 12.2(53r)SE1, RELEASE SOFTWARE (fc1)

Switch uptime is 1 minute
System returned to ROM by power-on
System image file is "flash:c3750e-universalk9-mz.150-1.SE2.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: ipbase
License Type: Permanent
Next reload license Level: ipbase

cisco WS-C3750X-48 (PowerPC405) processor (revision A0) with 262144K bytes of memory.
Processor board ID FDO1533P0YC
Last reset from power-on
1 Virtual Ethernet interface
1 FastEthernet interface
52 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : C4:64:13:18:2F:80
Motherboard assembly number     : 73-12552-05
Motherboard serial number       : FDO154919PB
Model revision number           : A0
Motherboard revision number     : C0
Model number                    : WS-C3750X-48T-S
Daughterboard assembly number   : 800-32727-01
Daughterboard serial number     : FDO15490XGB
System serial number            : FDO1533P0YC
Top Assembly Part Number        : 800-31326-02
Top Assembly Revision Number    : E0
Version ID                      : V02
CLEI Code Number                : COMJT00ARB
Hardware Board Revision Number : 0x03

Switch Ports Model              SW Version            SW Image
------ ----- -----              ----------            ----------
*    2 54    WS-C3750X-48       15.0(1)SE2            C3750E-UNIVERSALK9-M

Configuration register is 0xF

Hans