Losing the IOS

Unanswered Question
May 3rd, 2010

I have alot of cisco 3550 switches out at different sites. So far we have been rebooting all switches and when they power back up the IOS is missing.

I end up using xmodem to get them working.


What could cause the IOS to be lost.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (4 ratings)
Loading.
jfeo Mon, 05/03/2010 - 07:53

Let me add to this note I may have 6 3550 and 2 3560 the 3560 never lose the IOS and 3 out of say 6 3550 loose the IOS

Leo Laohoo Mon, 05/03/2010 - 15:28

I'd check the config register of each of the 3550.  Someone may be pulling a prank.

jfeo Mon, 05/03/2010 - 17:01

checked that we back up configs and save a show version

jfeo Tue, 05/04/2010 - 09:47

anybody with any Ideas what could be causing this

burleyman Tue, 05/04/2010 - 11:37

When you say you are losing the IOS do you mean the actual IOS file or the configuration? Can you post the config-register it should be 0X2102. Also what IOS version are you running? When you issue a write mem do you see any errors?

Mike

jfeo Tue, 05/04/2010 - 11:42

cisco 3550 switch

Losing the IOS I have to xmodem the switch.

Not sure if you can change a config-register on a switch

Version 12.1(22)EA4

Configuration register is 0x10F

jfeo Tue, 05/04/2010 - 11:52

I can you give you one from a switch that after a reboot it came back

This switch was done there other day 1 failed this one didn't

switch10#dir all
Directory of flash:/

    2  -rwx         346  Feb 28 1993 19:07:10 -05:00  system_env_vars
    3  -rwx         556  Feb 28 1993 19:00:54 -05:00  vlan.dat
    6  drwx         192  Feb 28 1993 19:06:57 -05:00  c3550-i5k2l2q3-mz.121-22.EA4
    8  -rwx           0  Feb 28 1993 19:07:09 -05:00  env_vars
    5  -rwx       11958  Jan 05 2010 01:20:35 -05:00  config.text
    9  -rwx        1532  Jan 05 2010 01:20:35 -05:00  private-config.text
   10  -rwx        1048  Jan 05 2010 01:20:35 -05:00  multiple-fs

15998976 bytes total (9503744 bytes free)
Directory of zflash:/

    2  -rwx         346  Feb 28 1993 19:07:10 -05:00  system_env_vars
    3  -rwx         556  Feb 28 1993 19:00:54 -05:00  vlan.dat
    6  drwx         192  Feb 28 1993 19:06:57 -05:00  c3550-i5k2l2q3-mz.121-22.EA4
    8  -rwx           0  Feb 28 1993 19:07:09 -05:00  env_vars
    5  -rwx       11958  Jan 05 2010 01:20:35 -05:00  config.text
    9  -rwx        1532  Jan 05 2010 01:20:35 -05:00  private-config.text
   10  -rwx        1048  Jan 05 2010 01:20:35 -05:00  multiple-fs

15998976 bytes total (9503744 bytes free)
Directory of nvram:/

  369  -rw-       11958                      startup-config
  370  ----        1532                      private-config
    1  -rw-           0                      ifIndex-table

393216 bytes total (378650 bytes free)
Directory of system:/

    2  dr-x           0                      memory
    1  -rw-       11889                      running-config

No space information available

burleyman Tue, 05/04/2010 - 11:58

It looks like the files are set to delete on reload....notice the drwx   i believe the d means delete. I am not overly firmilar on how to change that but I am looking it up now.

Mike

Leo Laohoo Tue, 05/04/2010 - 15:06

the "d" means directory.

No.  It means I need a drink.  A stiff one!
burleyman Tue, 05/04/2010 - 12:01

Sorry....I did not mean delete...I meant directory. What that means it the IOS is in a directory.

Mike

Jon Marshall Tue, 05/04/2010 - 12:04

burleyman wrote:

Sorry....I did not mean delete...I meant directory. What that means it the IOS is in a directory.

Mike

Mike

It's when you have multiple other files with the actual IOS image such as the files needed for web management of the switch etc.

Jon

burleyman Tue, 05/04/2010 - 12:10

Thanks Jon....I actually did know that....I think I am still in vacation mode..... 

I have had routers lose the running config due to a bug just resently but not the actual IOS.....

Mike

burleyman Tue, 05/04/2010 - 12:06

Also post the output of

dir flash:c3550-i5k2l2q3-mz.121-22.EA4

dir zflash:c3550-i5k2l2q3-mz.121-22.EA4

And the show run

Mike

jfeo Tue, 05/04/2010 - 12:37

I can give the sh run what are you looking for note this switch booted up fine after reboot it is some of the switches that loose the IOS

switch10#dir flash:c3550-i5k2l2q3-mz.121-22.EA4
Directory of flash:/c3550-i5k2l2q3-mz.121-22.EA4/

    7  drwx        4032  Feb 28 1993 19:04:43 -05:00  html
  441  -rwx     4968676  Feb 28 1993 19:06:57 -05:00  c3550-i5k2l2q3-mz.121-22.EA4.bin
  442  -rwx         263  Feb 28 1993 19:06:57 -05:00  info

15998976 bytes total (9503744 bytes free)
switch10#dir zflash:c3550-i5k2l2q3-mz.121-22.EA4
Directory of zflash:/c3550-i5k2l2q3-mz.121-22.EA4/

    7  drwx        4032  Feb 28 1993 19:04:43 -05:00  html
  441  -rwx     4968676  Feb 28 1993 19:06:57 -05:00  c3550-i5k2l2q3-mz.121-22.EA4.bin
  442  -rwx         263  Feb 28 1993 19:06:57 -05:00  info

15998976 bytes total (9503744 bytes free)

burleyman Tue, 05/04/2010 - 12:51

We would need to see this for the switches that lost the IOS.

Mike

jfeo Tue, 05/04/2010 - 12:54

I would also like to see it to because I am not sure what is causing this seems 1 out 10 cisco 3550 fail to boot up after reload and it just started to happen.

I will start to gather that info for sites going forward and see if I find anything. Wasn't sure if someone had same problem and new a reason why old equipment etc.

burleyman Tue, 05/04/2010 - 13:03

This could be a bug. Are you running all the same level of IOS for all the swithes? Also could you upgrade the IOS on one of the switches and reboot it to see if the IOS remains.

I have had switches lose the config on reboot but not the IOS itself. Mine ended up being a bug in the IOS which had to do with a kron job trying to run to a location that was no longer valid.

actually could you run the following command on a few of the problem switches and post the output......  show file descriptors

Mike

jfeo Tue, 05/04/2010 - 13:17

File Descriptors:

  No open file descriptors

and it is the same IOS on all switch 2,000 switches

Version 12.1(22)EA4

burleyman Tue, 05/04/2010 - 13:30

I am at a loss on what would cause this. I would open a TAC case if you can to see if there is a bug that would be causing this, or you could check you IOS in the bug tracker tool.

Mike

vvasisth Wed, 05/05/2010 - 02:10

When you say after reload the switch looses ios, this means after the reload the switch goes to rommon mode correct ?

if it goes to rommon then do a "dir flash:" this will show you if there is any IOS image in the flash or not.

If there is no IOS image there then its possibly due to currupted flash

However if the ISO image is there issue a "boot" command in rommon the switch will come up with the IOS in flash.

Then its an issue with the boot variable or configuration register. Check the output of "show boot"

the boot variable field should have the IOS and config register should be 0x2102.

If you open a TAC case let me know i can take owenership of it and help you out. My id vvasisth

Hope that helps and if this does then dont forget to rate the post.

Regards,

Varun Vasisth

A register value of 0x10f indicates that the switch config-register is set to the default.  Indicating that the switch was either never configured properly to start with or that it has reverted to the default.

This also is the reason your having a problem.  The IOS could be valid and intact and this will not mean jack because 0x10f will boot the BOOT environmental variable.  However, by default this is blank.  Since you are able to load the IOS manually this indicates to me that the IOS is most likely valid and without error.  Another indication that the IOS is not the primary issue is that if the IOS was bad or that the flash was bad it would boot into ROMMON as the cisco employee already posted.

Your symptoms do not indicate a bad IOS or bad flash as they have been described and as I read them.

You can change the config-register and you do so by running the enabled command "set boot config-register (value)" i.e. "set boot config-register 0X2102"

I would set the config-register and then reload.  If the switch reverts and goes back to 0x10f then I would look at faulty hardware as my next step.

If this doesn't fix your problem but the switch is retaining 0x2102 as the config-register then let us know and we can do additional troubleshooting for that issue.

(edit)

before someone comments and contradicts 0x10f is the default.  I should clarify and say that it is the default value for the CATOS.  Cisco IOS doesn't have this value, to my knowledge, and hence why it is a little harder to know.  Afterall, who uses CATOS anymore?

(edit2)

Also if you get an error when you do "set boot config-register" (which is the valid CATOS command) then try  the config command "config-register (value)".  That is the valid IOS command to change the config-register value. 

jfeo Wed, 05/05/2010 - 08:57

the config reg has never been changed I also get the switches with that. I can't even figure how to change it on a 3550

I have switches coming in daily with this problem so I will be open a tac case shortly

jfeo Wed, 05/05/2010 - 09:10

These are cisco WS-C3550-24 NO CATOS

and is 0x10F is default for cisco 3550 switch and I am not sure if that can be changed

WS-C3550-24 is the model.  CatOS and Cisco IOS both work on 3550 and if you are getting these devices used then it is very possible the previous owner used CatOS on them.  If they were sold to you new then I would question that they were actually new devices and not refurbs or something is wrong because I don't think resellers can sell the WS-C3550-24 with CatOS pre-installed anymore.

When you take in a new switch the 1st couple of steps should include identifying the config-register and then if it is a CatOS value you change it to a Cisco IOS value.  tftp the correct Cisco IOS over and then reload the switch so that it boots up properly into Cisco IOS.  Make a final verification of the Switch config-register and version of IOS running.

I suspect that since you didn't know how to do this that these steps were being skipped during intake.

Just so your clear here are the 3 most common scenarios and commands to set/change the config-register.

ROMMON: confreg [value]

CatOS: (enabled) set boot config-register [value]

Cisco IOS: (config) config-register [value]

EXAMPLES:

ROMMON:

rommon 1 >confreg 0x2102

CatOS:

Console> (enable) set boot config-register 0x10f

Cisco IOS:

Switch(config)#config-register 0x2102

I just went to check and test this recommendation and it didn't validate the config-register command.  Then I remembered that the 3550 doesn't come with the command config-register.  I have two 3550 48's next to me so I will try and replicate your issue and see what I come up with.

In the meantime I would follow this guide:

http://www.cisco.com/en/US/products/hw/switches/ps646/products_configuration_example09186a0080169623.shtml#configure2

See if the steps provided there allow you to set the boot more effectively.  For all of your other switches the register value is most likely settable.  The reason the 0x10f is the default for the 3550's you have is because the 3550's were designed and built, initially, for CatOS.

jfeo Wed, 05/05/2010 - 10:06

I was not aware of catos and we have over 1,000 switches out there that were bought new. I did do a IOS upgrade 4 years ago on all of them so we could use ssh so there were upgrade and reboot years ago now if someone is deleting the flash I don't know and if there is some switches out there there were used could be.

Like I said there is over 1,000 maybe moire of them so it is hard to track all of them

Thanks for all of the info

glen.grant Wed, 05/05/2010 - 10:27

  I think with that specific config register value you have to make sure it has a specific boot statement in your

config pointing to the image in flash otherwise it will stop booting .  Try putting in the specific boo

t statement .   Also 3550's were never a catos box , strictly IOS.

I was able to replicate your issue.

To respond to Glen.  I checked and I can't find a record of Cisco selling the 3550 with a base CatOS pre-installed.  So to that I was mistaken, but the 3550 is a unique system that runs with a CatOS backbone.  This is why it uses 0x10f as the default config-register.  This is also why it prevents a change to the register.  It appears to be a safety measure to prevent invalid config-register values.

Now with all of that said it leaves the question.  Why is your 3550 behaving this way and how do you fix it.

Glen is also correct in how to resolve the issue.  If you follow the link I provided you it will tell you how to staticly define a boot variable.  Incase it doesn't the config command is "boot system [path]".

Out of curiosity please post the results of "show boot" I am curious what you have before you make the change.  When you do this also make sure that the Manual Boot field is set to "no"

jfeo Wed, 05/05/2010 - 11:59

I will do I am out of the office and will do either tomorrow or Friday

jfeo Thu, 05/06/2010 - 09:24

Here is from one site I found this flash was missing from switch1 how it went missing is unknown.
Switch has been up for 1 year 45 week.

switch1#show flash

Directory of flash:/

    3  -rwx           0  Jun 23 2005 14:37:50 -04:00  env_vars
    4  -rwx          73  Jun 23 2005 14:37:50 -04:00  system_env_vars
    6  -rwx        1048  Jun 02 2009 01:10:35 -04:00  multiple-fs
    2  -rwx         556  Feb 28 1993 19:00:52 -05:00  vlan.dat
    5  -rwx        1530  Jun 02 2009 01:10:35 -04:00  private-config.text
    8  -rwx       13250  Jun 02 2009 01:10:35 -04:00  config.text

15998976 bytes total (15980032 bytes free)

switch1#show boot
BOOT path-list:       flash:c3550-i5k2l2q3-mz.121-22.EA4/c3550-i5k2l2q3-mz.121-22.EA4.bin
Config file:          flash:/config.text
Private Config file:  flash:/private-config.text
Enable Break:         no
Manual Boot:          no
HELPER path-list:    
NVRAM/Config file
      buffer size:    393216

===========================================================

switch2#sh flash

Directory of flash:/

   17  drwx         192  Jun 23 2005 14:37:38 -04:00  c3550-i5k2l2q3-mz.121-22.EA4
    3  -rwx           0  Jun 23 2005 14:37:48 -04:00  env_vars
    4  -rwx          73  Jun 23 2005 14:37:48 -04:00  system_env_vars
    6  -rwx        1048  Jun 02 2009 01:10:42 -04:00  multiple-fs
    2  -rwx         556  Feb 28 1993 19:00:52 -05:00  vlan.dat
    5  -rwx        1533  Jun 02 2009 01:10:42 -04:00  private-config.text
    8  -rwx       13256  Jun 02 2009 01:10:42 -04:00  config.text

15998976 bytes total (9502720 bytes free)


switch#show boot
BOOT path-list:       flash:c3550-i5k2l2q3-mz.121-22.EA4/c3550-i5k2l2q3-mz.121-22.EA4.bin
Config file:          flash:/config.text
Private Config file:  flash:/private-config.text
Enable Break:         no
Manual Boot:          no
HELPER path-list:    
NVRAM/Config file
      buffer size:    393216

Actions

This Discussion