Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

6800 ia upgrade loop on 6880-X w 15.1SY

Have received our first couple of 6800ia switches and tried to get them to work on a 6880-X in VSS. All works fine until the 6800IA concludes it has an older image then the one included in 15.1SY(2) that is on the 6880-X.

The feature set on the 6880-X is Advanced Enterprice

This is what I see on the 6880-X

Router#show fex
FEX       FEX              FEX                        FEX                  
Number    Description      State            Model              Serial      
---------------------------------------------------------------------------
101       FEX0101          img_dnld         C6800IA-48TD       FOC1742S2L4
121       FEX0121          img_dnld         C6800IA-48TD       FOC1742S2KK
Router#

Router#
*May  6 06:30:33.528: %SATMGR-SW1-5-IMAGE_VERSION_MISMATCH: Image version mismatch detected for FEX 101: image download initiated.  FEX will reboot on completion.

This is what I see on both 6880IA

FEX-101#

 Could not buffer tarfile...using multiple downloads
examining image...
%Error opening tftp://192.1.1.1/c6800ia-universalk9-mz.150-2.EX5.bin (Timed out)
%Error opening flash:update/info (No such file or directory)
ERROR: Image is not a valid IOS image archive.

I can not find any documentation on how the upgrade works on CCO. Is the ip address 192.1.1.1 a default loopback on the parent switch or do I need to create it? Been dealing with FEX on Nexus switches before (which is pretty similar) but never had to config anything else then fex association on the parent switch ports and it have worked.

Everyone's tags (1)
19 REPLIES
New Member

Hmmm this just gets weirder.

Hmmm this just gets weirder. I rebooted the active switch in the VSS pair and after the switchover to the other sup one of the FEX:s came up upgraded. The other one is stuck in rommon and can not be booted using the image on the flash. Opened a TAC case

New Member

hello,I'm in the same

hello,

I'm in the same situation as you. I have an IA that does not download the OS since 6800. I rebooted the VSS and now my IA is rommon. Have you had a response from cisco tac?

Mathieu

 

 

New Member

Hi,look herehttp://www.cisco

Hi,

look here

http://www.cisco.com/c/en/us/support/docs/field-notices/638/fn63838.html 

BR

New Member

HelloThe name of the .bin

Hello

The name of the .bin should be short, like c6800ia-image.bin and copy the file directly under flash: on the IA. If you use the whole path like flash:/update/etc... it won't work (see error below).

 

Thanks,

Eduard

 

switch: BOOT=flash:/update/c6800ia-universalk9-mz.150-2.EX6/c6800ia-universalk9-mz.150-2.EX6.bin

 

switch: boot

Loading "flash:/update/c6800ia-universalk9-mz.150-2.EX6/c6800ia-universalk9-mz.150-2.EX6.bin"...flash:/update/c6800ia-universalk9-mz.150-2.EX6/c6800ia-universalk9-mz.150-2.EX6.bin: magic number mismatch: bad mzip file

 

Error loading "flash:/update/c6800ia-universalk9-mz.150-2.EX6/c6800ia-universalk9-mz.150-2.EX6.bin"

 

Interrupt within 5 seconds to abort boot process.

Boot process failed...

New Member

Hi,

Hi,

This is the list of steps which worked for me together with solved issue relating the missing boot variable.

It's for later version of IOS but I hope it will help either way.

https://howdoesinternetwork.com/2016/6880-issu

Regards,

Valter

New Member

Me too.  I have 5 switches

Me too.  I have 5 switches from different stacks stuck in ROMON after the stack doing a 6880 initiated upgrade......

New Member

Followed this method from TAC

Followed this method from TAC and got my 6800ia switches working.

 

1. Check the FEX file and extract it into boot disk:
dir microcode:
Sh fex image bundle location

archive tar /xtract microcode:FEX bootdisk:    <<<< will extract .bin file from FEX
        You should get a directory that looks like c6800ia-universalk9-mz.150-2.EX5, which if
        you look inside of will have a c6800ia-universalk9-mz.150-2.EX5.bin file.

dir bootdisk:

2. Copy c6800ia-universalk9-mz.150-2.EX5.bin to tftp server

3. Delete "c6800ia-universalk9-mz.150-2.EX5.bin" from 6800IA

4. Copy image from TFTP into 6800IA

you should be able to do this via TFTP and the use of the management port on the 6800IA (left-hand top-most port)
Now I connected the management port of the fex and set up an IP address. The update folder I deleted before already.

switch: set IP_ADDR 1.1.1.1/255.255.255.0
switch: set DEFAULT_ROUTER 1.1.1.2  

switch: ping 1.1.1.2    <TFTP server ip>
ping 1.1.1.2 with 32 bytes of data ...
Host 192.168.101.55 is alive.

switch: copy tftp://1.1.1.2/c6800ia-universalk9-mz.150-2.EX5.bin flash:update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin

5. Booting by boot command
switch: boot flash:update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


FEX-111#

 

switch: 

 

switch: set IP_ADDR 1.1.1.1/255.255.255.0 

 

switch: set DEFAULT_ROUTER 1.1.1.2

 

switch: delete flash:/update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin

Are you sure you want to delete "flash:/update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin" (y/n)?y

File "flash:/update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin" deleted

 

switch: copy tftp://1.1.1.2/c6800ia-universalk9-mz.150-2.EX5.bin flash:update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin

...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

File "tftp://1.1.1.2/c6800ia-universalk9-mz.150-2.EX5.bin" successfully copied to "flash:update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin"


switch: boot

Loading "flash:/c6800ia-universalk9-mz.150-2.EX2/c6800ia-universalk9-mz.150-2.EX2.bin"...flash:/c6800ia-universalk9-mz.150-2.EX2/c6800ia-universalk9-mz.150-2.EX2.bin: no such file or directory

 

Error loading "flash:/c6800ia-universalk9-mz.150-2.EX2/c6800ia-universalk9-mz.150-2.EX2.bin"

 

Interrupt within 5 seconds to abort boot process.

Loading "flash:/update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin"...Verifying image flash:/update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin.................................Image passed digital signature verification

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ (etc)
File "flash:/update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin" uncompressed and installed, entry point: 0x3000

executing...


              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706

New Member

You have to set the ip tftp

You have to set the ip tftp source Interface that it works. The internal usage for this is Vlan1006.

New Member

I have a case open and

I have a case open and instructions on how to unpack the FEX image from the 6880-X image and tftp it over but the fex refuse to boot on the manually extracted file as well. Just updated the case.

You can try this as well and see if any one else is lucky. I suspect everyone is running the same 6880-X version since there is only one available for download.

I used the 688-X as tftp-server and connected a cable between mgmt0 on the 6880 and the EOB ethernet port (top left corner) of the 6800ia. I guess one can forget using USB...

####### Instruction start
Copy image via TFTP and boot 6800IA in case the IA-image is corrupted

SETUP TFTP SERVER :

1. a. If you have a Lab TFTP Server :

      Connect 6800IA TFTP Port (left hand topmost port)  to the TFTP Server .

 

   b. If you don’t have a TFTP Server :

      i. Using USB Stick/compact-flash copy the image to bootdisk.

        Internally image available at :

            /release/150/bin/150-2.EX5/.3DES    (Please refer to the compatibility chart above for right image)

 

        bash-3.2$ ls /release/150/bin/150-2.EX5/.3DES/c6800ia*

        /release/150/bin/150-2.EX5/.3DES/c6800ia-universalk9-mz.150-2.EX5 <<<< Can be used directly

                /release/150/bin/150-2.EX5/.3DES/c6800ia-universalk9-tar.150-2.EX5 <<<< Will need to extract as below.

 

    OR

 

1. Get the image present on the controller by executing following on Controller

        archive tar /xtract microcode:FEX bootdisk:

 

IA-PARENT#archive tar /xtract microcode:FEX bootdisk:

c6800ia-universalk9-mz.150-2.5.10.MK1D/ (directory)

extracting c6800ia-universalk9-mz.150-2.5.10.MK1D/info (821 bytes)

c6800ia-universalk9-mz.150-2.5.10.MK1D/html/ (directory)

extracting c6800ia-universalk9-mz.150-2.5.10.MK1D/html/foo.html (0 bytes)

extracting c6800ia-universalk9-mz.150-2.5.10.MK1D/c6800ia-universalk9-mz.150-2.5.10.MK1D.bin (15414400 bytes)

extracting c6800ia-universalk9-mz.150-2.5.10.MK1D/dc_default_profiles.txt (71241 bytes)

extracting info (118 bytes)

 

2. Identify the FEX image to use : It will have a .bin suffix in the directory created above.

 

IA-PARENT#dir bootdisk:c6800ia-universalk9-mz.150-2.5.10.MK1D

Directory of bootdisk:/c6800ia-universalk9-mz.150-2.5.10.MK1D/

 

   23  -rw-         821  Apr 10 2014 13:54:22 +05:00  info

   24  drw-           0  Apr 10 2014 13:54:24 +05:00  html

   32  -rw-    15414400  Apr 10 2014 13:54:52 +05:00  c6800ia-universalk9-mz.150-2.5.10.MK1D.bin <<<<<====== FEX IMAGE

   33  -rw-       71241  Apr 10 2014 13:54:52 +05:00  dc_default_profiles.txt

 

1981050880 bytes total (348753920 bytes free)

 

3. Copy the FEX image to the bootdisk: and "RENAME" the file . It should NOT MATCH the directory name or the FEX image name.

 

IA-PARENT#copy bootdisk:c6800ia-universalk9-mz.150-2.5.10.MK1D/c6800ia-universalk9-mz.150-2.5.10.MK1D.bin bootdisk:c6800ia-universalk9-mz.150-2.5.10.MK1

Destination filename [c6800ia-universalk9-mz.150-2.5.10.MK1]?

Copy in progress...CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

15414400 bytes copied in 7.488 secs (2058547 bytes/sec)

IA-PARENT#

 

 

      ii. Make the Controller Box as the TFTP Server like this :-

            tftp-server bootdisk:c6800ia-universalk9-mz.150-2.5.10.MK1 <<<====RENAMED FILE

 

      iii.Connect the IA TFTP Port with an Interface on Controller Box.

 

2.

    a.Setup the IP ADDRESS and GATEWAY on the 6800IA

        switch: IP_ADDR=8.10.8.19/255.255.0.0    (Based on the connection to tftp network)

        switch: DEFAULT_ROUTER=8.10.0.1        (IP address of the gateway to reach tftp-server or

                               IP address of the controller interface)

 

    b. If using Controller as TFTP-SERVER

     Setup the IP ADDRESS on the Controller interface

    This is same as the DEFAULT_ROUTER configured on 6800IA

 

4. Check the reachability of the tftp server.

switch: ping 202.153.144.25

ping 202.153.144.25 with 32 bytes of data ...

Host 202.153.144.25 is alive.

 

5. Copy the image to the 6800IA :

switch: copy tftp://x.x.x.x/<imagename> flash:<imagename>

 

 

6. Set the boot variable :

switch: BOOT=flash:/<image-name>

    Example image-name = c6800ia-universalk9-mz.150-2.EX4.bin

 

7. Check if the auto-boot is enabled, if not :

switch: MANUAL_BOOT=no

 

8. Boot the 6800IA

switch: boot

 

New Member

Same problem here with 15.1(2

Same problem here with 15.1(2)SY3 on a 6880-X-LE VSS and a stack of 2x C6800IA-48TDs.

I'm wondering whether I have to do this for every future IA client addition... frown

New Member

Just to let you know, the

Just to let you know, the problem went away by rebooting the whole VSS...

Cisco Employee

Gentlemen, I understand that

Gentlemen,

 

I understand that a lot of you are having trouble getting your IA clients upgraded. I am going to try and address your concerns and talk about how this process works.

I am going to take the example of a 6880-X running 15.1(2)SY2 with the IA client already booted up and running with the EX5 code. Now, let's say you're upgrading the 6880-X to 15.1(2)SY3 - this has the EX6 microcode, which implies that on reboot, the IA client needs to be automatically upgraded.

The first thing to understand is that communication between the IA client and the parent happens over a dedicated VLAN interface that is created on boot up for both. On sup2T setups, this is most likely VLAN 1006 and on 6880-X setups this is most likely VLAN 1013.

On the parent, this VLAN is given the IP address 192.168.1.1/24 while on the IA client this is given the IP address 192.168.1.<fex ID>/24.

Let's say the parent has come up in 15.1(2)SY3 now. The IA client first boots up with its older, existing image (EX5) in our case and comes up as FEX0. If you're consoled into the FEX, you'd see the host name to be FEX0 at this stage.

Now it initializes its SVI for communication with the parent and moves over to the configured FEX ID. At this stage, it attempts a download of the microcode from the parent. Once the image is downloaded, the FEX reboots and now it boots with the new image. 

Unfortunately, there are several defects around this entire process which can break it leading to situations that y'all run into.

 

For this reason, here are some troubleshooting tips to narrow down where exactly the issue is.

 

Troubleshooting:

- Verify that your SVI on both the parent and the IA client is up/up. There is a defect (which is still internal) that causes the SVI on either the parent or the client to be up/down. Due to this, there is no reachability between the parent and the client over this SVI and thus the image never gets downloaded.

Typically, if the SVI is down on the IA client, you'd get socket errors on the FEX console. If the SVI is down on the parent, you'd get timed out errors on the FEX console. This is also why a reboot resolves the issue. When you reboot the VSS, the SVI is reinitialized on bootup and it probably did not run into that race condition issue again and the SVI came up fine.

 

- Verify that you can ping between the SVIs. On the parent, the SVI is put into a FEX management VRF. So when pinging from the parent, you need to initiate the ping from the VRF. Use ping vrf <vrf name> <IP address of the FEX SVI> for this. On the FEX, the SVI is not under any VRF, so you can directly ping the parent using ping 192.168.1.1.

 

- Verify that your interfaces in the fabric port-channel are bundled. You might see one in a waiting state some times (w). This is another defect that is fixed in 15.1(2)SY3. However, this is not something you should be extremely concerned about with regards to the download. As long as at least one interface is bundled, it should be okay.

 

If you still have questions or are continuing to have trouble, please feel free to inbox me here or directly send me an email. Please make sure you specify your SR number (if you have a TAC case open) and I'll take a look at whats happening in the SR.

 

Regards,

Aninda

New Member

Aninda,I have 6880 VSS pair

Aninda,

I have 6880 VSS pair running 15.1(2)SY3 and ran into the issue where FEX cannot receive upgraded image due to the SVI for VLAN 1013 being in UP/Down state. I was able to clear the interface VLAN 1013 and bring the SVI into an UP/UP state. The FEX was then able to upgrade without issue. You mentioned that this bug is internal, do you or anyone else know if this or these other FEX upgrade bugs are to be fixed in the November release? These issues leave us with concern about the viability of this platform in any high profile deployments if we need to worry this much about the upgrade process.

Router#sho ip int brief

Interface              IP-Address      OK? Method Status                Protocol

Te1/1/1                unassigned      YES NVRAM  up                    up     

.

.

.

Vlan1                  unassigned      YES NVRAM  administratively down down 

Vlan1013               192.1.1.1       YES unset  up                    down   

Note the VLAN1013 interface used by the FEX is down

Note you cannot shut no shut this interface

Router#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

Router(config)#int vlan 1013

% This interface cannot be modified

Clearing the interface resolved the down state

Router#clear int vlan 1013

Router#sho ip int brief

Interface              IP-Address      OK? Method Status                Protocol

Te1/1/1                unassigned      YES NVRAM  up                    up     

.

 

Cisco Employee

Hi, This FEX internal SVI is

Hi,

 

This FEX internal SVI is put into a restricted mode which is why you cannot shut/no-shut it. The issue arises from a timing problem due to which the SVI is not initialized in software.

 

Unfortunately, this specific concern is not fixed in the upcoming November release (15.1(2)SY4).

 

I can work with you on any other FEX related concerns that you're seeing if you'd like to talk about it.

 

Regards,

Aninda

New Member

I have had this problem. I

I have had this problem. I opened a TAC case via our support provider for this, who could only point me to this thread!

A few weeks ago when testing the 6807-XL with FEX, I managed to reboot the device before the FEX image downloaded, which wiped off the image from the IA clients, and I had to go through the switch: 'rommon' recovery procedure as others have pointed out, so I won't repeat that

However, here is a another insight. Yesterday, I installed two 6880-X devices in a remote location. I only had console access to the devices, and none of the management interfaces were connected. I found I was in the booting loop

6880-X#sh fex
FEX       FEX              FEX                        FEX                  
Number    Description      State            Model              Serial      
---------------------------------------------------------------------------
101       FEX0101          img_dnld         C6800IA-48TD       FCWXXXXXXX
102       FEX0102          img_dnld         C6800IA-48TD       FCWXXXXXXX 

Dec  1 04:01:05.703: fex_chassis: Reg Response - version mismatch
Dec  1 04:01:05.703:     fex_chassis (101)189c.5d66.8e80: during state registered, got event 11(version_mismatch)
Dec  1 04:01:05.703: @@@ fex_chassis (101)189c.5d66.8e80: registered -> version_mismatch
Dec  1 04:01:05.703: image incompatible: image to download:tftp://192.1.1.1/c6800ia-universalk9-mz.150-2.EX6.bin
Dec  1 04:01:05.717: fexx_cm:: async svi op 2 (1006, 192.1.1.101, 255.255.255.0) completed, status:success(1)
Dec  1 04:01:05.717: %SATMGR-5-IMAGE_DOWNLOAD_START: Download of image to FEX started.  FEX will reboot on completion.
Dec  1 04:02:21.823:     fex_chassis (101)189c.5d66.8e80: during state version_mismatch, got event 12(download_fail)
Dec  1 04:02:21.823: @@@ fex_chassis (101)189c.5d66.8e80: version_mismatch -> discovery
Dec  1 04:02:21.823: fex_chassis: Image download fail. 101

From previous experience, I know that if you have CoPP enabled, then you must allow TFTP connectivity for 192.1.1.0/24 as the FEX uses to that communicate with the chassis for the download. I already had this, but removed CoPP just in case. 

I thought that I am going to have to upgrade manually, but had no mgmt or console access to the FEX, so I did extracted the FEX image from the microcode as in the rommon recovery procedure, and turned the chassis into a TFTP server for the FEX image. I then telnet'd to the FEX via its management VRF (which is only possible if the FEX is still alive and not in rommon

6880-X(config)#archive tar /xtract microcode:FEX bootdisk:
6880-X(config)#copy bootdisk:/c6800ia-universalk9-mz.150-2.EX6/c6800ia-universalk9-mz.150-2.EX6.bin bootdisk:6800IA-EX6.bin
6880-X(config)#tftp-server bootdisk:6800IA-EX6.bin
6880-X(config)#ip tftp source-interface Vlan1006
6880-X(config)#^Z
6880-X#telnet 192.1.1.101 /vrf __SVI_fex_mgmt_vrf__
Trying 192.1.1.101 ... Open

User Access Verification

Password: 
FEX-101>en
Password: 
FEX-101#copy tftp://192.1.1.1/6800IA-EX6.bin flash:c6800ia-universalk9-mz.150-2.EX6.bin
Destination filename [c6800ia-universalk9-mz.150-2.EX6.bin]? 
Accessing tftp://192.1.1.1/6800IA-EX6.bin...
Loading 6800IA-EX6.bin from 192.1.1.1 (via Vlan1006): !!!!!!!!!!!

The image downloaded, and I was just about to verify it, when the FEX rebooted itself. I thought I had done something wrong, but after a few minutes, the FEX came back on line. I had another FEX which I hadn't even touched yet, and I saw this was also online.

#sh fex
FEX       FEX              FEX                        FEX                  
Number    Description      State            Model              Serial      
---------------------------------------------------------------------------
101       FEX0101          online           C6800IA-48TD       FCW1XXXXXXX
102       FEX0102          online           C6800IA-48TD       FCW1XXXXXXX

 

From this, I gathered that when I configured 'ip tftp source-interface Vlan1006' which is the FEX vlan interface, the auto download actually worked and beat me to the FEX software upgrade.

I tried this on another chassis which had the same problem, and changed the TFTP source interface to the FEX vlan, and after 20 mins the FEXs were on line

 

New Member

Very good catch Dean!

Very good catch Dean!

Many thanks for the post. I

Many thanks for the post. I had a FEX stuck in the "%SATMGR-SW2-5-IMAGE_VERSION_MISMATCH" loop after upgrading a C6807-XL from 151-2.SY1 to 151-2.SY4a. I had changed the "ip tftp source-interface" setting a few weeks before. I changed it to "ip tftp source-interface vlan 1006" and the FEX upgraded successfully.

Thanks again
Andy

New Member

Thanks for the help. You save

Thanks for the help. You save my day.

 

New Member

hi,I also had this problem on

hi,

I also had this problem on upgrading from 151-2.SY3 to152-1.SY0a

The loop was every 3 minutes: SATMGR-SW2-5-IMAGE_VERSION_MISMATCH: Image version mismatch detected for FEX 105: image download initiated.  FEX will reboot on completion.

On sh fex: 105       FEX0105          img_dnld         C6800IA-48FPD  

but there was no new file on the flash, also no traffic on the uplinks.

 

I copied the extracted image through tftp manually on every fex, restarted them and it finally worked.

Next time i will copy the image on the fex before starting the upgrade.

 

Thanks for sharing this solution!

 

12315
Views
15
Helpful
19
Replies
CreatePlease login to create content