Calibration fix for 1260, 3500 series APs (CSCtu24972, CSCty68030)

Document

Jun 27, 2012 4:18 PM
Jun 27th, 2012

Issue

There is a manufacturing calibration issue on AP 3500 and AP 1260 (VID V01). It is addressed in manufacturing via VID V02.

AP1262-VID.jpg - login to see the image

The calibration problem can cause random memory corruption, which in turn leads to a variety of symptoms.

This is Field Notice FN63537.

#Scope

This problem may affect any 1260 and 3500 series AP for which both of the following conditions are true:

  • VID is V01 (verify via visual inspection, or via the AP command show inventory)
  • Bootloader has not been upgraded (verify via the AP command show version | include BOOTLDR; if it says "Version 12.4(23c)JA", then it needs to be upgraded)
#Symptoms

A 1260 or 3500 series access point may exhibit one or more of the following symptoms:

  • The AP may be unable to boot. When power is applied, the LED shows as
    white, and no console output is seen.

  • The AP may reboot periodically with:
    System returned to ROM by watchdog timer expired
    Last reset from watchdog timer expired

    Example:
    ap1260#show version
    [...]
    ap1260 uptime is 10 hours, 52 minutes
    System returned to ROM by watchdog timer expired

  • The AP may crash with a message in crashinfo similar to one of the following:

Unexpected exception to CPUvector 700, PC = 0x5F784 , LR = 0x1A5524
-Traceback= 0x5F784 0x1A5524 0x1A56F0 0x177AE4 0x276F88 0x1A5A70

Unexpected exception to CPUvector 200, PC = 0x237CB0 , LR = 0x237C94
-Traceback= 0x237CB0 0x237D18 0x1F2824 0x51CC30 0x51B898 0x508E5C 0x511794
0x511B7C 0x503B44 0x51C49C 0x51CB30 0x526378 0x530184 0x52FD0C 0x5DA6AC 0x584C4C

reset_interrupt_level: reverse reset level current level 4 new level 1174028828
%Software-forced reload
Unexpected exception to CPUvector 700, PC = 0x1A19C8 , LR = 0x1A1918
-Traceback= 0x1A19C8 0x1A1918 0x5DEB84 0x5DEEA4 0x5DEF54 0x5DF0CC 0x1A5A70

In a majority of instances when an AP is down with a white LED and there is no console access, physically rebooting the AP by toggling the switch port does recover it.

Solution

A new AP IOS image will be provided with new bootloader/uboot code. When IOS boots, it will check the bootloader version that is installed - if it does not have the fix (i.e. either an AP which has not undergone this process or new APs purchased before June 2012, prior to the bootloader change in manufacturing), then the following will happen:

  • Bootloader code will be upgraded and uboot will be copied to flash and the AP will reboot.
  • The new bootloader code will come up and call the new uboot. The new uboot will check the DDR calibration status. If the calibration needs to be rerun, then uboot will run the calibration routines and the AP will reboot.
  • Finally, the bootloader will boot IOS to continue AP's normal operation.

The bootloader/uboot/calibration upgrade procedure will run only the first time the AP reboots after loading the new IOS. This upgrade is a one-time process, which will take about 7 minutes. The new DDR values shall remain even if the AP is upgraded/downgraded to any different code version.

For autonomous APs, the fix will be provided in an normal IOS tar image ("tarball"). For lightweight APs, the fix will be provided in AP IOS, bundled in wireless LAN controller images.

The solution is implemented via CSCty68030 (which provides for the bootloader/uboot automatically upgrade) and by CSCtu24972 (which is the DDR calibration fix itself.)

Fixed Images

The fixed autonomous IOS and wireless LAN controller images are now available on CCO:

  • WLC 7.3.101.0
  • WLC 7.2.111.3
  • WLC 7.0.235.3
  • aIOS 15.2(2)JA
Upgrade Process

For unified deployments, there are three options:

  • One time upgrade using the fixed image
    • Perform a backup of the configuration
    • Load the fixed WLC image
    • Wait for all the APs to re-join. AP3500/1260 shall take additional time to rejoin due to the calibration process, which involves two reboots - one after copying of new bootloader and u-boot and second after performing the calibration.
    • Wait for all AP3500s/1260s to rejoin.
    • Boot the backup WLC image to return to the original run time image.
  • Upgrade AP images only, via predownload, while keeping the controller on your regular image
      • Download to the controller the fixed WLC image with the calibration fix, but do not reboot the controller.
      • After the download, the primary image will be the newly loaded fixed image, and the original image will be the backup (as seen with "show boot".) Use the command "config boot backup" to make sure that the original image is the active one:
        (2106-1) >config boot backup
        (2106-1) >show boot
        Primary Boot Image............................... 7.0.230.11

        Backup Boot Image................................ 7.0.116.0 (default) (active)

      • Issue "config ap image predownload primary all" to push the special IOS image to all APs.

      • Verify via "show ap image all" that all APs now have both the original and the fixed IOS image. This may take from several minutes to several hours depending on the number of APs.

      • Issue "config ap image swap all" to set the AP's BOOT variable to the special IOS image

      • Now reboot the 3500/1260 APs via any desired method.

      • The APs will boot the special IOS image, which will upgrade the bootloader, run the DDR calibration, boot the fixed IOS image again, join the controller. The controller and the AP figure out that the right (original) image is on the AP flash, so the AP sets its BOOT variable to the original IOS image, reboots with the original IOS image, and joins the controller.

      • Verify that the 3500/1260 APs have the new bootloader image installed via the AP IOS command "show version".
    • Normal update to the fixed image
      • Upgrade the WLC to the fixed image as per normal.
      • Let all the APs re-join. AP3500/1260 will take additional time to rejoin due to the calibration process, which involves two reboots - one after copying of new bootloader and u-boot and second after performing the calibration.

    For upgrading autonomous IOS deployments:

    • Load the fixed IOS tarball directly on the AP, through the AP telnet/ssh/console (via 'archive download' command)
    • AP shall download the new bootloader and perform a recalibration; this step involves two reboots.
    • After the calibration is performed, IOS boots up
    • If so desired, downgrade to an earlier version of IOS software.

    Upgrade Details

    The new AP tarball contains the following three files:

    • u-boot - It contains the new calibration logic
    • RAM-based secondary bootloader - It contains the new automatic upgrade logic, which performs the function of replacing the bootloader and also performing calibration via u-boot
    • ROM-based bootloader - This will replace the current bootloader

    1. The present BS: bootloader that was created at FCS of AP3500/1260 will call a new RAM-based bootloader.
    2. LED will glow AMBER. The RAM-based “secondary bootloader” contains all of the upgrade logic, and will do the one time copy of u-boot to BB:, and the one time copy of the new BS: bootloader to BS:.
    3. LED will turn TEAL. This denotes the completion of step (2) and the AP will reset.
    4. LED will glow PINK. The new BS: bootloader will jump to u-boot after the reset. This will calibrate the DDR and reset the AP.
    5. LED will turn GREEN, as it usually does during the normal boot up operation. The BS: bootloader will use the new DDR values, and boot the RAM-based bootloader. The RAM-based bootloader will boot IOS and continue its normal operation after joining the controller.
    6. If the system is ever downgraded or upgraded and the RAM-based bootloader disappears, the ROM-based BS: bootloader will boot IOS. The new DDR values are still used.

    To summarize, the sequence of LEDs during the upgrade process shall be - AMBER- TEAL- <AP reset> - PINK- <AP reset> - GREEN.

    Sample console logs during the entire operation can be found here: https://supportforums.cisco.com/docs/DOC-25461

    NOTE: The original AP3500/1260 Cisco Bootloader is overwritten. This step takes 5-10 seconds. It is important power is not removed during this operation. This is the only time the AP is at risk of becoming a “brick”.

    Verification

    Sanity Check - The easiest way to verify is to check the bootloader version on the APs via "show version" once they have been upgraded.

    • A bootloader that has not been upgraded will look like this:

    • BOOTLDR: C3500 Boot Loader (AP3G1-BOOT-M) Version 12.4(23c)JA, RELEASE SOFTWARE(fc3)

      • A bootloader that has been upgraded should show one of the following:
        • BOOTLDR: C3500 Boot Loader (AP3G1-BOOT-M), Version 12.4 [mpleso-ap_jmr3_esc_0514 116]
        • BOOTLDR: C1260 Boot Loader (AP3G1-BOOT-M), Version 12.4 [mpleso-ap_jmr3_0328 155]
        • BOOTLDR: C3500 Boot Loader (AP3G1-BOOT-M) Version 12.4(23c)JA5, RELEASE SOFTWARE (fc1)
      • To validate that the DDR calibration procedure has actually been run, issue the following command:

        ap#show soap | include DDR

        If, and only if, the DDR calibration has been performed, output similar to the following will be seen:


        *NEW* DDR Calibration values detected in EEPROM at 0x1E0
      FAQs

      What is the increase in size of the AP tarball due to the addition of these files?
      About 800kb on top of the normal IOS tarball.

      How much time does the entire process take?
      The entire process after the AP's have downloaded the new tarball should take under 10 minutes.
      As described above it involves the following steps:- Copy bootloader - Copy u-boot - Reset - Perform Calibration - Reset - Boot IOS and join the controller via CAPWAP.

      What if the controller or IOS image is downgraded or upgraded in the future?
      Since the current bootloader is replaced with the new bootloader, the fix shall persist no matter what run time image is used. This is just an one time calibration process.

      What happens to APs which have already been calibrated either via the new image or new bootloader from manufacturing?
      There is a check at the very beginning for both those conditions and hence the entire new upgrade and calibration logic will be skipped. The AP will incur no additional delay during its bootup if it already has the new calibration.

      What is the probability of APs becoming "brick" during the upgrade process?
      The probability is low, since this can only occur when the new bootloader is being copied to flash. If at that time power to the AP is lost, it will not recover. It is just a 5 - 10 second window when this can occur if there any anomaly in the PoE source.

      Apart from copying of the new bootloader, what happens if any other part of the new calibration/upgrade is skipped or fails?
      If any other part of the calibration/upgrade is skipped or fails, the AP will resume its normal operation and continue to use the old calibration value. It won't be stuck in a loop of trying to re-calibrate itself.

      How does a boot-looping access-point get the new boot image?
      Unless the flash has a problem, AP should be able to stay up long enough to load the image. If the AP is bricked or continuously looping, it will need to be RMA.

      Average Rating: 5 (1 ratings)

      Comments

      remysyaku Tue, 07/24/2012 - 23:12

      Hi,

      what are the special image for this? we given this image ap3g1-rcvk9w8-tar.124-23c.JA5.tar. Is there a guide/step how to load this image to wlc/ap?

      please advice.

      Aaron Leonard Wed, 07/25/2012 - 06:57 (reply to remysyaku)

      That image does not contain the CSCtu24972/CSCty68030 fix.  If you have APs that are encountering this CSCtu24972 problem, at present you will need to open a TAC case to get a special image with the fix.

      remysyaku Wed, 07/25/2012 - 18:28 (reply to Aaron Leonard)

      Thanks for your reply.

      I have this CSCtu24972 problem but its only effect 2 out of 150 APs. This special image, is it for wlc or ap? are there any special image just applied to this 2 APs?

      Aaron Leonard Thu, 07/26/2012 - 08:29 (reply to remysyaku)

      The fix is contained in AP IOS, which can be distributed as either a lightweight IOS tarball (k9w8), an autonomous IOS tarball (k9w7), or bundled into a WLC image (.aes.)  You can use whatever technique you want to load the IOS with the fix into your APs.  See the section above entitled "Upgrade process" for details.

      msb7 Wed, 09/26/2012 - 08:00

      Hi.  I have ~200 AUTONOMOUS 1262N's that are potentially affected.  I see that ap3g1-k9w7-tar.124-25d.JA2.tar is available but doesn't seem to address this problem as per its release notes.  ap3g1-k9w7-tar.152-2.JA.tar is available and fixes the problem, but is supported for "SITE SURVEY MODE ONLY" on the 1262N as per the download page.  (Though 15.2(2)JA seems to have full support for the 1142N.)  What code can I run on an autonomous 1262N AP that is fully supported for stand-alone use and that resolves this problem?               ---- Thanks, Mike

      Aaron Leonard Thu, 09/27/2012 - 09:53

      Mike - go ahead and use the ap3g1-k9w7-tar.152-2.JA.tar image with your 1262s.  This image fixes the DDR calibration problem, and is fully supported for all autonomous features on the 1262s.

      (The fact that the image was showing up with the "SITE SURVEY MODE ONLY" description is an artifact of a limitation of our tools - this image is fully supported for all aIOS features on the 1260s, but only for site survey features on the 3500s - for more details, see the "Autonomous IOS Support for 3500, 3600 and 1550 Series Access Points" article.)

      Aaron

      ligyan Mon, 11/12/2012 - 00:52

      Aron,

      I am facing almost same exact problem from a customer in China:

      -- Wism2/WLC-1 & -2, around 200 APs under each WLC, respectively;

      -- AP lost randomly, NOT seen from WLC;

      -- white light on dropped AP;

      -- "Ieee PD" status in "show power inline" in problematic AP connecting Switch;

      -- AP can recover by "shut/no shut" the interface on Switch;

      What have done:

      -- upgrade IOS to 7.2.113.3, according to bug CSCty68030;

      -- verified bootloader, DDR calibration, power recycle all APs;

      But there are still APs randomly lost.

      Any more idea ?

      SR623585397:TECH JL:AIR-CAP3502I-C-K9--AP lost randomly

      Aaron Leonard Mon, 11/12/2012 - 07:26 (reply to ligyan)

      Li-Gang,

      Please continue to work with the assigned TAC engineer, who has escalated your case to the business unit.

      Best regards,

      Aaron

      ligyan Mon, 11/12/2012 - 19:22

      Aaron,

      Sorry. What did you mean ? I am actually handling this case for now.

      Who you meant has esclated to BU ?

      thanks a lot !

      donglejack83 Wed, 09/11/2013 - 14:18

      Hi, perhaps a bit outdated, but u-boot copy failed for an 3502i (I forgot to unset BOOT variable before upgrading, thus pointing to former IOS):

      === 5. Writing MICRON static values to eeprom at addr 0x210

          ---> write to eeprom complete

      === 6. Copy u-boot to BB:

      flash:/ap3g1-k9w7-mx.124-25d.JA2/u-boot.bin: no such file or directory

          ---> error on u-boot copy -- skip upgradeBoot CMD: 'boot  flash:/ap3g1-k9w7-mx.124-25d.JA2/ap3g1-k9w7-xx.124-25d.JA2;flash:/ap3g1-k9w7-mx.152-4.JA1/ap3g1-k9w7-mx.152-4.JA1'

      Loading "flash:/ap3g1-k9w7-mx.124-25d.JA2/ap3g1-k9w7-xx.124-25d.JA2"...flash:/ap3g1-k9w7-mx.124-25d.JA2/ap3g1-k9w7-xx.124-25d.JA2: no such file or directory

      Error loading "flash:/ap3g1-k9w7-mx.124-25d.JA2/ap3g1-k9w7-xx.124-25d.JA2"

      Interrupt within 5 seconds to abort boot process.

      Boot process terminated.

      The system is unable to boot automatically.  The BOOT

      environment variable needs to be set to a bootable

      image.

      Hence, how to rerun u-boot process (or whole process)?

      Thanks!

      Aaron Leonard Wed, 09/11/2013 - 14:40 (reply to donglejack83)

      Will,

      So I assume that that AP is at the ap: bootloader prompt, right?  So the, you would do stuff like this:

      ap: dir flash:

      [ hopefully you have at least one legit IOS folder, e.g. something like: ]

      13   drwx  768       <date>               ap3g1-k9w7-mx.152-4.JA1

      so then set your AP to boot this image:

      ap: set boot flash:/ap3g1-k9w7-mx.152-4.JA1/ap3g1-k9w7-mx.152-4.JA1

      ap: boot

      more info on this kind of stuff here:

      http://www.cisco.com/en/US/docs/wireless/access_point/12.4.25d.JA/Configuration/guide/scg12.4.25d.JA-chap22-trouble.html

      https://supportforums.cisco.com/docs/DOC-14636

      Hth,

      Aaron

      donglejack83 Wed, 09/11/2013 - 15:23 (reply to Aaron Leonard)

      Hi Aaron, many thanks for your really quick update!

      Well, in fact, I'm not stuck in ROMMON at all since running 15.2(4)JA1, but copy of u-boot failed while upgrading.

      Device seems to run fine, but I'm uncertain if it is stable or not (apparently yes ...).

      I did a manual upgrade of bootloader (command: ap#copy flash:/ap3g1-k9w7-mx.152-4.JA1/ap3g1-boot-m_upg bs:), it succeded. But how to manually upgrade u-boot to complete whole process, that's the question ... (knowing that file 'u-boot.bin' is present on IOS directory)

      Also, FYI, I have:

      ap#show soap | include DDR

      *ORIGINAL* DDR Calibration values detected in EEPROM at 0x0

      (APs that succeded had an eeprom value of 0x1E0, exactly as in your example above).

      So, if any idea, it is welcome!

      Again, many thanks!

      Have a nice day!

      Aaron Leonard Wed, 09/11/2013 - 15:39 (reply to donglejack83)

      Well I've never known an AP's uboot upgrade to fail, and honestly I don't know what to do make it work.  I personally would not worry about it, and just replace this AP should it happen to crash as described above.

      Best,


      Aaron

      Actions

      Login or Register to take actions

      This Document

      Posted June 27, 2012 at 4:18 PM
      Stats:
      Comments:14 Avg. Rating:5
      Views:7239 Contributors:5
      Shares:0
      Tags: No tags.

      Documents Leaderboard