- Cisco Employee,
There is a manufacturing calibration issue on AP 3500 and AP 1260 (VID V01). It is addressed in manufacturing via VID V02.
The calibration problem can cause random memory corruption, which in turn leads to a variety of symptoms.
This is Field Notice FN63537.
This problem may affect any 1260 and 3500 series AP for which both of the following conditions are true:
- VID is V01 (verify via visual inspection, or via the AP command show inventory)
- Bootloader has not been upgraded (verify via the AP command show version | include BOOTLDR; if it says "Version 12.4(23c)JA", then it needs to be upgraded)
A 1260 or 3500 series access point may exhibit one or more of the following symptoms:
- The AP may be unable to boot. When power is applied, the LED shows as
white, and no console output is seen.
- The AP may reboot periodically with:
System returned to ROM by watchdog timer expired
Last reset from watchdog timer expired
ap1260 uptime is 10 hours, 52 minutes
System returned to ROM by watchdog timer expired
- The AP may crash with a message in crashinfo similar to one of the following:
Unexpected exception to CPUvector 700, PC = 0x5F784 , LR = 0x1A5524
-Traceback= 0x5F784 0x1A5524 0x1A56F0 0x177AE4 0x276F88 0x1A5A70
Unexpected exception to CPUvector 200, PC = 0x237CB0 , LR = 0x237C94
-Traceback= 0x237CB0 0x237D18 0x1F2824 0x51CC30 0x51B898 0x508E5C 0x511794
0x511B7C 0x503B44 0x51C49C 0x51CB30 0x526378 0x530184 0x52FD0C 0x5DA6AC 0x584C4C
reset_interrupt_level: reverse reset level current level 4 new level 1174028828
Unexpected exception to CPUvector 700, PC = 0x1A19C8 , LR = 0x1A1918
-Traceback= 0x1A19C8 0x1A1918 0x5DEB84 0x5DEEA4 0x5DEF54 0x5DF0CC 0x1A5A70
In a majority of instances when an AP is down with a white LED and there is no console access, physically rebooting the AP by toggling the switch port does recover it.
A new AP IOS image will be provided with new bootloader/uboot code. When IOS boots, it will check the bootloader version that is installed - if it does not have the fix (i.e. either an AP which has not undergone this process or new APs purchased before June 2012, prior to the bootloader change in manufacturing), then the following will happen:
- Bootloader code will be upgraded and uboot will be copied to flash and the AP will reboot.
- The new bootloader code will come up and call the new uboot. The new uboot will check the DDR calibration status. If the calibration needs to be rerun, then uboot will run the calibration routines and the AP will reboot.
- Finally, the bootloader will boot IOS to continue AP's normal operation.
The bootloader/uboot/calibration upgrade procedure will run only the first time the AP reboots after loading the new IOS. This upgrade is a one-time process, which will take about 7 minutes. The new DDR values shall remain even if the AP is upgraded/downgraded to any different code version.
For autonomous APs, the fix will be provided in an normal IOS tar image ("tarball"). For lightweight APs, the fix will be provided in AP IOS, bundled in wireless LAN controller images.
The fixed autonomous IOS and wireless LAN controller images are now available on CCO:
- WLC 220.127.116.11
- WLC 18.104.22.168
- WLC 22.214.171.124
- aIOS 15.2(2)JA
For unified deployments, there are three options:
- One time upgrade using the fixed image
- Perform a backup of the configuration
- Load the fixed WLC image
- Wait for all the APs to re-join. AP3500/1260 shall take additional time to rejoin due to the calibration process, which involves two reboots - one after copying of new bootloader and u-boot and second after performing the calibration.
- Wait for all AP3500s/1260s to rejoin.
- Boot the backup WLC image to return to the original run time image.
- Upgrade AP images only, via predownload, while keeping the controller on your regular image
- Download to the controller the fixed WLC image with the calibration fix, but do not reboot the controller.
After the download, the primary image will be the newly loaded fixed image, and the original image will be the backup (as seen with "show boot".) Use the command "config boot backup" to make sure that the original image is the active one:
(2106-1) >config boot backup
(2106-1) >show boot
Primary Boot Image............................... 126.96.36.199
Backup Boot Image................................ 188.8.131.52 (default) (active)
Issue "config ap image predownload primary all" to push the special IOS image to all APs.
Verify via "show ap image all" that all APs now have both the original and the fixed IOS image. This may take from several minutes to several hours depending on the number of APs.
Issue "config ap image swap all" to set the AP's BOOT variable to the special IOS image
Now reboot the 3500/1260 APs via any desired method.
The APs will boot the special IOS image, which will upgrade the bootloader, run the DDR calibration, boot the fixed IOS image again, join the controller. The controller and the AP figure out that the right (original) image is on the AP flash, so the AP sets its BOOT variable to the original IOS image, reboots with the original IOS image, and joins the controller.
- Verify that the 3500/1260 APs have the new bootloader image installed via the AP IOS command "show version".
- Normal update to the fixed image
- Upgrade the WLC to the fixed image as per normal.
- Let all the APs re-join. AP3500/1260 will take additional time to rejoin due to the calibration process, which involves two reboots - one after copying of new bootloader and u-boot and second after performing the calibration.
For upgrading autonomous IOS deployments:
- Load the fixed IOS tarball directly on the AP, through the AP telnet/ssh/console (via 'archive download' command)
- AP shall download the new bootloader and perform a recalibration; this step involves two reboots.
- After the calibration is performed, IOS boots up
- If so desired, downgrade to an earlier version of IOS software.
The new AP tarball contains the following three files:
- u-boot - It contains the new calibration logic
- RAM-based secondary bootloader - It contains the new automatic upgrade logic, which performs the function of replacing the bootloader and also performing calibration via u-boot
- ROM-based bootloader - This will replace the current bootloader
1. The present BS: bootloader that was created at FCS of AP3500/1260 will call a new RAM-based bootloader.
2. LED will glow AMBER. The RAM-based “secondary bootloader” contains all of the upgrade logic, and will do the one time copy of u-boot to BB:, and the one time copy of the new BS: bootloader to BS:.
3. LED will turn TEAL. This denotes the completion of step (2) and the AP will reset.
4. LED will glow PINK. The new BS: bootloader will jump to u-boot after the reset. This will calibrate the DDR and reset the AP.
5. LED will turn GREEN, as it usually does during the normal boot up operation. The BS: bootloader will use the new DDR values, and boot the RAM-based bootloader. The RAM-based bootloader will boot IOS and continue its normal operation after joining the controller.
6. If the system is ever downgraded or upgraded and the RAM-based bootloader disappears, the ROM-based BS: bootloader will boot IOS. The new DDR values are still used.
To summarize, the sequence of LEDs during the upgrade process shall be - AMBER- TEAL- <AP reset> - PINK- <AP reset> - GREEN.
Sample console logs during the entire operation can be found here: https://supportforums.cisco.com/docs/DOC-25461
NOTE: The original AP3500/1260 Cisco Bootloader is overwritten. This step takes 5-10 seconds. It is important power is not removed during this operation. This is the only time the AP is at risk of becoming a “brick”.
Sanity Check - The easiest way to verify is to check the bootloader version on the APs via "show version" once they have been upgraded.
- A bootloader that has not been upgraded will look like this:
BOOTLDR: C3500 Boot Loader (AP3G1-BOOT-M) Version 12.4(23c)JA, RELEASE SOFTWARE(fc3)
- A bootloader that has been upgraded should show one of the following:
- BOOTLDR: C3500 Boot Loader (AP3G1-BOOT-M), Version 12.4 [mpleso-ap_jmr3_esc_0514 116]
- BOOTLDR: C1260 Boot Loader (AP3G1-BOOT-M), Version 12.4 [mpleso-ap_jmr3_0328 155]
- BOOTLDR: C3500 Boot Loader (AP3G1-BOOT-M) Version 12.4(23c)JA5, RELEASE SOFTWARE (fc1)
- To validate that the DDR calibration procedure has actually been run, issue the following command:
ap#show soap | include DDR
If, and only if, the DDR calibration has been performed, output similar to the following will be seen:
*NEW* DDR Calibration values detected in EEPROM at 0x1E0
What is the increase in size of the AP tarball due to the addition of these files?
About 800kb on top of the normal IOS tarball.
How much time does the entire process take?
The entire process after the AP's have downloaded the new tarball should take under 10 minutes.
As described above it involves the following steps:- Copy bootloader - Copy u-boot - Reset - Perform Calibration - Reset - Boot IOS and join the controller via CAPWAP.
What if the controller or IOS image is downgraded or upgraded in the future?
Since the current bootloader is replaced with the new bootloader, the fix shall persist no matter what run time image is used. This is just an one time calibration process.
What happens to APs which have already been calibrated either via the new image or new bootloader from manufacturing?
There is a check at the very beginning for both those conditions and hence the entire new upgrade and calibration logic will be skipped. The AP will incur no additional delay during its bootup if it already has the new calibration.
What is the probability of APs becoming "brick" during the upgrade process?
The probability is low, since this can only occur when the new bootloader is being copied to flash. If at that time power to the AP is lost, it will not recover. It is just a 5 - 10 second window when this can occur if there any anomaly in the PoE source.
Apart from copying of the new bootloader, what happens if any other part of the new calibration/upgrade is skipped or fails?
If any other part of the calibration/upgrade is skipped or fails, the AP will resume its normal operation and continue to use the old calibration value. It won't be stuck in a loop of trying to re-calibrate itself.
How does a boot-looping access-point get the new boot image?
Unless the flash has a problem, AP should be able to stay up long enough to load the image. If the AP is bricked or continuously looping, it will need to be RMA.