Solved: ACE card reboots in VSS mode

RAMAN AZIZIAN · ‎12-19-2009

Hello, and sorry for the lenghty message.

Yesterday we implemented VSS in our network and encountered a problem with the ACE module that I am trying to resolve.

Here's the hardware layout

two 6509 with the following line cards:

Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
1   24 CEF720 24 port 1000mb SFP              WS-X6724-SFP       SAL1329U0TN
2   48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX     SAL1327T950
3   48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX     SAL1325SHWN
5    5 Supervisor Engine 720 10GE (Active)    VS-S720-10G        SAL1327T9A7
6   48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX     SAL1328TM43
9    1 Application Control Engine Module      ACE20-MOD-K9       SAD133201AN

---- --------------------------- ------------------ ----------- ------- -------
1 Centralized Forwarding Card WS-F6700-CFC       SAL1327TH0F 4.1    Ok
2 Centralized Forwarding Card WS-F6700-CFC       SAL1317NUCC 4.1    Ok
3 Centralized Forwarding Card WS-F6700-CFC       SAL1322R6K6 4.1    Ok
5 Policy Feature Card 3       VS-F6K-PFC3C       SAL1326T3ME 1.0    Ok
5 MSFC3 Daughterboard         VS-F6K-MSFC3       SAL1326T183 2.0    Ok
6 Centralized Forwarding Card WS-F6700-CFC       SAL1327TA00 4.1    Ok

Mod Online Diag Status
---- -------------------
1 Pass
2 Pass
3 Pass
5 Pass
6 Pass
9 Pass --More--

Switch Number:     2   Role: Virtual Switch Standby
---------------------- -----------------------------
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
2   48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX     SAL1327TG40
3   48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX     SAL1328TM1R
4   48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX     SAL1328TQCP
5    5 Supervisor Engine 720 10GE (Hot)       VS-S720-10G        SAL1327T9B1
6   48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX     SAL1328TQDS
8   48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX     SAL1324S80K
9    1 Application Control Engine Module      ACE20-MOD-K9       SAD133201A3

---- --------------------------- ------------------ ----------- ------- -------
2 Centralized Forwarding Card WS-F6700-CFC       SAL1327T9ZY 4.1    Ok
3 Centralized Forwarding Card WS-F6700-CFC       SAL1327T80P 4.1    Ok
4 Centralized Forwarding Card WS-F6700-CFC       SAL1327TCWJ 4.1    Ok
5 Policy Feature Card 3       VS-F6K-PFC3C       SAL1327T9X8 1.0    Ok
5 MSFC3 Daughterboard         VS-F6K-MSFC3       SAL1326T4QY 2.0    Ok
6 Centralized Forwarding Card WS-F6700-CFC       SAL1326T2GC 4.1    Ok
8 Centralized Forwarding Card WS-F6700-CFC       SAL1252EA2J 4.1    PwrDown

Mod Online Diag Status
---- -------------------
2 Pass
3 Pass
4 Pass
5 Pass
6 Pass
8 Not Applicable
9 Not Applicable

=================================================

Here's the problem:

On the standby switch 2, the ACE module keeps rebooting. I first thought may be the power was not sufficient, so I checked the power utilization, and I have about 425 watts available.

I shut the power to one of the 10/100/1000 line card and I still had the same problem.

I moved the ACE module to SW1, trying to see if it would follow the problem and it did not have any issues.

I shut down two (2) modules on switch 2, and I still have the same problem.

Here's the log:

odulee
sm(cygnus_oir_bay slot9), running yes, state wait_til_online
Last transition recorded: (disabled)-> stay_disabled (operator_power_on)-> can_p
ower_on (yes_power)-> powered_on (real_power_on)-> check_power_on (timer)-> chec
k_power_on (power_on_ok)-> wait_til_online (reset_timer_online)-> wait_til_onlin
e (reset_timer_online)-> wait_til_online (reset_timer_online)-> wait_til_online
(reset_timer_online)-> wait_til_online xit
N-MSFC-VDR1-PRI#
Dec 19 13:14:50 UTC: %C6KPWR-SW2_SPSTBY-4-DISABLED: power to module in slot 9 se
t off (Module Failed SCP dnld)
Dec 19 13:14:52 UTC: %SYS-5-CONFIG_I: Configured from console by console

=================================================================

Does VSS shut down the ACE module because it sees two cards in one virtual switch?

Also we have no configuration setting on the either one of the ACE module. We are doing this implementation in phases, and I was planning on applying some basic Load-balancing in a couple of weeks.

I am not sure if this is a Chassis issue or VSS.

Any help would greatly be appreciated.

lars.molstad · ‎12-19-2009

Hi

This is probable due to old software on the ACE Modules.

I had the same issue with A2(1) release on the ACE, but upgrading software solved the issue.

Have a look at this link: - This states that A2(1.3) is the first supported release for VSS.

http://www.cisco.com/en/US/products/ps9336/products_tech_note09186a0080a7c72b.shtml

Br

Lars

CCIE 14647

View solution in original post

rvavale · ‎12-20-2009

Hi,

ACE software version A2(1.2) or later and installed in C6k running IOS 12.2(33)SXI or later support VSS.

If the ACE is running lower version then you could see this issue. To resolve this issue you need to upgrade ACE modules.

You could follow below process to Upgrade ACE from EOBC,

You can follow below process to boot from EOBC where ACE is in Switch 1 , slot 2.

Make sure ACE A2(1.2) or later image is on flash/disk0 on SUP,

Step 1: On Switch configure ACE module to boot from Switch disk0

!
conf term
boot device switch 1 module 2 disk0:c6ace-t1k9-mz.A2_1_5a.bin
end

Step 2: Power off ACE module in slot 2 by using command,

(Config t) no power enable switch 1 module 2

Step 3: To boot ACE module using EOBC

hw-module switch 1 module 2 boot eobc

Step 4: Then power on ACE module in slot 2 by using command,

(Config t) power enable switch 1 module 2

Step 5: Make sure ACE boots up and has correct version running.

If ACE is already running support version for VSS or if you need any assistance to upgrade then please raise Service request with TAC.

Hope this helps,

Best Regards,
Rahul