ACE Module - SSL Certificate Error

dedra_live · ‎08-04-2010

Hello,

I am running the following version for ACE module

Version A2(2.0) [build 3.0(0)A2(2.0)]

We are offloading SSL on ACE. The Production module failed over to the secondary, the certificate configurations 'cypto chaingroup' on the earlier primary (now secondary) lost all the root certificate configurations and the ft group status became ACTIVE / COLD.

While accessing the website in this state gave certificate errors even though the old secondary (now Primary) unit had the certificate configurations in place.

Please advise whether this is a normal behaviour in ACTIVE / COLD state while the active unit has the correct configurations. Also could you advise from the below logs what was the reason for failover. All the groups had failed over one after the other.

>sh ft history ha_mgr

1:164 => Aug 04 19:13:17: ha_process_message:1891 Recd MTS_OPC_CFG_SYNC_STATUS f
or FT Group 1
1:165 => Aug 04 19:13:17: ha_process_message:1924 Running sync info: mode 1, sta
tus 1, reason Error on Standby device when applying configuration file replicate
d from active
1:166 => Aug 04 19:13:17: ha_process_message:1928 Startup sync info: mode 1, sta
tus 0, reason Startup configuration sync has completed
1:167 => Aug 04 19:13:17: fsm_ft_action:205 FSM: FT Group 1, Current State FSM_F
T_STATE_STANDBY_CONFIG, Event FSM_FT_EV_CFG_SYNC_STATUS
1:168 => Aug 04 19:13:17: fsm_ft_check_cfg_sync:1301 Config-Sync failed, rc 1
1:169 => Aug 04 19:13:17: fsm_ft_goto_standby_cold_state:544 Stopping the timer
for FT 1 state FSM_FT_STATE_STANDBY_CONFIG
1:170 => Aug 04 19:13:17: notify_contexts:101 Notifying FT Group 1, Opcode MTS_O
PC_STANDBY_COLD

2:171 => Aug 04 19:13:17: ft_state_change:40 FSM State Change for Ft Group 1, Ol
d State FSM_FT_STATE_STANDBY_CONFIG New State FSM_FT_STATE_STANDBY_COLD Event FS
M_FT_EV_CFG_SYNC_STATUS
2:172 => Aug 04 19:13:17: send_ft_state_to_peer:246 Sending FT State Update to P
eer.FT Group 1, FT State FSM_FT_STATE_STANDBY_COLD, Config Priority 200, Net Pr
iority 200, Preempt 0 peer msg compatible true
2:173 => Aug 04 19:13:17: fsm_ft_action:211 FSM: pre_action function failed.
2:174 => Aug 04 19:13:22: handle_mts_message:3485 HA MGR: Received MTS notif, fr
om: 0x00000301/526, To: 0x00000301/509, Opcode: MTS_OPC_CFG_SYNC_STATUS(4042), M
sgID: 4546

yushimaz · ‎08-09-2010

ACE module went standby_cold status since error occurred when applying config file

as below.

> 1:165 => Aug 04 19:13:17: ha_process_message:1924 Running sync info: mode 1, sta
> tus 1, reason Error on Standby device when applying configuration file replicate
> d from active

You may find out the root cause of sync failure with 'show ft group detail' command.

The following output is an example of standby_cold. I created this status that I

imported key and cert on active ACE but didn't import on standby ACE. Since

standby ACE doesn't have cert, config sync failed.

To investigate the cause of your issue, please check the sync status with 'show ft

group detail' command.

---

ACE20b/Admin# show ft gr detail

FT Group : 1

No. of Contexts : 1

Context Name : Admin

Context Id : 0

Configured Status : in-service

Maintenance mode : MAINT_MODE_OFF

My State : FSM_FT_STATE_STANDBY_COLD

My Config Priority : 100

My Net Priority : 100

My Preempt : Enabled

Peer State : FSM_FT_STATE_ACTIVE

Peer Config Priority : 110

Peer Net Priority : 110

Peer Preempt : Enabled

Peer Id : 1

Last State Change time : Sun Aug 8 23:57:58 2010

Running cfg sync enabled : Disabled

Running cfg sync status : Incremental Sync Failure: SSL Certificate does not exist

Startup cfg sync enabled : Disabled

Startup cfg sync status : Incremental Sync Failure: SSL Certificate does not exist

---

I don't know the reason why failover occurred since I can see the status change from

standby_config to standby_cold only. However, I cannot see the status change from

active to standby. Probably, failover occurred and then config sync occurred.

To investigate the cause of failover, I need 'show ft history ha_mgr' output when status

change event occurred.

dedra_live · ‎08-15-2010

Hello,

Please find below output of 'show ft group detail'

FT Group                     : 1
No. of Contexts              : 1
Context Name                 : Admin
Context Id                   : 0
Configured Status            : in-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_ACTIVE
My Config Priority           : 200
My Net Priority              : 200
My Preempt                   : Disabled
Peer State                   : FSM_FT_STATE_STANDBY_HOT
Peer Config Priority         : 190
Peer Net Priority            : 190
Peer Preempt                 : Disabled
Peer Id                      : 1
Last State Change time       : Thu Aug 5 09:47:51 2010
Running cfg sync enabled     : Enabled
Running cfg sync status      : Running configuration sync has completed
Startup cfg sync enabled     : Enabled
Startup cfg sync status      : Startup configuration sync has completed
Bulk sync done for ARP: 0
Bulk sync done for LB: 0
Bulk sync done for ICM: 0

Please find below the output of 'show ft history ha_mgr'

2:171 => Aug 04 19:13:17: ft_state_change:40 FSM State Change for Ft Group 1, Ol
d State FSM_FT_STATE_STANDBY_CONFIG New State FSM_FT_STATE_STANDBY_COLD Event FS
M_FT_EV_CFG_SYNC_STATUS
2:172 => Aug 04 19:13:17: send_ft_state_to_peer:246 Sending FT State Update to P
eer.FT Group 1, FT State FSM_FT_STATE_STANDBY_COLD, Config Priority 200, Net Pr
iority 200, Preempt 0 peer msg compatible true
2:173 => Aug 04 19:13:17: fsm_ft_action:211 FSM: pre_action function failed.
2:174 => Aug 04 19:13:22: handle_mts_message:3485 HA MGR: Received MTS notif, fr
om: 0x00000301/526, To: 0x00000301/509, Opcode: MTS_OPC_CFG_SYNC_STATUS(4042), M
sgID: 4546
2:175 => Aug 04 19:13:22: ha_process_message:1891 Recd MTS_OPC_CFG_SYNC_STATUS f
or FT Group 2
2:176 => Aug 04 19:13:22: ha_process_message:1924 Running sync info: mode 1, sta
tus 1, reason Error on Standby device when applying configuration file replicate
d from active
3:177 => Aug 04 19:13:22: ha_process_message:1928 Startup sync info: mode 1, sta
tus 0, reason Startup configuration sync has completed
3:178 => Aug 04 19:13:22: fsm_ft_action:205 FSM: FT Group 2, Current State FSM_F
T_STATE_STANDBY_CONFIG, Event FSM_FT_EV_CFG_SYNC_STATUS
3:179 => Aug 04 19:13:22: fsm_ft_check_cfg_sync:1301 Config-Sync failed, rc 1
3:180 => Aug 04 19:13:22: fsm_ft_goto_standby_cold_state:544 Stopping the timer
for FT 2 state FSM_FT_STATE_STANDBY_CONFIG

3:181 => Aug 04 19:13:22: notify_contexts:101 Notifying FT Group 2, Opcode MTS_O
PC_STANDBY_COLD
3:182 => Aug 04 19:13:22: ft_state_change:40 FSM State Change for Ft Group 2, Ol
d State FSM_FT_STATE_STANDBY_CONFIG New State FSM_FT_STATE_STANDBY_COLD Event FS
M_FT_EV_CFG_SYNC_STATUS
3:183 => Aug 04 19:13:22: send_ft_state_to_peer:246 Sending FT State Update to P
eer.FT Group 2, FT State FSM_FT_STATE_STANDBY_COLD, Config Priority 200, Net Pr
iority 200, Preempt 0 peer msg compatible true
4:184 => Aug 04 19:13:22: fsm_ft_action:211 FSM: pre_action function failed.
4:185 => Aug 04 19:13:32: handle_mts_message:3485 HA MGR: Received MTS notif, fr
om: 0x00000301/526, To: 0x00000301/509, Opcode: MTS_OPC_CFG_SYNC_STATUS(4042), M
sgID: 5829
4:186 => Aug 04 19:13:32: ha_process_message:1891 Recd MTS_OPC_CFG_SYNC_STATUS f
or FT Group 3
4:187 => Aug 04 19:13:32: ha_process_message:1924 Running sync info: mode 1, sta
tus 0, reason Running configuration sync has completed
4:188 => Aug 04 19:13:32: ha_process_message:1928 Startup sync info: mode 1, sta
tus 0, reason Startup configuration sync has completed
4:189 => Aug 04 19:13:32: fsm_ft_action:205 FSM: FT Group 3, Current State FSM_F
T_STATE_STANDBY_CONFIG, Event FSM_FT_EV_CFG_SYNC_STATUS
4:190 => Aug 04 19:13:32: fsm_ft_check_cfg_sync:1297 Config-Sync succeeded
4:191 => Aug 04 19:13:32: fsm_ft_goto_standby_bulk_sync_state:643 Stopping the t

imer for FT 3 state FSM_FT_STATE_STANDBY_CONFIG
5:192 => Aug 04 19:13:32: notify_contexts:101 Notifying FT Group 3, Opcode MTS_O
PC_STANDBY_BULK
5:193 => Aug 04 19:13:32: ft_state_change:40 FSM State Change for Ft Group 3, Ol
d State FSM_FT_STATE_STANDBY_CONFIG New State FSM_FT_STATE_STANDBY_BULK Event FS
M_FT_EV_CFG_SYNC_STATUS
5:194 => Aug 04 19:13:32: send_ft_state_to_peer:246 Sending FT State Update to P
eer.FT Group 3, FT State FSM_FT_STATE_STANDBY_BULK, Config Priority 200, Net Pr
iority 200, Preempt 1 peer msg compatible true
5:195 => Aug 04 19:14:06: handle_mts_message:3485 HA MGR: Received MTS notif, fr
om: 0x00000301/510, To: 0x00000301/509, Opcode: MTS_OPC_BULK_SYNC_STATUS(4043),
MsgID: 5940
5:196 => Aug 04 19:14:06: ha_process_message:1947 Recd MTS_OPC_BULK_SYNC_STATUS
for FT Group 3
5:197 => Aug 04 19:14:06: fsm_ft_action:205 FSM: FT Group 3, Current State FSM_F
T_STATE_STANDBY_BULK, Event FSM_FT_EV_BULK_SYNC_STATUS
5:198 => Aug 04 19:14:06: fsm_ft_goto_standby_hot_state:687 Stopping the timer f
or FT 3 state FSM_FT_STATE_STANDBY_BULK
6:199 => Aug 04 19:14:06: notify_contexts:101 Notifying FT Group 3, Opcode MTS_O
PC_STANDBY_HOT
6:200 => Aug 04 19:14:06: ft_state_change:40 FSM State Change for Ft Group 3, Ol
d State FSM_FT_STATE_STANDBY_BULK New State FSM_FT_STATE_STANDBY_HOT Event FSM_F

T_EV_BULK_SYNC_STATUS
6:201 => Aug 04 19:14:06: send_ft_state_to_peer:246 Sending FT State Update to P
eer.FT Group 3, FT State FSM_FT_STATE_STANDBY_HOT, Config Priority 200, Net Pri
ority 200, Preempt 1 peer msg compatible true
6:202 => Aug 04 19:14:06: handle_mts_message:3485 HA MGR: Received MTS notif, fr
om: 0x00000502/509, To: 0xFFFFFFFF/0, Opcode: MTS_OPC_RELINQUISH(4029), MsgID: 2
591176
6:203 => Aug 04 19:14:06: ha_process_message:1997 Recd FSM_FT_EV_RELINQUISH for
FT Group 3
6:204 => Aug 04 19:14:06: ha_process_message:2007 Got FSM_FT_EV_RELINQUISH from
peer for ft group 3
6:205 => Aug 04 19:14:06: fsm_ft_action:205 FSM: FT Group 3, Current State FSM_F
T_STATE_STANDBY_HOT, Event FSM_FT_EV_RELINQUISH
7:206 => Aug 04 19:14:06: fsm_ft_handle_relinquish_msg:798 Recd a FSM_FT_EV_RELI
NQUISH event from Peer for FT Group 3, my state FSM_FT_STATE_STANDBY_HOT peer st
ate FSM_FT_STATE_ACTIVE
7:207 => Aug 04 19:14:06: notify_contexts:101 Notifying FT Group 3, Opcode MTS_O
PC_ACTIVE
7:208 => Aug 04 19:14:06: ft_state_change:40 FSM State Change for Ft Group 3, Ol
d State FSM_FT_STATE_STANDBY_HOT New State FSM_FT_STATE_ACTIVE Event FSM_FT_EV_R
ELINQUISH
7:209 => Aug 04 19:14:06: send_ft_state_to_peer:246 Sending FT State Update to P

eer.FT Group 3, FT State FSM_FT_STATE_ACTIVE, Config Priority 200, Net Priority
200, Preempt 1 peer msg compatible true
7:210 => Aug 04 19:14:13: handle_mts_message:3485 HA MGR: Received MTS notif, fr
om: 0x00000502/509, To: 0xFFFFFFFF/0, Opcode: MTS_OPC_FT_STATE(4028), MsgID: 259
1193
7:211 => Aug 04 19:14:13: ha_process_message:1826 Got FT State from peer for ft
group 3
7:212 => Aug 04 19:14:13: fsm_ft_action:205 FSM: FT Group 3, Current State FSM_F
T_STATE_ACTIVE, Event FSM_FT_EV_STATE
8:213 => Aug 04 19:14:13: fsm_ft_process_peer_ft_state_msg:953 Got FT State Upda
te from Peer.FT Group 3, FT State FSM_FT_STATE_ACTIVE, Config Priority 200, Net
Priority 200, Preempt 1, Peer FT Group 3, Peer FT State FSM_FT_STATE_STANDBY_BU
LK, Peer Config Priority 190, Peer Net Priority 190 Peer preempt 1
8:214 => Aug 04 19:14:13: notify_contexts:101 Notifying FT Group 3, Opcode MTS_O
PC_START_BULK_SYNC
8:215 => Aug 04 19:22:06: handle_mts_message:3485 HA MGR: Received MTS notif, fr
om: 0x00000502/509, To: 0xFFFFFFFF/0, Opcode: MTS_OPC_FT_STATE(4028), MsgID: 259
1709
8:216 => Aug 04 19:22:06: ha_process_message:1826 Got FT State from peer for ft
group 3
8:217 => Aug 04 19:22:06: fsm_ft_action:205 FSM: FT Group 3, Current State FSM_F
T_STATE_ACTIVE, Event FSM_FT_EV_STATE

9:218 => Aug 04 19:22:06: fsm_ft_process_peer_ft_state_msg:953 Got FT State Upda
te from Peer.FT Group 3, FT State FSM_FT_STATE_ACTIVE, Config Priority 200, Net
Priority 200, Preempt 1, Peer FT Group 3, Peer FT State FSM_FT_STATE_STANDBY_HO
T, Peer Config Priority 190, Peer Net Priority 190 Peer preempt 1
9:219 => Aug 04 19:22:06: notify_contexts:101 Notifying FT Group 3, Opcode MTS_O
PC_START_PERIODIC_SYNC

yushimaz · ‎08-16-2010

Hello

> My State : FSM_FT_STATE_ACTIVE
> Peer State : FSM_FT_STATE_STANDBY_HOT

> Last State Change time : Thu Aug 5 09:47:51 2010

FT status has changed active/standby_hot on Aug 5.

To confirm the cause of standby_cold, I have to check the output when problem occurs.

And, pasted 'show ft history ha_mgr' output was after changing status to standby_cold.

I want the log before Aug 04 19:13:17 since status change from active to standby occurred

earlier in this time.

Regards,

Yuji