We are trying to configure vrf aware GET VPN with COOP having primary and secondary key servers and also 3 GM routers. All GM routers we use are Cisco 888 and Key servers we use cisco 2911 routers. All GMs crypto maps have been applied into Vlan interface as there's no L3 interface on 888 routers.
Always members can form a tunnel with primary KS, we have configured redundancy with secondary key server and listed on each GM primary and secondary KS on GDOI group.
The issue we facing is that whenever we shutdown the primary or secondary servers the tunnel is not forming with available KS unless otherwise we mannually clear the crypto session. In another way when primary KS down it doest not fall back to secondary KS and no GM get registered. We have already played with all the timers such as DPD, SA lifetimes, GDOI rekey lifetime etc and also exchanging the keys (import/export) with KS and COOPs but there's no luck. We could see the following message was seen on both KS.
on KS1 (Primary)
Jun 19 08:52:23.071: %GDOI-3-COOP_KS_UNREACH: Cooperative KS 192.168.1.6 Unreachable in group test-g. IKE SA Status = Failed to establish
on KS2 (Secondary)
Jun 19 08:54:26.074: %GDOI-3-COOP_KS_UNREACH: Cooperative KS 192.168.1.3 Unreachable in group test-g. IKE SA Status = Failed to establish
192.168.1.3 is the primary KS and 192.168.1.6 is the secondary KS.
I captured attached debug output from 1 GM and secondary KS while I shutdonw the primary KS and also attached is our senario we were trying get work.
Also attached is the show output from both KSs when it form a tunnel with GM.
Appreciate if someone can advise us what exactly causing the above issue/error.
thanks in advance