Solved: GetVPN rekey sequence numbers on GMs

ggalteroo · ‎02-19-2010

Hello

I have a few routers running getvpn so far. Two of them are 7206 NPE-G2 VAM2+ which were the CORE under the traditional IPSec tunnel solution. The rest of them are 2811s. The KS are two 2851s.

The solution is doing well but we are having some log messages I’d like your opinion on.

The 7200s show higher rekey counters than the 2800s.

RCTOBEKS01#sh cry gdoi ks members <- This is the key server

Group Member Information :

Number of rekeys sent for group getvpn : 1139

Group Member ID   : 110.32.0.100                                  <- exCORE
Group ID          : 25909
Group Name        : getvpn
Key Server ID     : 110.32.0.12
Rekeys sent       : 311
Rekeys retries    : 0
Rekey Acks Rcvd   : 311
Rekey Acks missed : 0

Sent seq num : 8 9 0 0
Rcvd seq num : 8 9 0 0

Group Member ID   : 110.38.64.100                                  <- exCORE
Group ID          : 25909
Group Name        : getvpn
Key Server ID     : 110.32.0.12
Rekeys sent       : 311
Rekeys retries    : 0
Rekey Acks Rcvd   : 311
Rekey Acks missed : 0

Sent seq num : 8 9 0 0
Rcvd seq num : 8 9 0 0

Group Member ID   : 110.40.0.1                                  <- Branch x
Group ID          : 25909
Group Name        : getvpn
Key Server ID     : 110.32.0.12
Rekeys sent       : 2
Rekeys retries    : 0
Rekey Acks Rcvd   : 2
Rekey Acks missed : 0

Sent seq num : 8 9 0 0
Rcvd seq num : 8 9 0 0

Group Member ID   : 110.40.0.3                                  <- Branch x
Group ID          : 25909
Group Name        : getvpn
Key Server ID     : 110.32.0.12
Rekeys sent       : 2
Rekeys retries    : 0
Rekey Acks Rcvd   : 2
Rekey Acks missed : 0

Sent seq num : 8 9 0 0
Rcvd seq num : 8 9 0 0

Group Member ID   : 110.40.0.4                                  <- Branch x
Group ID          : 25909
Group Name        : getvpn
Key Server ID     : 110.32.0.12
Rekeys sent       : 2
Rekeys retries    : 0
Rekey Acks Rcvd   : 2
Rekey Acks missed : 0

The timers are those recommended on the implementation guide. 7200 secs.

Feb 11 12:54:37.916: %GDOI-5-GM_REKEY_TRANS_2_UNI: Group getvpn transitioned to Unicast Rekey.
Feb 11 12:54:37.948: %GDOI-5-GM_REGS_COMPL: Registration to KS 110.32.0.12 complete for group getvpn using address 110.40.0.7
Feb 11 14:33:04.480: %GDOI-5-GM_RECV_REKEY: Received Rekey for group getvpn from 110.32.0.12 to 110.40.0.7 with seq # 8
Feb 11 16:19:39.513: %GDOI-5-GM_RECV_REKEY: Received Rekey for group getvpn from 110.32.0.12 to 110.40.0.7 with seq # 9
Feb 11 18:06:14.544: %GDOI-5-GM_RECV_REKEY: Received Rekey for group getvpn from 110.32.0.12 to 110.40.0.7 with seq # 10
Feb 11 19:52:49.566: %GDOI-5-GM_RECV_REKEY: Received Rekey for group getvpn from 110.32.0.12 to 110.40.0.7 with seq # 11
Feb 11 21:39:24.586: %GDOI-5-GM_RECV_REKEY: Received Rekey for group getvpn from 110.32.0.12 to 110.40.0.7 with seq # 12
Feb 11 23:25:59.619: %GDOI-5-GM_RECV_REKEY: Received Rekey for group getvpn from 110.32.0.12 to 110.40.0.7 with seq # 13
Feb 12 01:12:34.646: %GDOI-5-GM_RECV_REKEY: Received Rekey for group getvpn from 110.32.0.12 to 110.40.0.7 with seq # 14
Feb 12 02:59:09.673: %GDOI-5-GM_RECV_REKEY: Received Rekey for group getvpn from 110.32.0.12 to 110.40.0.7 with seq # 15
Feb 12 04:45:44.710: %GDOI-5-GM_RECV_REKEY: Received Rekey for group getvpn from 110.32.0.12 to 110.40.0.7 with seq # 16
Feb 12 06:32:19.739: %GDOI-5-GM_RECV_REKEY: Received Rekey for group getvpn from 110.32.0.12 to 110.40.0.7 with seq # 17
Feb 12 08:18:54.773: %GDOI-5-GM_RECV_REKEY: Received Rekey for group getvpn from 110.32.0.12 to 110.40.0.7 with seq # 18
Feb 12 10:05:29.802: %GDOI-5-GM_RECV_REKEY: Received Rekey for group getvpn from 110.32.0.12 to 110.40.0.7 with seq # 19
Feb 12 11:22:24.768: %GDOI-3-GDOI_REKEY_SEQ_FAILURE: Failed to process rekey seq # 2 in seq payload for group getvpn, last seq # 19
Feb 12 11:22:24.768: %GDOI-3-GDOI_REKEY_FAILURE: Processing of REKEY payloads failed on GM 110.40.0.7 in the group getvpn, with peer at 110.32.0.12
Feb 12 11:22:24.768: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode failed with peer at 110.32.0.12
Feb 12 11:23:45.394: %GDOI-3-GDOI_REKEY_SEQ_FAILURE: Failed to process rekey seq # 4 in seq payload for group getvpn, last seq # 19
Feb 12 11:23:45.394: %GDOI-3-GDOI_REKEY_FAILURE: Processing of REKEY payloads failed on GM 110.40.0.7 in the group getvpn, with peer at 110.32.0.12
Feb 12 11:23:45.394: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode failed with peer at 110.32.0.12
Feb 12 11:59:51.014: %GDOI-4-GM_RE_REGISTER: The IPSec SA created for group getvpn may have expired/been cleared, or didn't go through. Re-register to KS.
Feb 12 11:59:51.014: %CRYPTO-5-GM_REGSTER: Start registration to KS 110.32.0.12 for group getvpn using address 110.40.0.7
Feb 12 11:59:51.586: %GDOI-5-GM_REKEY_TRANS_2_UNI: Group getvpn transitioned to Unicast Rekey.
Feb 12 11:59:51.710: %GDOI-5-GM_REGS_COMPL: Registration to KS 110.32.0.12 complete for group getvpn using address 110.40.0.7

Finally, do you happen to know why the rekey might have failed on seq # 19? It usually happens around this number.

The COREs run 12.4(15)T9

The Branches now run 12.4(15)T12. We did an upgrade from T11 because of bug CSCtb13421.

Thank you!

Guido

dsandre-toh · ‎05-18-2010

does this help: check ver# of each router:

> 
> 

http://www.ciscosystems.com/en/US/docs/ios/sec_secure_connectivity/configura
> tion/guide/sec_encrypt_trns_vpn_ps6441_TSD_Products_Configuration_Guide_Chap
> ter.html
> 
> A control plane replay protection mechanism was added to Cisco IOS releases
> 12.4(15)T10, 12.4(22)T3, 12.4(24)T2, 15.0(1)M, and 12.2(33)XNE. This
> mechanism is not backward-compatible, so if any GET VPN group member in the
> network is running any of these (or later) releases, you must also upgrade
> all key servers to one of these (or newer) releases. Otherwise, network
> disruption might occur because of a failed rekey, which causes one of the
> following system logging (syslog) messages to appear:
> 
> %GDOI-3-GDOI_REKEY_SEQ_FAILURE: Failed to process rekey seq # 2 in seq
> payload for 
> group get-group, last seq # 6

DJS

View solution in original post

dsandre-toh · ‎05-18-2010

does this help: check ver# of each router:

> 
> 

http://www.ciscosystems.com/en/US/docs/ios/sec_secure_connectivity/configura
> tion/guide/sec_encrypt_trns_vpn_ps6441_TSD_Products_Configuration_Guide_Chap
> ter.html
> 
> A control plane replay protection mechanism was added to Cisco IOS releases
> 12.4(15)T10, 12.4(22)T3, 12.4(24)T2, 15.0(1)M, and 12.2(33)XNE. This
> mechanism is not backward-compatible, so if any GET VPN group member in the
> network is running any of these (or later) releases, you must also upgrade
> all key servers to one of these (or newer) releases. Otherwise, network
> disruption might occur because of a failed rekey, which causes one of the
> following system logging (syslog) messages to appear:
> 
> %GDOI-3-GDOI_REKEY_SEQ_FAILURE: Failed to process rekey seq # 2 in seq
> payload for 
> group get-group, last seq # 6

DJS

ggalteroo · ‎05-18-2010

That’s exactly right.

Thanks!

Rakhmadian Purnama · ‎12-18-2014

Hi all,

I have two Key Servers (KS1 and KS2 that running COOP) and some GMs. I'm running in two version of iOS; 15.0(1)M5 and 15.3(3)M2. While failover testing from KS1 to KS2, I've encountered same problem with this case. Both two KSs were running at 15.0(1)M5 version, but some GMs were running at 15.0(1)M5 and the rest was running at 15.3(3)M2.

I got this message on all GMs version, does it caused by the GM that running 15.3(3)M2 iOS version?

What should we do? Do we need upgrade all KSs and some GMs to 15.3(3)M2 or downgrade the rest of GMs to 15.0(1)M5?

It's seems strange, we never got this problem while KS1 running as Primary.

Thanks and Regards,

Rp