Multiple VRF BGP/GRE/IPsec Failing

Unanswered Question
May 18th, 2011

I'm trying to configure a Cisco 1941 to connect to multiple Amazon VPC instances. Each VPC instance brings up 2 x IPsec over GRE tunnels with BGP in to the EC2 cloud and enables flat extension of the corporate LAN. Basically. you can spin up EC2 instances in a private subnet and route to them across the VPC link from the corporate LAN.

The Amazon configuration is templated and not designed to support multiple instances on one customer access gateway - however, I want to overcome this and find a technical solution around bringing up a second physical router. I've got VRF configured and working for the first instance, but when we add a second VRF to the configuration IPsec fails. The second VRF is essentially identical to the first.

I've had someone I trust look at this and the only explanation they've been able to turn is that we're potentially looking at a licensing issue with IOS 15.x, the version we're running is...

      ipbase        ipbasek9      Permanent     ipbasek9
      security      securityk9    Permanent     securityk9
      data          None          None          None

When we bring the second VRF online the IPsec errors seen in the logs are...

*May  4 02:09:57.807: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of
      Main mode failed with peer at 72.21.209.193
      *May  4 02:10:02.807: %CRYPTO-6-IKMP_NOT_ENCRYPTED: IKE packet       from 72.21.209.225 was not encrypted and it should've been.
      *May  4 02:10:02.835: %CRYPTO-6-IKMP_NOT_ENCRYPTED: IKE packet       from 72.21.209.193 was not encrypted and it should've been.

However, the IPsec configuration is complete and all keychains etc. are in place as they should be. If anyone could provide guidance I'd greatly appreciate it.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Florin Barhala Thu, 05/19/2011 - 05:12

Pretty tricky . IPSEC config is placed under a VRF or in the Global Routing table?

Mohamed Sobair Thu, 05/19/2011 - 07:57

Hi,

I am not sure why VRF config here can impact your IPsec, I need some more details and attach your config please in order to look at it. (remove sensitive info like Public IPs..etc)

However, the Log message indicates that (IPsec Spoof detected), this means the encrypted packet should have been recieved encrypted but its not which causing you issues with tunnel.

I have observed such behaviour with a misconfigured Crypto or the Interesting traffic (Encrypted packet) are not taking the original IPsec path, casuing Assymetrical Path and therfore dropping your traffic. Since You cant loadshare traffic between Two Identical Ipsec tunnels, but you can have it as Backup.

Regards,

Mohamed

tim.march Thu, 05/19/2011 - 17:24

I've attached redacted copies of both configurations. The 1VPC configuration works correctly and we see the errors when running the 2VPC config. Some notes about the configuration...

  • Because the VPC gateway won't establish two connections to the same customer GW IP we get around this by adding a secondary IP to the Gi0/0 interface. IPsec connections from the vrf_172_20 VRF have their tunnel-source set to the primary and vrf_172_18 to the secondary IP.
  • To get around using route-maps we use two internal VLAN's and route each VRF via the gateway on each back to our core. This bit works fine and I don't suspect it's part of the problem.
  • All configuration (GRE / IPsec / BGP / NAT / Routing) works 100% correctly with the 1VPC configuration.

Wondering if the obscurity we're seeing might be due to the secondary address on Gi0/0, although this seems unlikely?

@Florin - It's all under the VRF.

Cheers,

T

tim.march Tue, 05/24/2011 - 19:23

Hi All - If anyone could privide feedback on this I'd really appreciate it?

Mohamed Sobair Wed, 05/25/2011 - 04:59

Hi Tim,

Apologize for the delay reply,

Could you please resend the running-config files again as they are not clear on the notepad. Please copy and paste it on a word document and send it back.

Regards,

Mohamed

tim.march Wed, 05/25/2011 - 17:11

Cheers for getting back to me. The files are just scp'd straight off the router itself. I'm not sure how Notepad treats these, but you should be able to open them in Wordpad if you're having trouble.

Mohamed Sobair Thu, 05/26/2011 - 06:16

Hi Tim,

I am afraid this problem is due to the Secondary Interface Sourcing the IPsec Security association with the IPsec peer. Even if you specify the option of (local address) in the crypto map , it will ONLY allow you to specify the Physical interface (Primary Interface) for IPsec tunnel for IPsec tunnel association.

Could you perhaps use two subinterfaces on G0/0 instead of the Secondary IP , and then associate the Source of the tunnel from each Subinterface.

Let me know How it goes.

Regards,

Mohamed

tim.march Thu, 05/26/2011 - 18:02

That makes sense, it's what I was suspecting. I'll break the public interface out in to vlans at the start of next week and post an update here. Many thanks for your assistance.

msv Wed, 09/21/2011 - 20:07

Hi Tim,

I need to implement a very similar setup.  Were you able to get it to work?  If so, any hints?

Thanks

ueda_i@worksap.co.jp Mon, 10/03/2011 - 03:42

--- SOLVED ---

I need help in configuring vrf with amazon vpc.

I'm very new to Cisco, and copied Tim's 1vrf configuration and it worked.

However, I think I have to setup nat and internal interface in order to connect to/from internal n/w

I think I'm having problems in 3 things below:

1. Our router is 892 and has only 1 GigabitEthernet, so I temporarily confitured as

    interface GigabitEthernet0

      ip address 221.xxx.xxx.xxx 255.255.255.248

      ip nat outside

      ip virtual-reassembly

      !

    !

    interface GigabitEthernet0.1

      !description --- vrf1 terminator

       encapsulation dot1Q 101

       ip vrf forwarding vrf1

       ip address 221.xxx.xxx.xxx 255.255.255.248

    I wonder if this is OK.  I can reach VPC instance by

    # ping vrf vrf1 ama.zon.vpc.xxx source GigabitEtherenet0.1

2. Currently FastEthernet8 is configured as internal interface with ip 172.19.xx.23, and

    can't change because this is the port I use to ssh to.  So I added a sub-interface :

    interface FastEthernet8.1

      !description --- internal network

      encapsulation dot1Q 81

      ip address 172.19.xx.23 255.255.255.0

    However, I can not connect to/from internal n/w even though 'ip show route vrf vrf1' shows

        172.19.xxx.xxx/27 is subnetted, 1 subnets

     C       172.19.xxx.0 is directly connected, FastEthernet8.1.

3. nat:

    ip route vrf vrf1 0.0.0.0 0.0.0.0 221.xxx.xxx.yyy global(next hop which I can not ping)

    ip route vrf vrf 1 172.xxx.xxx.xxx/12 (internal n/w) via 172.xxx.xxx.yyy(next hop which I can not ping)

Can anyone help configure?

Chris Dixon Fri, 03/08/2013 - 01:49

Hi,

I knwo this is really old, just cam across it looking for something else and rememebred the struggle I had to get this exact thing working with AWS. I did solve it after a few days melon scratching so if you still need it please let me know

Mohamed Sobair Fri, 03/08/2013 - 02:41

Chris,

Glad you have it solved!

Would you elaborate on the Steps you made to solve your problem, this could benefit other people in the Future.

Regards,

Mohamed

Mark Pottebaum Wed, 05/07/2014 - 13:13

I'm trying to do the same thing as you with AWS, would you be willing to send me a full config (with private info deleted)?

 

Thanks!

Chris Dixon Sat, 03/09/2013 - 03:48

Hi,

The key is to use different end points for the tunnel and keys for aws and do not use the local-address command or secondary addressing. I got it to work by using multiple vrfs and loopback interfaces for the tunnel endpoints. So, when you declare the crypto keyring be sure to tie it to the vrf containing that loop back for example
Ip vrf lo2
Cry keyring Aws2 vrf lo2

And then tie the vrf to the identity in isakmp profile eg
Cry isakmp profile awsprof
Keyring Aws2
Match identity address X.x.x.x lo2
Then within each tunnel you probably already are placing tunnel interfaces into vrfs to keep your vpc connections separate, using ip vrf forwarding, but you also need to specify a front door vrf for the tunnel (eg vrf to be used to reach tunnel peer) with the command tunnel vrf VrfName.

I think those are the bits that people struggle to get working, the other component is to make sure your loop backs can see the amazon public ip so you need to add a route into each external vrf to allow it to reach amazon public ip, and use mp-bgp to route between vrfs where you need to.
I hope this is detailed enough shout me if you're struggling to get it working or if you need anything in more detail

Sent from Cisco Technical Support iPhone App

jarnold@fuse.net Fri, 06/27/2014 - 07:18

I know this is an old thread, but how do you get the external routes to the VPC vrfs?  I'm struggling to get this working and any help would be great.  Could you provide a working config example?  Scrubbed, of course.  Thanks

ccallison@iname.com Fri, 01/23/2015 - 12:54

One thing I noticed in applying this template to my configuration (Cisco 2901 running 15.4(1)T2) was in the keyring configuration.

The example did not specify the local address statement, but Amazon's downloaded configuration always has it, so I added it to my configuration and was not able to get Phase 1 to come up.  Adding the VRF instance to the local address statement fixed it and now it comes up as expected.

Original snippet from template posted here:

crypto keyring GREEN-1 vrf GREEN
  pre-shared-key address 192.168.2.119 key ***

My initial version of the keyring including the local-address statement:

crypto keyring GREEN-1 vrf GREEN
  local-address 192.168.1.103
  pre-shared-key address 192.168.2.119 key ***

Correct usage of the local address in the keyring:

crypto keyring GREEN-1 vrf GREEN
  local-address 192.168.1.103 GREEN
  pre-shared-key address 192.168.2.119 key ***

Seems strange to have to specify the VRF instance twice but it worked.

Otherwise, this was a great starting template for setting up AWS tunnels.

Chris Dixon Fri, 06/27/2014 - 07:42

I'' try and have a look next week - if you don't hear from me post a comment to nudge me

 

Thanks

 

Chris

jarnold@fuse.net Thu, 07/10/2014 - 19:17

Here is a working config with three VRFs for three VPCs.  Since only the tunnel and endpoints might be duplicated, the VRFs only encompass the loopbacks and tunnel interfaces and I didn't extend them to the physical interfaces.

 

 

 

 !
ip vrf RED
 rd 65000:104
 route-target export 65000:104
 route-target import 65000:104
!
ip vrf BLUE
 rd 65000:105
 route-target export 65000:105
 route-target import 65000:105
!
ip vrf GREEN
 rd 65000:103
 route-target export 65000:103
 route-target import 65000:103
!
!
!
!
track 103 ip sla 103 reachability
!
track 104 ip sla 104 reachability
!
track 105 ip sla 105 reachability
!
track 203 ip sla 203 reachability
!
track 204 ip sla 204 reachability
!
track 205 ip sla 205 reachability
!
!
crypto keyring GREEN-1 vrf GREEN
  pre-shared-key address 192.168.2.119 key ***
crypto keyring GREEN-2 vrf GREEN
  pre-shared-key address 192.168.2.120 key ***
crypto keyring BLUE-1 vrf BLUE
  pre-shared-key address 192.168.3.42 key ***
crypto keyring BLUE-2 vrf BLUE
  pre-shared-key address 192.168.3.46 key ***
crypto keyring RED-1 vrf RED
  pre-shared-key address 192.168.3.46 key ***
crypto keyring RED-2 vrf RED
  pre-shared-key address 192.168.3.42 key ***
!
crypto isakmp policy 200
 encr aes
 authentication pre-share
 group 2
 lifetime 28800
!
crypto isakmp keepalive 10 10
!
crypto isakmp profile GREEN-1
   vrf GREEN
   keyring GREEN-1
   match identity address 192.168.2.119 255.255.255.255
   local-address 192.168.1.103
crypto isakmp profile GREEN-2
   vrf GREEN
   keyring GREEN-2
   match identity address 192.168.2.120 255.255.255.255
   local-address 192.168.1.103
crypto isakmp profile BLUE-1
   vrf BLUE
   keyring BLUE-1
   match identity address 192.168.3.42 255.255.255.255
   local-address 192.168.1.105
crypto isakmp profile BLUE-2
   vrf BLUE
   keyring BLUE-2
   match identity address 192.168.3.46 255.255.255.255
   local-address 192.168.1.105
crypto isakmp profile RED-1
   vrf RED
   keyring RED-1
   match identity address 192.168.3.46 255.255.255.255
   local-address 192.168.1.104
crypto isakmp profile RED-2
   vrf RED
   keyring RED-2
   match identity address 192.168.3.42 255.255.255.255
   local-address 192.168.1.104
!
crypto ipsec security-association replay window-size 128
!         
crypto ipsec transform-set AMAZON esp-aes esp-sha-hmac
 mode tunnel
crypto ipsec df-bit clear
!
crypto ipsec profile AMAZON
 set transform-set AMAZON
 set pfs group2
!
!
!
!
!
!
!
interface Loopback103
 ip vrf forwarding GREEN
 ip address 192.168.1.103 255.255.255.255
!
interface Loopback104
 ip vrf forwarding RED
 ip address 192.168.1.104 255.255.255.255
!
interface Loopback105
 ip vrf forwarding BLUE
 ip address 192.168.1.105 255.255.255.255
!
interface Tunnel103
 ip vrf forwarding GREEN
 ip address 192.168.4.26 255.255.255.252
 ip virtual-reassembly in
 ip tcp adjust-mss 1387
 tunnel source 192.168.1.103
 tunnel mode ipsec ipv4
 tunnel destination 192.168.2.119
 tunnel vrf GREEN
 tunnel protection ipsec profile AMAZON
!
interface Tunnel104
 ip vrf forwarding RED
 ip address 192.168.5.62 255.255.255.252
 ip virtual-reassembly in
 ip tcp adjust-mss 1387
 tunnel source 192.168.1.104
 tunnel mode ipsec ipv4
 tunnel destination 192.168.3.46
 tunnel vrf RED
 tunnel protection ipsec profile AMAZON
!
interface Tunnel105
 ip vrf forwarding BLUE
 ip address 192.168.5.58 255.255.255.252
 ip virtual-reassembly in
 ip tcp adjust-mss 1387
 tunnel source 192.168.1.105
 tunnel mode ipsec ipv4
 tunnel destination 192.168.3.42
 tunnel vrf BLUE
 tunnel protection ipsec profile AMAZON
!
interface Tunnel203
 ip vrf forwarding GREEN
 ip address 192.168.4.30 255.255.255.252
 ip virtual-reassembly in
 ip tcp adjust-mss 1387
 tunnel source 192.168.1.103
 tunnel mode ipsec ipv4
 tunnel destination 192.168.2.120
 tunnel vrf GREEN
 tunnel protection ipsec profile AMAZON
!
interface Tunnel204
 ip vrf forwarding RED
 ip address 192.168.5.58 255.255.255.252
 ip virtual-reassembly in
 ip tcp adjust-mss 1387
 tunnel source 192.168.1.104
 tunnel mode ipsec ipv4
 tunnel destination 192.168.3.42
 tunnel vrf RED
 tunnel protection ipsec profile AMAZON
!
interface Tunnel205
 ip vrf forwarding BLUE
 ip address 192.168.5.62 255.255.255.252
 ip virtual-reassembly in
 ip tcp adjust-mss 1387
 tunnel source 192.168.1.105
 tunnel mode ipsec ipv4
 tunnel destination 192.168.3.46
 tunnel vrf BLUE
 tunnel protection ipsec profile AMAZON
!
interface GigabitEthernet0/0
 description PUBLIC
 ip address 192.168.1.101 255.255.255.0
 ip access-group OUTSIDE in
 duplex auto
 speed auto
!
interface GigabitEthernet0/1
 description LAN
 ip address 192.168.10.101 255.255.0.0
 duplex auto
 speed auto
!
!
ip route vrf GREEN 172.20.0.0 255.255.0.0 192.168.4.25 track 103
ip route vrf RED 172.21.0.0 255.255.0.0 192.168.5.61 track 104
ip route vrf BLUE 172.22.0.0 255.255.0.0 192.168.5.57 track 105
ip route vrf GREEN 172.20.0.0 255.255.0.0 192.168.4.29 track 203
ip route vrf RED 172.21.0.0 255.255.0.0 192.168.5.57 track 204
ip route vrf BLUE 172.22.0.0 255.255.0.0 192.168.5.61 track 205
ip route 0.0.0.0 0.0.0.0 192.168.1.254
ip route 172.20.0.0 255.255.0.0 Tunnel103
ip route 172.20.0.0 255.255.0.0 Tunnel203
ip route 172.21.0.0 255.255.0.0 Tunnel104
ip route 172.21.0.0 255.255.0.0 Tunnel204
ip route 172.22.0.0 255.255.0.0 Tunnel105
ip route 172.22.0.0 255.255.0.0 Tunnel205
ip route 192.168.1.103 255.255.255.255 Loopback103
ip route 192.168.1.104 255.255.255.255 Loopback104
ip route 192.168.1.105 255.255.255.255 Loopback105
ip route vrf GREEN 0.0.0.0 0.0.0.0 192.168.1.254 global
ip route vrf GREEN 10.0.0.0 255.0.0.0 192.168.10.20 global
ip route vrf BLUE 0.0.0.0 0.0.0.0 192.168.1.254 global
ip route vrf BLUE 10.0.0.0 255.0.0.0 192.168.10.20 global
ip route vrf RED 0.0.0.0 0.0.0.0 192.168.1.254 global
ip route vrf RED 10.0.0.0 255.0.0.0 192.168.10.20 global
!
ip access-list extended OUTSIDE
 permit icmp any any
 permit ip host 192.168.2.119 any
 permit ip host 192.168.2.120 any
 permit ip host 192.168.3.42 any
 permit ip host 192.168.3.46 any
!
ip sla auto discovery
ip sla 103
 icmp-echo 192.168.4.25 source-interface Tunnel103
 vrf GREEN
 frequency 5
ip sla schedule 103 life forever start-time now
ip sla 104
 icmp-echo 192.168.5.61 source-interface Tunnel104
 vrf RED
 frequency 5
ip sla schedule 104 life forever start-time now
ip sla 105
 icmp-echo 192.168.5.57 source-interface Tunnel105
 vrf BLUE
 frequency 5
ip sla schedule 105 life forever start-time now
ip sla 203
 icmp-echo 192.168.4.29 source-interface Tunnel203
 vrf GREEN
ip sla schedule 203 life forever start-time now
ip sla 204
 icmp-echo 192.168.5.57 source-interface Tunnel204
 vrf RED
 frequency 5
ip sla schedule 204 life forever start-time now
ip sla 205
 icmp-echo 192.168.5.61 source-interface Tunnel205
 vrf BLUE
 frequency 5
ip sla schedule 205 life forever start-time now
!
!
end

Actions

Login or Register to take actions

This Discussion

Posted May 18, 2011 at 10:29 PM
Stats:
Replies:18 Overall Rating:
Views:2277 Votes:0
Shares:0
Tags: No tags.