Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. If you'd prefer to explore, try our test area to get started. And see here for current known issues.

New Member

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet connections

Hello

I have an AVPN cloud

My current sites have their own external IP subnet and an  centralized internet connection.

I am integrating sevetral new sites onto my avpn cloud.

The new sites have their own external ip subnet asnd a different centralized internet connection

Since we are all on 1 AVPN cloud, how do I force the new sites to use their centralized internet connection and not mine?

ip route 0.0.0.0 0.0.0.0 (WAN ip address of AVPN router that is located at site of their centralized internet connection)?

I would somehow need to direct throiugh the LAN side of the AVPN router so it could then be directed out their default gateway firewall?

sMc
2 ACCEPTED SOLUTIONS

Accepted Solutions

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Hi Steve,

I can imagine several possible reasons.

I'd need to know more details of your case:

IP addresses used for your BGP peering, GRE tunnels, sh ip bgp ... output for that subnet which is taking incorrect path, etc.

One crucial question though:

When your GRE tunnel is Up, is your BGP peering still Up on both sites?

Are you still receiving BGP prefixes?

Best regards,

Milan

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Hi Steve,

a) when your Site 3 is using the tunnel to reach the Internet through Site 5, how does the router on Site 5 forward the traffic returning from the Internet?

Is there a static route for Site 3 subnets configured with the Tunnel as next-hop?

Or is it just using the prefix received via BGP - an asymmetric routing used then though?

b) I suppose each site is using a different AS number?

c) Can you check on Site 1 by

sh ip bgp nei ... avd

if the router is advertising the LAN subnets correctly to the backbone?

Best regards,

Milan

19 REPLIES
New Member

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

I assume it will some sort of policy based routing.

Currently we get our defaulte route thru BGP

sMc
New Member

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Essentially, new sites want to cross our mutual AVPN and go out their own internet.

sMc
Hall of Fame Super Blue

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Steve

When you say AVPN are you referring to a VPN network ie. site to site VPNs from external sites ?

It's not clear how the topology is setup. Do VPNs terminate on the router ?  If so where is the firewall they are meant to use in relation to the router and which interfaces are used for the traffic ie.

remote site VPN -> WAN interface of router -> ??? 

Are the clients from remote sites then meant to go through their own firewall and be natted before going out ?

The way you have desribed it is that all VPNs terminate on one router and then you want a subset of those VPNs to go via a different firewall but i may have misunderstood.

Jon

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Hi,

are you talking about the AT&T AVPN MPLS service here?

If yes, then you need to make some agreement with them how to ensure some sites to prefer the default route advertised from one of your sites while other sites to use the default route advertised from another site.

They could play with communities or route targets, but the configuration would need to be changed on their PE routers.

Another possibilty would be creating tunnels from the sites to your  Intenet gateways - but quite complicated I'm afraid.

And from security point of view:

Why not to use proxy connections only to connect to the Internet?

Then you could simply configure per site (or even per user) which proxy to use.

Best regards,

Milan

New Member

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Jon and Milan

Thank you for your responses

You are correct in that we need to make an agreement with AT&T on how to differentiate traffic from various sites. Attached is a topology. The devices in red are the sites that need to be segmented, grouped off on their own.

I think the answer may be in BGP communities. Do you have any experience with this?

sMc
New Member

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

sMc
New Member

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

sMc

Re: 1 avpn cloud. 2 seperate External ip blocks. 2 seperate inte

Yes, 

we have done some prefix filtering based on communities in our network with AT&T and it worked fine.

But generally, I definitely prefer using proxies for an Internet access.

Best regards,

Milan

New Member

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Milan

Thank you for the response

Can you add more detail of how the proxy would work with this scenario

sMc

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Hi Steve,

many applications are proxy-capable nowadays (http, https browsing, ftp, etc.).

I.e., you can just configure on the client device which proxy (IP address and  tcp port) to use to to access the Internet.

And then you can have an Internet proxy device in each of  your DCs and you can just choose which proxy to use by the client settings.

You don't need any default route in an ideal case - your clients are connectiong to the proxies only.

And the proxies are connecting to the Internet then.

There are some applications not capable to use proxies sometimes.

In that case, they don't need to connect to the whole Internet though, just to a specific small subnet or even a host.

You could then advertise those small public destination prefixes from particular DCs to your network and make the applications to choose the proper Internet gateway this way.

Best regards,

Milan

New Member

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

All

GRE TUNNEL examples using statics to advertise the default route: It looks as though the interim solution will be to create a GRE tunnel from  (4) of the new sites to the (5th) new site that has the internet link advertising the 0.0.0.0 across the tunnel. anyone have experience in building that config?

The next step in the longer term solution is to create a separate AVPN for the (5) new sites ad use a service called UNILINK which allows access between AVPN tunnels for specific sites.

These (2) interim soltuins will give way to the final solution which is to integrate ALL sites onto the same AVPN network. The reason for delay is for security implementations.

sMc

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Hi,

yes, I can go the way with GRE tunnels, I think..

For a small number of sites.

You just need to be careful and avoid the default route received via a tunnel from being advertised back to the backbone.

And  there might be some issues with MTU through the tunnesl in theory. But current IOS should not suffer with that anymore.

Best regards,

Milan

New Member

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Milan

Thank you for the reply

I implemented the GRE tunnel in the (5) sites required. Now I am having BGP issues with several of those sites. When I traceroute from site_1 to LAN subnet being advertised via BGP on site_2, the GRE default route is taken instead of crossing the AVPN. The GRE tunnel appears to be blocking the BGP updates.

Any experience with this problem?

Any guidance/solution to help resolve this issue?

sMc

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Hi Steve,

I can imagine several possible reasons.

I'd need to know more details of your case:

IP addresses used for your BGP peering, GRE tunnels, sh ip bgp ... output for that subnet which is taking incorrect path, etc.

One crucial question though:

When your GRE tunnel is Up, is your BGP peering still Up on both sites?

Are you still receiving BGP prefixes?

Best regards,

Milan

New Member

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Milan

The way the GRE tunnel is setup is as follows

5 sites using tunnel

Site 5 is the Internet link for the other 4.

Site 5 has (4) GRE tunnel interfaces. The tunnel int's look like

Site_5#

interface Tunnel3

description GRE2_Site_3

bandwidth 10000

ip address 10.254.0.13 255.255.255.252

ip mtu 1400

ip tcp adjust-mss 1360

tunnel source 172.16.41.1 - SIte_5 WAN ip

tunnel destination 172.16.43.1 Site_3 WAN ip

Site_3#

interface Tunnel0

description GRE2_Site_5

ip address 10.254.0.14 255.255.255.252

ip mtu 1400

ip tcp adjust-mss 1360

tunnel source 172.16.43.1 Site_3 WAN ip

tunnel destination 172.16.41.1 Site_5 WAN ip

ip route 0.0.0.0 0.0.0.0 10.254.0.13

Site_3#sh ip bgp sum

BGP router identifier 172.16.43.1, local AS number 64xxy

BGP table version is 67, main routing table version 67

49 network entries using 7252 bytes of memory

49 path entries using 3136 bytes of memory

8/8 BGP path/bestpath attribute entries using 1088 bytes of memory

7 BGP AS-PATH entries using 168 bytes of memory

1 BGP community entries using 24 bytes of memory

0 BGP route-map cache entries using 0 bytes of memory

0 BGP filter-list cache entries using 0 bytes of memory

BGP using 11668 total bytes of memory

BGP activity 146/97 prefixes, 156/107 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

172.16.43.2     4        13979    5931    6495       67    0    0 4d02h          49

Site_3#

Site_3#sh ip bgp

BGP table version is 67, local router ID is 172.16.43.1

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,

              x best-external, a additional-path, c RIB-compressed,

Origin codes: i - IGP, e - EGP, ? - incomplete

RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path

r>  0.0.0.0          172.16.43.2                            0 13979 64531 i

*>  10.40.50.0/24 - Site_4 LAN subnets   172.16.43.2        0 13979 64550 i

*>  10.40.51.0/24    172.16.43.2                            0 13979 64550 i

*>  10.40.52.0/24    172.16.43.2                            0 13979 64550 i

    MISSING         Site_1 LAN subnets

    MISSING         Site_2 LAN subnets

*>  10.44.40.0/24 - Site_5 LAN subnets   172.16.43.2        0 13979 64554 i

*>  10.44.41.0/24    172.16.43.2                            0 13979 64554 i

*>  10.44.42.0/24    172.16.43.2                            0 13979 64554 i

BGP Peer WAN ip

*>  172.16.40.0/30      Site_4       172.16.43.2 - Site_3 BGP Peer (PER router) 0 13979 ?

*>  172.16.40.12/30      Site_2      172.16.43.2                            0 13979 ?

*>  172.16.41.0/30     Site_5      172.16.43.2                            0 13979 ?

*>  172.16.42.4/30    Site_1       172.16.43.2                            0 13979 ?

r>  172.16.43.0/30    Site_3        172.16.43.2              0             0 13979 ?

Sites 1, 2 & 3 are not advertising via BGP. The tunnel is used when tracerouting to the other sites.

sMc

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Hi Steve,

a) when your Site 3 is using the tunnel to reach the Internet through Site 5, how does the router on Site 5 forward the traffic returning from the Internet?

Is there a static route for Site 3 subnets configured with the Tunnel as next-hop?

Or is it just using the prefix received via BGP - an asymmetric routing used then though?

b) I suppose each site is using a different AS number?

c) Can you check on Site 1 by

sh ip bgp nei ... avd

if the router is advertising the LAN subnets correctly to the backbone?

Best regards,

Milan

New Member

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Milan

Site 5 has a default route that points to an L3 switch directly connected off the AVPN router Gi0/0 interface. that L3 switch has a default route pointing to the firewall and the mout the internet.

The Site_3 router has statics for the LAN pointing into the L3 switch Vlan management int. The BGP is redistributing statics. The ISP has the default originate command in it's BGP statements.

Each site has it'sown AS#

Site_1#sh ip bgp nei

BGP neighbor is 172.16.42.6,  remote AS 13aba, external link

  BGP version 4, remote router ID 12.123.x.y

  BGP state = Established, up for 20:04:42

  Last read 00:00:08, last write 00:00:04, hold time is 180, keepalive interval is 60 seconds

  Neighbor sessions:

    1 active, is not multisession capable (disabled)

  Neighbor capabilities:

    Route refresh: advertised and received(new)

    Four-octets ASN Capability: advertised and received

    Address family IPv4 Unicast: advertised and received

    Graceful Restart Capability: received

      Remote Restart timer is 120 seconds

      Address families advertised by peer:

        IPv4 Unicast (was not preserved

    Enhanced Refresh Capability: advertised

    Multisession Capability:

    Stateful switchover support enabled: NO for session 1

  Message statistics:

    InQ depth is 0

    OutQ depth is 0

                         Sent       Rcvd

    Opens:                  1          1

    Notifications:          0          0

    Updates:                1         28

    Keepalives:          1327       1203

    Route Refresh:          0          0

    Total:               1329       1232

  Default minimum time between advertisement runs is 30 seconds

For address family: IPv4 Unicast

  Session: 172.16.42.6

  BGP table version 49, neighbor version 49/0

  Output queue size : 0

  Index 2, Advertise bit 0

  2 update-group member

  AF-dependant capabilities:

    Outbound Route Filter (ORF) type (128) Prefix-list:

      Receive-mode: received

  Incoming update network filter list is 10

  Slow-peer detection is disabled

  Slow-peer split-update-group dynamic is disabled

                                 Sent       Rcvd

  Prefix activity:               ----       ----

    Prefixes Current:               0         39 (Consumes 2496 bytes)

    Prefixes Total:                 0         43

    Implicit Withdraw:              0          0

    Explicit Withdraw:              0          4

    Used as bestpath:             n/a         39

    Used as multipath:            n/a          0

                                   Outbound    Inbound

  Local Policy Denied Prefixes:    --------    -------

    distribute-list                       0         63

    Bestpath from this peer:             44        n/a

    Invalid Path:                         4        n/a

    Total:                               48         63

  Number of NLRIs in the update sent: max 3, min 0

  Last detected as dynamic slow peer: never

  Dynamic slow peer recovered: never

  Refresh Epoch: 1

  Last Sent Refresh Start-of-rib: never

  Last Sent Refresh End-of-rib: never

  Last Received Refresh Start-of-rib: never

  Last Received Refresh End-of-rib: never

                                       Sent       Rcvd

        Refresh activity:              ----       ----

          Refresh Start-of-RIB          0          0

          Refresh End-of-RIB            0          0

  Address tracking is enabled, the RIB does have a route to 172.16.42.6

  Connections established 2; dropped 1

  Last reset 20:04:55, due to Peer closed the session

  Transport(tcp) path-mtu-discovery is enabled

  Graceful-Restart is disabled

Connection state is ESTAB, I/O status: 1, unread input bytes: 0

Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1

Local host: 172.16.42.5, Local port: 46193

Foreign host: 172.16.42.6, Foreign port: 179

Connection tableid (VRF): 0

Maximum output segment queue size: 50

Enqueued packets for retransmit: 0, input: 0  mis-ordered: 0 (0 bytes)

Event Timers (current time is 0x572BEEBC):

Timer          Starts    Wakeups            Next

Retrans          1330          1             0x0

TimeWait            0          0             0x0

AckHold          1212       1192             0x0

SendWnd             0          0             0x0

KeepAlive           0          0             0x0

GiveUp              0          0             0x0

PmtuAger        71078      71077      0x572BF02F

DeadWait            0          0             0x0

Linger              0          0             0x0

ProcessQ            0          0             0x0

iss:  355325512  snduna:  355350806  sndnxt:  355350806

irs: 2461159334  rcvnxt: 2461184009

sndwnd:  32407  scale:      0  maxrcvwnd:  16384

rcvwnd:  15206  scale:      0  delrcvwnd:   1178

SRTT: 1000 ms, RTTO: 1003 ms, RTV: 3 ms, KRTT: 0 ms

minRTT: 4 ms, maxRTT: 1000 ms, ACK hold: 200 ms

Status Flags: active open

Option Flags: nagle, path mtu capable

IP Precedence value : 6

Datagrams (max data segment is 1460 bytes):

Rcvd: 2537 (out of order: 0), with data: 1211, total data bytes: 24674

Sent: 2543 (retransmit: 1, fastretransmit: 0, partialack: 0, Second Congestion: 0), with data: 1329, total data bytes: 25293

Packets received in fast path: 0, fast processed: 0, slow path: 0

fast lock acquisition failures: 0, slow path: 0

TCP Semaphore      0x2311C6A0  FREE

Site_1#

router bgp 64xxa

no bgp log-neighbor-changes

network 10.42.20.0 mask 255.255.255.0

network 10.42.21.0 mask 255.255.255.0

network 10.42.22.0 mask 255.255.255.0

redistribute static

neighbor 172.16.42.6 remote-as 13aba

ip route 10.42.20.0 255.255.255.0 10.42.21.2 track 2

ip route 10.42.21.0 255.255.255.0 10.42.21.2 track 2

ip route 10.42.22.0 255.255.255.0 10.42.21.2 track 2

ip route 0.0.0.0 0.0.0.0 10.254.0.9

sMc

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Hi Steve,

I don't fully understand your Site_1 configuration:

Is 10.42.21.0/24 directly connected?

Why are you using

redistribute static

and

network ...

command concurrently to get your static routes redistributed into BGP?

What are you tracking by track 2?

Is your provider configuring the CE router on your sites or are you configuring them?

In any case:

Your provider should be able to check if they are receiving the LAN prefixes from your Site_1

and if yes, if they are advertising them to your Site_3.

Best regards,

Milan

New Member

1 avpn cloud. 2 seperate External ip blocks. 2 seperate internet

Milan

Issue appears to be resolved

The LAN int was down and consequently there was no route to the networks since BGP does not advertise the route, rather it permits the  route to be advertised when it is there.

sMc
473
Views
0
Helpful
19
Replies