Using BGP RTBH in VPNv4

Document

Jan 8, 2011 2:22 PM
Jan 8th, 2011
  • OBJECTIVE:

This is a ‘how to’ type of article about implementation of the BGP RTBH feature in VPNv4 environment. We will briefly describe how this feature might be useful for an ISP and how to adapt it to the MPLS VPN service. Before reading further, it is advised to take a look on the RFC 5635, which describes the concepts behind it and on the Cisco's whitepaper, which details the implementation steps in IPv4 Unicast environment using Cisco IOS.

  • OVERVIEW:

In general, RTBH, which stands for Remotely Triggered Black Hole, is a BGP based mechanism to black-hole some traffic based on source or destination and thus there are two options exist for RTBH: Source based and Destination based. In this example we will focus on Source based implementation as it is more efficient than Destination based. 

For an ISP offering MPLS VPN service this feature might be needed if it also offers a ‘Central Services VPN’, like VoIP Services, “Internet access via VRF” or “Overlapping VPNs” to let customers interact and collaborate. In other words, when the number of VPNs is big and connectivity requirements are complex, making number of possible threats significant.

Typically black-holes are what we definitely do not want to have in our BGP core as it means that some traffic will be lost or dropped and we have a set of techniques to prevent such type of issues. But under certain conditions, like DDOS(*) attacks coming into our core and subsequently destined to our customers,  we might be interested to drop some traffic before it will flood vital circuits. There are plenty of solutions against such type of attacks, like for instance Cisco Traffic Anomaly Detector and third-party products.  In this scenario we are not going to discuss them further, as we are interested in a built-in IOS capability to defend a network against a DDOS attack. To be clear, there is no such a feature like “RTBH capability” listed in the Feature Navigator. This technique is solely based on normal BGP operations.

One may recall the old-days CAR - Committed Access Rate approach, available via ‘rate-limit’ command under interface configuration mode and used to restrict amount of traffic coming in/out based on the configured limit (nowadays it is more or less substituted by set of MQC style commands). This is also a good and easy tool to protect network core and customers’ circuits against flood. The only thing missing here is…scalability. Imagine you have a dozen of border routers and hundreds of exit interfaces. Even assuming that you need to configure only external interfaces to drop unwanted traffic, it turns out to be a configuration overhead. The BGP RTBH way is much more automated compared to the CAR approach. However, since we do not use any access/prefix-lists for filtering in RTBH, there is no way to select which type of traffic to drop.

For a global view of the methodology how to mitigate DDOS attacks please review RFC 2827.

For basic configuration examples on IPv4 unicast please refer to the document specified in the beginning of this article.

In this case-study we are using VPNv4 as infrastructure.

LEGEND:

RTBH2.1.jpg

Let’s assume that we are running a big ISP under BGP AS 65023 and among others offering MPLS VPN service to our customers. The big ISP means lots of iBGP sessions (we don’t like synchronization ) and this is why we would usually divide a network into multiple confederations or deploy Route-Reflectors. In this case-study R2 acts as RR, R3 is a client of R2 for VPNv4 iBGP, and R1 belongs to a vrf TYCHO, using eBGP as a PE-CE protocol. Another possible implementation of eBGP in a VPNv4 environment and thus relevant to our setup might be the Option B for Inter-AS VPN deployment - Single-hop Multiprotocol eBGP for VPNv4.

On the diagram R3 has a vrf named BAUD, where malicious host(172.16.1.254/32) resides – the attacker. In Source based RTBH filtering for a given source will occur on every router which learnt the advertisement, no matter where exactly an attacker resides.

To generate an advertisement we normally use a special device called ‘trigger’ to send a BGP update to the rest of the network. It might be a separate BGP speaker or even a host running a BGP daemon. The only requirement is that the trigger should have BGP peerings with all the routers, or with a route-reflector in case one is deployed. In our case R2 will act as a trigger too.

Let’s take a look on the initial configurations:

hostname R2
hostname R3
!
ip vrf TYCHO
rd 65001:1
route-target export 65001:1
route-target import 65001:1
route-target import 65123:123
!
router bgp 65023
no bgp default ipv4-unicast
bgp log-neighbor-changes
neighbor 1.1.1.3 remote-as 65023
neighbor 1.1.1.3 update-source Loopback0
!
address-family vpnv4
  neighbor 1.1.1.3 activate
  neighbor 1.1.1.3 send-community extended
  neighbor 1.1.1.3 route-reflector-client
  exit-address-family
!
address-family ipv4 vrf TYCHO
  neighbor 10.0.12.1 remote-as 65001
  neighbor 10.0.12.1 activate
exit-address-family
!
!
ip vrf BAUD
rd 65123:123
route-target export 65123:123
route-target import 65123:123
!
router bgp 65023
no bgp default ipv4-unicast
bgp log-neighbor-changes
neighbor 1.1.1.2 remote-as 65023
neighbor 1.1.1.2 update-source Loopback0
!
address-family vpnv4
  neighbor 1.1.1.2 activate
  neighbor 1.1.1.2 send-community extended
exit-address-family
!
address-family ipv4 vrf BAUD
  no synchronization
  network 172.16.1.0 mask 255.255.255.0
exit-address-family
!
hostname R1
!
router bgp 65001
no bgp default ipv4-unicast
bgp log-neighbor-changes
neighbor 10.0.12.2 remote-as 65023
!
address-family ipv4
  neighbor 10.0.12.2 activate
  no auto-summary
  no synchronization
exit-address-family
!

All sessions are up:

R2#sh bgp vpn u all s


Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
1.1.1.3         4 65023    5896    5912        9    0    0 06:02:07        1
10.0.12.1       4 65001     500     532        9    0    0 06:02:06        0

  • CONFIGURATION:

Now let’s introduce RTBH configuration on the ‘trigger’ router R2:

!

route-map TRIGGER permit 10       

match tag 666
set local-preference 200  <-------- optional

set ip next-hop 192.0.2.1
set community 666 <----------- will be used later

!

To make next-hop 192.0.2.1/32 reachable we need to introduce it to vrf BAUD, but before doing that, we have to define a vrf, which is normally not the case on a RR:

!
ip vrf BAUD                                                      
rd 65123:123

route-target export 65123:123                                        
route-target import 65123:123
!

This might be considered as a drawback, depending on how many offending vrf’s will be there, because each of them should have been defined prior to IOS will let a static route to be created:

ip route vrf BAUD 192.0.2.1 255.255.255.255 Null0

Instead of creating multiple vrf’s on a RR, we can use one dummy vrf and fine-tune routes to be black-holed using the export-map feature:

!
ip vrf PROXY
rd 65123:2
export map RTBH
!
ip prefix-list RTBH_BAUD seq 5 permit 172.16.1.254/32
!
ip prefix-list RTBH_TYCHO seq 5 permit 172.16.1.254/32                          

!
route-map RTBH permit 10
match ip address prefix-list RTBH_BAUD
set extcommunity rt  65123:123
!
route-map RTBH permit 20   
match ip address prefix-list RTBH_TYCHO
set extcommunity rt  65001:1                                                 
!

Then the next-hop will be defined as:

ip route vrf PROXY 192.0.2.1 255.255.255.255 Null0 <------ all packets will be dropped

Assume we have identified an attacker using a monitoring system(**) and his/her IP address is 172.16.1.254/32. Then we’d need to create a triggering route to let other routers know how to treat it:

ip route vrf PROXY 172.16.1.254 255.255.255.255 192.0.2.1 tag 666

Before the route will be installed into BGP/RIB tables, redistribution should be in place:

!
router bgp 65023    

address-family ipv4 vrf PROXY
redistribute static route-map TRIGGER                                                

!

And then we can see following:

R2(config)#do sh ip ro vrf PROXY

Routing Table: PROXY

     172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks

S       172.16.1.254/32 [1/0] via 192.0.2.1

S       192.0.2.1 is directly connected, Null0

R2(config)#do sh bgp vp u all

  Network          Next Hop            Metric LocPrf Weight Path

Route Distinguisher: 65001:1 (default for vrf TYCHO)

*>i172.16.1.0/24    1.1.1.3                  0    100      0 I                                                                                               

*> 172.16.1.254/32  192.0.2.1                0    200  32768 ?

Route Distinguisher: 65123:2 (default for vrf PROXY)

*> 172.16.1.254/32  192.0.2.1                0    200  32768 ?

Route Distinguisher: 65123:123

*>i172.16.1.0/24    1.1.1.3                  0    100      0 i
<!--[if !supportLineBreakNewLine]-->
<!--[endif]-->

So far all looks fine. Let’s check other two routers:

R3(config-router-af)#do sh bgp vp u all

   Network          Next Hop            Metric LocPrf Weight Path

Route Distinguisher: 65123:123 (default for vrf BAUD)

*> 172.16.1.0/24    0.0.0.0                  0         32768 I                                                         

*> i172.16.1.254/32  1.1.1.2                 0    200      0 ?

R1(config-router-af)#do sh ip bgp


   Network          Next Hop            Metric LocPrf Weight Path

*> 172.16.1.0/24    10.0.12.2                              0 65023 I                                                                                        

*> 172.16.1.254/32  10.0.12.2                              0 65023 ?

The route is present on both routers and apparently the next-hops are the peer-addresses instead of being 192.0.2.1 set by route-map TRIGGER.

The thing is that for VPNv4 prefixes next-hop address will be always set to a peer address, unless it hasn’t been modified explicitly:

On R2:

!
ip community-list 66 permit 666
    <---- remember we have set a community value 666 in route-map TRIGGER          
!

route-map NH_R3 permit 10
match community 66       
set ip next-hop 192.0.2.1
!
route-map NH_R3 permit 20                                                        
!
route-map NH_R1 permit 10
match community 66
set ip next-hop 192.0.2.1
!
route-map NH_R1 permit 20
!
router bgp 65023                                                       
!
address-family vpnv4
neighbor 1.1.1.3 route-map NH_R3 out
!                                                                                                                   

address-family ipv4 vrf TYCHO                                              
neighbor 10.0.12.1 route-map NH_R1 out
!

On both R1 and R2 outputs are better, but different:

R3(config)#do sh bgp vp u all

   Network          Next Hop            Metric LocPrf Weight Path

Route Distinguisher: 65123:123 (default for vrf BAUD)

*> 172.16.1.0/24    0.0.0.0                  0         32768 i

* i172.16.1.254/32  192.0.2.1                0    200      0 ?

R1#sh ip bgp

   Network          Next Hop            Metric LocPrf Weight Path
*> 172.16.1.0/24    10.0.12.2                              0 65023 i

So on R3 which is connected via iBGP the prefix is learnt, but not picked up as best, and on R1 there is no attacker's prefix at all.

To let R3 install the prefix we should tell it how to reach 192.0.2.1, i.e. we need a static route. Since we are dealing with vrf it would be logical to create a host route for 192.0.2.1 under vrf BAUD pointing to the Null0 interface to actually black-hole traffic, like this:

R3(config)#ip route vrf BAUD 192.0.2.1 255.255.255.255 null 0

But this will never allow R3 to install 172.16.1.254 into RIB:

R3(config)#do sh bgp vp u all

   Network          Next Hop            Metric LocPrf Weight Path

Route Distinguisher: 65123:123 (default for vrf BAUD)

*> 172.16.1.0/24    0.0.0.0                  0         32768 i

* i172.16.1.254/32  192.0.2.1                0    200      0 ?

Actually ‘debug  bgp vpnv4 unicast’ shows following output:

2d06h: BGP(2): no valid path for 65123:123:172.16.1.254/32

  • EXPLANATION:

The reason why is because the next-hop resolution process for the VPNv4 routes will use the global routing table and the static route for 192.0.2.1/32 therefore needs to be configured in the global routing table:

R3(config)#ip route 192.0.2.1 255.255.255.255 null 0

R3(config)#

2d06h: BGP(2): Revise route installing 1 of 1 routes for 172.16.1.254/32 -> 192.0.2.1(main) to BAUD IP table

R3(config)#do sh bg vp u al                                              

   Network          Next Hop            Metric LocPrf Weight Path

Route Distinguisher: 65123:123 (default for vrf BAUD)

*> 172.16.1.0/24    0.0.0.0                  0         32768 i

*>i172.16.1.254/32  192.0.2.1                0    200      0 ?

Let’s configure the route 192.0.2.1/32 on R1 as well:    

                                                                          

R1(config)#ip route 192.0.2.1 255.255.255.255 Null0

Nothing changed:

R1(config-router-af)#do sh bgp vp u all

   Network          Next Hop            Metric LocPrf Weight Path

Route Distinguisher: 65123:123 (default for vrf BAUD)

*> 172.16.1.0/24    10.0.12.2                0             0 65023 i

Despites the fact:

R1(config)#do sh ip ro 192.0.2.1

Routing entry for 192.0.2.1/32

  Known via "static", distance 1, metric 0 (connected)

  Routing Descriptor Blocks:

  * directly connected, via Null0

      Route metric is 0, traffic share count is 1

Route 172.16.1.254/32 is neither installed in RIB, nor being learnt in BGP table…


However, when we enable ‘debug bgp ipv4 u u’ on R1, we see:

BGP: 10.0.12.2 Advertised Nexthop 192.0.2.1: Non-local or Nexthop and peer Not on same interface

BGP(0): 10.0.12.2 rcv UPDATE about 172.16.1.254/32 -- DENIED due to: non-connected NEXTHOP;

BGP(0): no valid path for 172.16.1.254/32

  • EXPLANATION:

The BGP session between R1 and R2 is eBGP, therefore uses directly connected interfaces for peerings. By default IOS checks that the next-hop of a received update via eBGP belongs to a subnet directly connected to the local router. Otherwise next-hop attributes for those sessions would be useless. In our case the route to 192.0.2.1 considered not directly connected because the next hop(which is Null0 interface) is not on the same subnet(10.0.12.0/24) as both peer addresses. In turn it fails default verification requirements.

To overcome this requirement we can use commonly used approach by allowing multiple hops between eBGP neighbors:

R1(config-router-af)#nei 10.0.12.2 ebgp-multihop 2

Short afterwards we see:

BGP(0): Revise route installing 1 of 1 routes for 172.16.1.254/32 -> 192.0.2.1(main) to main IP table


R1(config-router)#do sh bgp ipv4 u

   Network          Next Hop            Metric LocPrf Weight Path

*> 172.16.1.0/24    10.0.12.2              0     0           65023 I                                                                                                 

*> 172.16.1.254/32  192.0.2.1              0     0           65023 ?

Fine tuning:

You might have noticed the command 'no ip unreachables' under interface Null0:

!

interface Null0

no ip unreachables

!

By default Cico IOS, as a good Internet citizen, will send an ICMP 3/1 message once it receives an unroutable packet. Which is 'should' and good habit in a friendly environment. However, since we are defendig against a DDOS attack, sending too much unreachables may overload a router itself. Apart from powerful CoPP, there is a built-in rate-limiter for ICMP type 3 messages with default of 1 message per 500 ms. It would be also a good idea to disable them for the Null0 interface, since it is mainly being used for intentional drops. Here is a good discussion about this command.

UPD:

Technically speaking, we have achieved the desired outcome and adapted BGP RTBH for VPNv4 prefixes, but there is another technique and RFC worth to mention here: RFC 3704 - Ingress Filtering for Multihomed Networks, which can complement Source-based RTBH.If enabled on autonomus system edge interfaces, it will check every incoming packet to verify whether it has corresponding route enry in the router FIB, and if not drop it. This effectively prevents spoofing attacks.

The command used to make it works is:

R1(config-if)# ip verify unicast source reachable-via any

In our setup we could enable it on R3's interface facing BAUD and on R1's interface facing BGP core, since DDOS supposed to come from there.

If you are interested to learn more about uRPF, please follow this link.

FINAL CONFIGS:

hostname R2
hostname R3
!      
ip vrf PROXY
rd 65123:2
export map RTBH
!
ip vrf TYCHO
rd 65001:1
route-target export 65001:1
route-target import 65001:1
route-target import 65123:123
!
interface Null0
no ip unreachables
!
router bgp 65023
no bgp default ipv4-unicast
bgp log-neighbor-changes
neighbor 1.1.1.3 remote-as 65023
neighbor 1.1.1.3 update-source Loopback0
!
address-family vpnv4
  neighbor 1.1.1.3 activate
  neighbor 1.1.1.3 send-community extended
  neighbor 1.1.1.3 route-reflector-client
  neighbor 1.1.1.3 route-map NH_R3 out
exit-address-family
!
address-family ipv4 vrf TYCHO
  neighbor 10.0.12.1 remote-as 65001
  neighbor 10.0.12.1 activate
  neighbor 10.0.12.1 route-map NH_R1 out
  no synchronization
exit-address-family
!
address-family ipv4 vrf PROXY
  redistribute static route-map TRIGGER
  no synchronization
exit-address-family
!
ip route vrf PROXY 172.16.1.254 255.255.255.255 192.0.2.1 tag 666
ip route vrf PROXY 192.0.2.1 255.255.255.255 Null0
!
ip community-list 66 permit 666
!
ip prefix-list RTBH_BAUD seq 5 permit 172.16.1.254/32
!
ip prefix-list RTBH_TYCHO seq 5 permit 172.16.1.254/32
!
route-map RTBH permit 10
match ip address prefix-list RTBH_BAUD
set extcommunity rt  65123:123
!
route-map RTBH permit 20
match ip address prefix-list RTBH_TYCHO
set extcommunity rt  65001:1
!
route-map NH_R3 permit 10
match community 66
set ip next-hop 192.0.2.1
!      
route-map NH_R3 permit 20
!
route-map NH_R1 permit 10
match community 66
set ip next-hop 192.0.2.1
!
route-map NH_R1 permit 20
!
route-map TRIGGER permit 10
match tag 666
set local-preference 200
set community 666
set ip next-hop 192.0.2.1 ---> turned out to be useless for VPNv4, can be omitted
!
!
ip vrf BAUD
rd 65123:123
route-target export 65123:123
route-target import 65123:123
!
interface Null0
no ip unreachable
!
router bgp 65023
no bgp default ipv4-unicast
bgp log-neighbor-changes
neighbor 1.1.1.2 remote-as 65023
neighbor 1.1.1.2 update-source Loopback0
!
address-family vpnv4
  neighbor 1.1.1.2 activate
  neighbor 1.1.1.2 send-community extended
exit-address-family
!
address-family ipv4 vrf BAUD
  no synchronization
  network 172.16.1.0 mask 255.255.255.0
exit-address-family
!
ip route 192.0.2.1 255.255.255.255 Null0
!

hostname R1

!
interface Null0
  no ip unreachable
!

router bgp 65001
no bgp default ipv4-unicast
bgp log-neighbor-changes
neighbor 10.0.12.2 remote-as 65023
neighbor 10.0.12.2 ebgp-multihop 2
!
address-family ipv4
  neighbor 10.0.12.2 activate
  no auto-summary
  no synchronization
exit-address-family
!
ip route 192.0.2.1 255.255.255.255 Null0
!

* You might be interested to learn the Cisco Vision about DDOS attacks.

** Instead of using separate NMS station, you can utilize Netflow to identify offensive flows coming in.

If you have any questions, suggestions or want to add something to the subject being discussed, please post your input in the comments section.

·

Average Rating: 4.5 (2 ratings)

Comments

Laurent Aubert Thu, 01/13/2011 - 22:14

Hi,

It seems you forgot the main piece required for source based RTBH: Loose URPF on PE edge interfaces.

Also I don't understand this static route configured on R3: ip route vrf BAUD 172.16.1.200 255.255.255.255 Null0

Who is 172.16.1.200 ?

Thanks

Laurent.

Ivan Krimmel Mon, 01/17/2011 - 05:42 (reply to Laurent Aubert)

Hi Laurent,

thanks for pointing this out, indeed I was primarily focusing on VPNv4 fixes and forgot to mention uRPF supplement. As for 1.200, it was a stale route left after some other tests. I have fixed both points.

Thanks,

Ivan.

Actions

Login or Register to take actions

This Document

Posted January 8, 2011 at 2:22 PM
Stats:
Comments:2 Avg. Rating:4.5
Views:5936 Contributors:2
Shares:0

Related Content