- OBJECTIVE:
This is a ‘how to’ type of article about implementation of the BGP RTBH feature in VPNv4 environment. We will briefly describe how this feature might be useful for an ISP and how to adapt it to the MPLS VPN service. Before reading further, it is advised to take a look on the RFC 5635, which describes the concepts behind it and on the Cisco's whitepaper, which details the implementation steps in IPv4 Unicast environment using Cisco IOS.
- OVERVIEW:
In general, RTBH, which stands for Remotely Triggered Black Hole, is a BGP based mechanism to black-hole some traffic based on source or destination and thus there are two options exist for RTBH: Source based and Destination based. In this example we will focus on Source based implementation as it is more efficient than Destination based.
For an ISP offering MPLS VPN service this feature might be needed if it also offers a ‘Central Services VPN’, like VoIP Services, “Internet access via VRF” or “Overlapping VPNs” to let customers interact and collaborate. In other words, when the number of VPNs is big and connectivity requirements are complex, making number of possible threats significant.
Typically black-holes are what we definitely do not want to have in our BGP core as it means that some traffic will be lost or dropped and we have a set of techniques to prevent such type of issues. But under certain conditions, like DDOS(*) attacks coming into our core and subsequently destined to our customers, we might be interested to drop some traffic before it will flood vital circuits. There are plenty of solutions against such type of attacks, like for instance Cisco Traffic Anomaly Detector and third-party products. In this scenario we are not going to discuss them further, as we are interested in a built-in IOS capability to defend a network against a DDOS attack. To be clear, there is no such a feature like “RTBH capability” listed in the Feature Navigator. This technique is solely based on normal BGP operations.
One may recall the old-days CAR - Committed Access Rate approach, available via ‘rate-limit’ command under interface configuration mode and used to restrict amount of traffic coming in/out based on the configured limit (nowadays it is more or less substituted by set of MQC style commands). This is also a good and easy tool to protect network core and customers’ circuits against flood. The only thing missing here is…scalability. Imagine you have a dozen of border routers and hundreds of exit interfaces. Even assuming that you need to configure only external interfaces to drop unwanted traffic, it turns out to be a configuration overhead. The BGP RTBH way is much more automated compared to the CAR approach. However, since we do not use any access/prefix-lists for filtering in RTBH, there is no way to select which type of traffic to drop.
For a global view of the methodology how to mitigate DDOS attacks please review RFC 2827.
For basic configuration examples on IPv4 unicast please refer to the document specified in the beginning of this article.
In this case-study we are using VPNv4 as infrastructure.
LEGEND:
Let’s assume that we are running a big ISP under BGP AS 65023 and among others offering MPLS VPN service to our customers. The big ISP means lots of iBGP sessions (we don’t like synchronization ) and this is why we would usually divide a network into multiple confederations or deploy Route-Reflectors. In this case-study R2 acts as RR, R3 is a client of R2 for VPNv4 iBGP, and R1 belongs to a vrf TYCHO, using eBGP as a PE-CE protocol. Another possible implementation of eBGP in a VPNv4 environment and thus relevant to our setup might be the Option B for Inter-AS VPN deployment - Single-hop Multiprotocol eBGP for VPNv4.
On the diagram R3 has a vrf named BAUD, where malicious host(172.16.1.254/32) resides – the attacker. In Source based RTBH filtering for a given source will occur on every router which learnt the advertisement, no matter where exactly an attacker resides.
To generate an advertisement we normally use a special device called ‘trigger’ to send a BGP update to the rest of the network. It might be a separate BGP speaker or even a host running a BGP daemon. The only requirement is that the trigger should have BGP peerings with all the routers, or with a route-reflector in case one is deployed. In our case R2 will act as a trigger too.
Let’s take a look on the initial configurations:
| hostname R2 | hostname R3 |
|---|---|
| ! ip vrf TYCHO rd 65001:1 route-target export 65001:1 route-target import 65001:1 route-target import 65123:123 ! router bgp 65023 no bgp default ipv4-unicast bgp log-neighbor-changes neighbor 1.1.1.3 remote-as 65023 neighbor 1.1.1.3 update-source Loopback0 ! address-family vpnv4 neighbor 1.1.1.3 activate neighbor 1.1.1.3 send-community extended neighbor 1.1.1.3 route-reflector-client exit-address-family ! address-family ipv4 vrf TYCHO neighbor 10.0.12.1 remote-as 65001 neighbor 10.0.12.1 activate exit-address-family ! | ! ip vrf BAUD rd 65123:123 route-target export 65123:123 route-target import 65123:123 ! router bgp 65023 no bgp default ipv4-unicast bgp log-neighbor-changes neighbor 1.1.1.2 remote-as 65023 neighbor 1.1.1.2 update-source Loopback0 ! address-family vpnv4 neighbor 1.1.1.2 activate neighbor 1.1.1.2 send-community extended exit-address-family ! address-family ipv4 vrf BAUD no synchronization network 172.16.1.0 mask 255.255.255.0 exit-address-family ! |
| hostname R1 |
|---|
| ! router bgp 65001 no bgp default ipv4-unicast bgp log-neighbor-changes neighbor 10.0.12.2 remote-as 65023 ! address-family ipv4 neighbor 10.0.12.2 activate no auto-summary no synchronization exit-address-family ! |
All sessions are up:
R2#sh bgp vpn u all s
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
1.1.1.3 4 65023 5896 5912 9 0 0 06:02:07 1
10.0.12.1 4 65001 500 532 9 0 0 06:02:06 0
- CONFIGURATION:
Now let’s introduce RTBH configuration on the ‘trigger’ router R2:
!
route-map TRIGGER permit 10
match tag 666
set local-preference 200 <-------- optional
set ip next-hop 192.0.2.1
set community 666 <----------- will be used later
!
To make next-hop 192.0.2.1/32 reachable we need to introduce it to vrf BAUD, but before doing that, we have to define a vrf, which is normally not the case on a RR:
!
ip vrf BAUD
rd 65123:123
route-target export 65123:123
route-target import 65123:123
!
This might be considered as a drawback, depending on how many offending vrf’s will be there, because each of them should have been defined prior to IOS will let a static route to be created:
ip route vrf BAUD 192.0.2.1 255.255.255.255 Null0
Instead of creating multiple vrf’s on a RR, we can use one dummy vrf and fine-tune routes to be black-holed using the export-map feature:
!
ip vrf PROXY
rd 65123:2
export map RTBH
!
ip prefix-list RTBH_BAUD seq 5 permit 172.16.1.254/32
!
ip prefix-list RTBH_TYCHO seq 5 permit 172.16.1.254/32
!
route-map RTBH permit 10
match ip address prefix-list RTBH_BAUD
set extcommunity rt 65123:123
!
route-map RTBH permit 20
match ip address prefix-list RTBH_TYCHO
set extcommunity rt 65001:1
!
Then the next-hop will be defined as:
ip route vrf PROXY 192.0.2.1 255.255.255.255 Null0 <------ all packets will be dropped
Assume we have identified an attacker using a monitoring system(**) and his/her IP address is 172.16.1.254/32. Then we’d need to create a triggering route to let other routers know how to treat it:
ip route vrf PROXY 172.16.1.254 255.255.255.255 192.0.2.1 tag 666
Before the route will be installed into BGP/RIB tables, redistribution should be in place:
!
router bgp 65023
address-family ipv4 vrf PROXY
redistribute static route-map TRIGGER
!
And then we can see following:
R2(config)#do sh ip ro vrf PROXY
Routing Table: PROXY
172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
S 172.16.1.254/32 [1/0] via 192.0.2.1
S 192.0.2.1 is directly connected, Null0
R2(config)#do sh bgp vp u all
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65001:1 (default for vrf TYCHO)
*>i172.16.1.0/24 1.1.1.3 0 100 0 I
*> 172.16.1.254/32 192.0.2.1 0 200 32768 ?
Route Distinguisher: 65123:2 (default for vrf PROXY)
*> 172.16.1.254/32 192.0.2.1 0 200 32768 ?
Route Distinguisher: 65123:123
*>i172.16.1.0/24 1.1.1.3 0 100 0 i
<!--[if !supportLineBreakNewLine]-->
<!--[endif]-->
So far all looks fine. Let’s check other two routers:
R3(config-router-af)#do sh bgp vp u all
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65123:123 (default for vrf BAUD)
*> 172.16.1.0/24 0.0.0.0 0 32768 I
*> i172.16.1.254/32 1.1.1.2 0 200 0 ?
R1(config-router-af)#do sh ip bgp
Network Next Hop Metric LocPrf Weight Path
*> 172.16.1.0/24 10.0.12.2 0 65023 I
*> 172.16.1.254/32 10.0.12.2 0 65023 ?
The route is present on both routers and apparently the next-hops are the peer-addresses instead of being 192.0.2.1 set by route-map TRIGGER.
The thing is that for VPNv4 prefixes next-hop address will be always set to a peer address, unless it hasn’t been modified explicitly:
On R2:
!
ip community-list 66 permit 666 <---- remember we have set a community value 666 in route-map TRIGGER
!
route-map NH_R3 permit 10
match community 66
set ip next-hop 192.0.2.1
!
route-map NH_R3 permit 20
!
route-map NH_R1 permit 10
match community 66
set ip next-hop 192.0.2.1
!
route-map NH_R1 permit 20
!
router bgp 65023
!
address-family vpnv4
neighbor 1.1.1.3 route-map NH_R3 out
!
address-family ipv4 vrf TYCHO
neighbor 10.0.12.1 route-map NH_R1 out
!
On both R1 and R2 outputs are better, but different:
R3(config)#do sh bgp vp u all
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65123:123 (default for vrf BAUD)
*> 172.16.1.0/24 0.0.0.0 0 32768 i
* i172.16.1.254/32 192.0.2.1 0 200 0 ?
R1#sh ip bgp
Network Next Hop Metric LocPrf Weight Path
*> 172.16.1.0/24 10.0.12.2 0 65023 i
So on R3 which is connected via iBGP the prefix is learnt, but not picked up as best, and on R1 there is no attacker's prefix at all.
To let R3 install the prefix we should tell it how to reach 192.0.2.1, i.e. we need a static route. Since we are dealing with vrf it would be logical to create a host route for 192.0.2.1 under vrf BAUD pointing to the Null0 interface to actually black-hole traffic, like this:
R3(config)#ip route vrf BAUD 192.0.2.1 255.255.255.255 null 0
But this will never allow R3 to install 172.16.1.254 into RIB:
R3(config)#do sh bgp vp u all
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65123:123 (default for vrf BAUD)
*> 172.16.1.0/24 0.0.0.0 0 32768 i
* i172.16.1.254/32 192.0.2.1 0 200 0 ?
Actually ‘debug bgp vpnv4 unicast’ shows following output:
2d06h: BGP(2): no valid path for 65123:123:172.16.1.254/32
- EXPLANATION:
The reason why is because the next-hop resolution process for the VPNv4 routes will use the global routing table and the static route for 192.0.2.1/32 therefore needs to be configured in the global routing table:
R3(config)#ip route 192.0.2.1 255.255.255.255 null 0
R3(config)#
2d06h: BGP(2): Revise route installing 1 of 1 routes for 172.16.1.254/32 -> 192.0.2.1(main) to BAUD IP table
R3(config)#do sh bg vp u al
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65123:123 (default for vrf BAUD)
*> 172.16.1.0/24 0.0.0.0 0 32768 i
*>i172.16.1.254/32 192.0.2.1 0 200 0 ?
Let’s configure the route 192.0.2.1/32 on R1 as well:
R1(config)#ip route 192.0.2.1 255.255.255.255 Null0
Nothing changed:
R1(config-router-af)#do sh bgp vp u all
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65123:123 (default for vrf BAUD)
*> 172.16.1.0/24 10.0.12.2 0 0 65023 i
Despites the fact:
R1(config)#do sh ip ro 192.0.2.1
Routing entry for 192.0.2.1/32
Known via "static", distance 1, metric 0 (connected)
Routing Descriptor Blocks:
* directly connected, via Null0
Route metric is 0, traffic share count is 1
Route 172.16.1.254/32 is neither installed in RIB, nor being learnt in BGP table…
However, when we enable ‘debug bgp ipv4 u u’ on R1, we see:
BGP: 10.0.12.2 Advertised Nexthop 192.0.2.1: Non-local or Nexthop and peer Not on same interface
BGP(0): 10.0.12.2 rcv UPDATE about 172.16.1.254/32 -- DENIED due to: non-connected NEXTHOP;
BGP(0): no valid path for 172.16.1.254/32
- EXPLANATION:
The BGP session between R1 and R2 is eBGP, therefore uses directly connected interfaces for peerings. By default IOS checks that the next-hop of a received update via eBGP belongs to a subnet directly connected to the local router. Otherwise next-hop attributes for those sessions would be useless. In our case the route to 192.0.2.1 considered not directly connected because the next hop(which is Null0 interface) is not on the same subnet(10.0.12.0/24) as both peer addresses. In turn it fails default verification requirements.
To overcome this requirement we can use commonly used approach by allowing multiple hops between eBGP neighbors:
R1(config-router-af)#nei 10.0.12.2 ebgp-multihop 2
Short afterwards we see:
BGP(0): Revise route installing 1 of 1 routes for 172.16.1.254/32 -> 192.0.2.1(main) to main IP table
R1(config-router)#do sh bgp ipv4 u
Network Next Hop Metric LocPrf Weight Path
*> 172.16.1.0/24 10.0.12.2 0 0 65023 I
*> 172.16.1.254/32 192.0.2.1 0 0 65023 ?
Fine tuning:
You might have noticed the command 'no ip unreachables' under interface Null0:
!
interface Null0
no ip unreachables
!
By default Cico IOS, as a good Internet citizen, will send an ICMP 3/1 message once it receives an unroutable packet. Which is 'should' and good habit in a friendly environment. However, since we are defendig against a DDOS attack, sending too much unreachables may overload a router itself. Apart from powerful CoPP, there is a built-in rate-limiter for ICMP type 3 messages with default of 1 message per 500 ms. It would be also a good idea to disable them for the Null0 interface, since it is mainly being used for intentional drops. Here is a good discussion about this command.
UPD:
Technically speaking, we have achieved the desired outcome and adapted BGP RTBH for VPNv4 prefixes, but there is another technique and RFC worth to mention here: RFC 3704 - Ingress Filtering for Multihomed Networks, which can complement Source-based RTBH.If enabled on autonomus system edge interfaces, it will check every incoming packet to verify whether it has corresponding route enry in the router FIB, and if not drop it. This effectively prevents spoofing attacks.
The command used to make it works is:
R1(config-if)# ip verify unicast source reachable-via any
In our setup we could enable it on R3's interface facing BAUD and on R1's interface facing BGP core, since DDOS supposed to come from there.
If you are interested to learn more about uRPF, please follow this link.
FINAL CONFIGS:
| hostname R2 | hostname R3 |
|---|---|
| ! ip vrf PROXY rd 65123:2 export map RTBH ! ip vrf TYCHO rd 65001:1 route-target export 65001:1 route-target import 65001:1 route-target import 65123:123 ! interface Null0 no ip unreachables ! router bgp 65023 no bgp default ipv4-unicast bgp log-neighbor-changes neighbor 1.1.1.3 remote-as 65023 neighbor 1.1.1.3 update-source Loopback0 ! address-family vpnv4 neighbor 1.1.1.3 activate neighbor 1.1.1.3 send-community extended neighbor 1.1.1.3 route-reflector-client neighbor 1.1.1.3 route-map NH_R3 out exit-address-family ! address-family ipv4 vrf TYCHO neighbor 10.0.12.1 remote-as 65001 neighbor 10.0.12.1 activate neighbor 10.0.12.1 route-map NH_R1 out no synchronization exit-address-family ! address-family ipv4 vrf PROXY redistribute static route-map TRIGGER no synchronization exit-address-family ! ip route vrf PROXY 172.16.1.254 255.255.255.255 192.0.2.1 tag 666 ip route vrf PROXY 192.0.2.1 255.255.255.255 Null0 ! ip community-list 66 permit 666 ! ip prefix-list RTBH_BAUD seq 5 permit 172.16.1.254/32 ! ip prefix-list RTBH_TYCHO seq 5 permit 172.16.1.254/32 ! route-map RTBH permit 10 match ip address prefix-list RTBH_BAUD set extcommunity rt 65123:123 ! route-map RTBH permit 20 match ip address prefix-list RTBH_TYCHO set extcommunity rt 65001:1 ! route-map NH_R3 permit 10 match community 66 set ip next-hop 192.0.2.1 ! route-map NH_R3 permit 20 ! route-map NH_R1 permit 10 match community 66 set ip next-hop 192.0.2.1 ! route-map NH_R1 permit 20 ! route-map TRIGGER permit 10 match tag 666 set local-preference 200 set community 666 set ip next-hop 192.0.2.1 ---> turned out to be useless for VPNv4, can be omitted ! | ! ip vrf BAUD rd 65123:123 route-target export 65123:123 route-target import 65123:123 ! interface Null0 no ip unreachable ! router bgp 65023 no bgp default ipv4-unicast bgp log-neighbor-changes neighbor 1.1.1.2 remote-as 65023 neighbor 1.1.1.2 update-source Loopback0 ! address-family vpnv4 neighbor 1.1.1.2 activate neighbor 1.1.1.2 send-community extended exit-address-family ! address-family ipv4 vrf BAUD no synchronization network 172.16.1.0 mask 255.255.255.0 exit-address-family ! ip route 192.0.2.1 255.255.255.255 Null0 ! |
| hostname R1 |
|---|
! router bgp 65001 |
* You might be interested to learn the Cisco Vision about DDOS attacks.
** Instead of using separate NMS station, you can utilize Netflow to identify offensive flows coming in.
If you have any questions, suggestions or want to add something to the subject being discussed, please post your input in the comments section.
·
