BGP null route in Cisco EVPN VxLAN fabric

[satish.txt1]

Hello,

I have very simple Cisco EVPN VxLAN fabric running on Cisco nexus 93180YC-EX switches. I am using OSFP + iBGP for underlay/overlay with Multicast. I have border-leaf switches connected to single ISP with eBGP peers and I am advertising my public subnets to ISP. 

Now I want to setup BGP Null route to stop DDoS attack. I have BGP community for RTBH from my ISP.  I have configured RTBH on border leaf and try to inject null route using tag 666 but it didn't work and I didn't see it advertise any /32 route in ISP peer. 

After google I came across with this doc [1] and they mentioned you have to configure RTBH on border-leaf + Remote leaf. I am confused why do I need to configure RTBH bgp community on border leaf? 

This is what I have currently setup on border leaf 

route-map RTBH permit 10
  match tag 666
  set community 1299:666

vrf context CUST
  description ** VRF-CUST **
  vni 10555
  ip route 85.xx.xx.128/32 Null0 tag 666
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

router bgp 65001
  vrf CUST
    log-neighbor-changes
    address-family ipv4 unicast
      redistribute static route-map RTBH
      aggregate-address 85.xx.xx.0/23 summary-only
      aggregate-address 85.xx.xx.0/24 summary-only
      aggregate-address 85.xx.xx.0/24 summary-only

I can see routes installed in table 

show ip route 85.xx.xx.128 vrf CUST
IP Route Table for VRF "CUST"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

85.xx.xx.128/32, ubest/mbest: 1/0
    *via Null0, [1/0], 11:17:20, static, tag 666

In BGP table 

2# show ip bgp vrf CUST 85.xx.xx.128
BGP routing table information for VRF CUST, address family IPv4 Unicast
BGP routing table entry for 85.xx.xx.128/32, version 3704
Paths: (3 available, best #1)
Flags: (0x880c0042) (high32 00000000) on xmit-list, is not in urib, suppressed, exported
  vpn: version 4217, (0x00000000100002) on xmit-list

  Advertised path-id 1, VPN AF advertised path-id 1
  Path type: redist, path is valid, is best path, no labeled nexthop
  AS-Path: NONE, path locally originated
    0.0.0.0 (metric 0) from 0.0.0.0 (85.xx.xx.1)
      Origin incomplete, MED 0, localpref 100, weight 32768
      Community: 1299:666
      Extcommunity: RT:65001:10555

  Path type: internal, path is valid, not best reason: Weight, no labeled nexthop
             Imported from 10.254.110.2:32869:[2]:[0]:[0]:[48]:[d89d.6769.8da0]:[32]:[85.xx.xx.128]/272
  AS-Path: NONE, path sourced internal to AS
    10.255.110.0 (metric 3) from 10.254.0.1 (10.254.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10102 10555
      Extcommunity: RT:65001:10102 RT:65001:10555 SOO:10.255.110.0:0 ENCAP:8
          Router MAC:6cb2.aeb1.6657
      Originator: 10.254.110.2 Cluster list: 10.254.0.1

  Path type: internal, path is valid, not best reason: Weight, no labeled nexthop
             Imported from 10.254.110.1:32869:[2]:[0]:[0]:[48]:[d89d.6769.8da0]:[32]:[85.xx.xx.128]/272
  AS-Path: NONE, path sourced internal to AS
    10.255.110.0 (metric 3) from 10.254.0.1 (10.254.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10102 10555
      Extcommunity: RT:65001:10102 RT:65001:10555 SOO:10.255.110.0:0 ENCAP:8
          Router MAC:500f.8041.ed07
      Originator: 10.254.110.1 Cluster list: 10.254.0.1

  VRF advertise information:
  Path-id 1 not advertised to any peer

  VPN AF advertise information:
  Path-id 1 not advertised to any peer

That null route didn't advertised on ISP peers

# show ip bgp vrf CUST neighbors 213.xx.xx.4 advertised-routes

Peer 213.xx.xx.4 routes for address family IPv4 Unicast:
BGP table version is 3704, Local Router ID is 85.xx.xx.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

   Network            Next Hop            Metric     LocPrf     Weight Path
*>a85.xx.xx.0/23     0.0.0.0                           100      32768 i
*>a85.xx.xx.0/24     0.0.0.0                           100      32768 i
*>a85.xx.xx.0/24     0.0.0.0                           100      32768 i

Just wanted to confirm with expert that do I need to configure BGP community on all my remote leaf also to get null route working? 

[1] https://www.cisco.com/c/en/us/td/docs/dcn/nx-os/nexus9000/104x/configuration/vxlan/cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-release-104x/m-configuring-vxlan-bgp-evpn-blackhole.html#concept_cky_r1d_jvb

[satish.txt1]

HI @Harold Ritter 

In your route-map example it works but If I don't add anything in prefix-list RTBH-prefixes then its no suppressing any route and all the route going out to ISP with /32 mask. 

I want to craft such a way that all routes will be suppress all the time but only when I add DDoS target ip in prefix-list that route doesn't get suppress and go out with /32 with blackhole community. 

I was trying to play with route-map and prefix-list but doesn't find a way to make it work. In your example if I remove "ip prefix-list RTBH-prefixes seq 10 permit x.x.x.x/32" list then everything go unsuppress. Hope you understand what I am trying to explain here. (null route IP isn't going to sit there for ever, it will only be in that list when we are under attack after that I have to remove the IP to make it functional) 

[Harold Ritter]

Hi @satish.txt1 ,

With the route-map I provided as an example, you can't have an empty prefix-list. If you don't have any prefix to leak, the prefix-list needs to contain this line:

ip prefix-list RTBH-prefixes seq 9999 deny 0.0.0.0/0 le 32

This line could be there all the time and should be at the very end of the prefix-list.

And if you wanted to leak prefix 192.168.100.1/32, the prefix-list would look like this:

ip prefix-list RTBH-prefixes seq 10 permit 192.168.100.1/32

ip prefix-list RTBH-prefixes seq 9999 deny 0.0.0.0/0 le 32

Regards,

[satish.txt1]

Hi @Harold Ritter 

+1 on route-map (will do same way) 

Today I have removed summary-only flag from BGP and now all my routes advertised as /32 to ISP (later I will play with suppress-map)

I inserted static IP with tag 666 for null route but I am seeing MAC Mobility flag so assuming it won't take that as best path. Why MAC mobility getting set here even none of the MAC moved. 

If MAC mobility will be there that I won't able to NULL route any IP. are there any workaround or remove MAC mobility for that specific I before doing null route?

This is the output of my production null route IP 

border-leaf-1# show ip bgp vrf CUST 85.xx.99.128
BGP routing table information for VRF CUST, address family IPv4 Unicast
BGP routing table entry for 85.xx.99.128/32, version 28416
Paths: (3 available, best #3)
Flags: (0xa8080012) (high32 00000000) on xmit-list, is in urib, is backup urib route, is in HW
  vpn: version 28982, (0x00000000100002) on xmit-list

  Path type: redist, path is valid, not best reason: MAC Mobility Sequence, no labeled nexthop
  AS-Path: NONE, path locally originated
    0.0.0.0 (metric 0) from 0.0.0.0 (85.xx.96.1)
      Origin incomplete, MED 0, localpref 100, weight 32768
      Community: 1299:666
      Extcommunity: RT:65001:10555

  Path type: internal, path is valid, not best reason: Router Id, no labeled nexthop
             Imported from 10.254.110.2:32869:[2]:[0]:[0]:[48]:[d89d.6769.8da0]:[32]:[85.xx.99.128]/272
  AS-Path: NONE, path sourced internal to AS
    10.255.110.0 (metric 3) from 10.254.0.1 (10.254.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10102 10555
      Extcommunity: RT:65001:10102 RT:65001:10555 SOO:10.255.110.0:0 ENCAP:8
          MAC Mobility Sequence:00:1 Router MAC:6cb2.aeb1.6657
      Originator: 10.254.110.2 Cluster list: 10.254.0.1

  Advertised path-id 1, VPN AF advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop, in rib
             Imported from 10.254.110.1:32869:[2]:[0]:[0]:[48]:[d89d.6769.8da0]:[32]:[85.xx.99.128]/272
  AS-Path: NONE, path sourced internal to AS
    10.255.110.0 (metric 3) from 10.254.0.1 (10.254.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10102 10555
      Extcommunity: RT:65001:10102 RT:65001:10555 SOO:10.255.110.0:0 ENCAP:8
          MAC Mobility Sequence:00:1 Router MAC:500f.8041.ed07
      Originator: 10.254.110.1 Cluster list: 10.254.0.1

  VRF advertise information:
  Path-id 1 advertised to peers:
    213.xx.99.6

  VPN AF advertise information:
  Path-id 1 not advertised to any peer

[Harold Ritter]

Hi @satish.txt1 ,

The "Configuring VXLAN BGP-EVPN Null Route" document you referred to in your initial post mentions setting the weight to 65535. I wonder if that could be used to override the MAC mobility sequence number in the BGP best past selection process.

Could you try this in your lab setup?

Regards, 

[satish.txt1]

Hi @Harold Ritter 

I have thought about that too and I will try to add Weight in LAB but I believe there won't be any impact routing. If you notice in following output weight is anyway weight 32768 on first path. Weight is locally significant value and setting up it on remote TOR (VTEP) switch doesn't influence border-leaf right? Or Am i missing something?  

Path type: redist, path is valid, not best reason: MAC Mobility Sequence, no labeled nexthop
  AS-Path: NONE, path locally originated
    0.0.0.0 (metric 0) from 0.0.0.0 (85.xx.96.1)
      Origin incomplete, MED 0, localpref 100, weight 32768
      Community: 1299:666
      Extcommunity: RT:65001:10555

[Harold Ritter]

Hi @satish.txt1 ,

I meant to set the weight to 65535 instead of 32768 on the border router. I am not sure this will work, as it looks like the first thing that is checked is the presence of the MAC mobility sequence extended community, but it is worth trying in your lab.

Regards,

[satish.txt1]

I have set weight to 65535 but no effect at all as you can see in following output. This is very strange issue that we can't influence route even with BGP attributes. Wish we can strip down MAC Mobility community from that route. 

Paths: (3 available, best #3)
Flags: (0xa8080012) (high32 00000000) on xmit-list, is in urib, is backup urib route, is in HW
  vpn: version 29089, (0x00000000100002) on xmit-list

  Path type: redist, path is valid, not best reason: MAC Mobility Sequence, no labeled nexthop
  AS-Path: NONE, path locally originated
    0.0.0.0 (metric 0) from 0.0.0.0 (85.xx.96.1)
      Origin incomplete, MED 0, localpref 100, weight 65535
      Community: 1299:666
      Extcommunity: RT:65001:10555

[Harold Ritter]

Hi @satish.txt1 ,

Unfortunately, I did not find a way for the null route to override the route coming from the remote vtep on the border router. The other option would be to originate the null route on the upstream router, if you control it. 

Is the upstream router under your control?

Regards,

[satish.txt1]

Hi @Harold Ritter 

No, The upstream routers are under ISP control. 

In my design I think I could have used Edge router connect to ISP instead my border-leaf connected to ISP that way I can control things better way. 

What is the best practice in EVPN to connect to ISP?

Still thinking that there has to be a way to handle this because I am not the only one who has this kind of design.

[Harold Ritter]

Hi @satish.txt1 ,

I have unfortunately not found a way to make the local route on the border leaf preferred. 

It is still possible to workaround this issue even if you don't control the upstream router. The solution I would use would be to have a back to back link between the Internet vrf and the global and do the bgp peering with the ISP router using the global. This solution would be just like using a separate Internet peering device, but does not require an additional device.

This would allow to inject the null route in the global and to have it preferred.

Regards,

[satish.txt1]

Hi @Harold Ritter 

You are suggesting to create a p-2-p link from default VRF to ISP core router and do null route from default VRF?  How does route exchange will work in that case? 

I was thinking other possible solution would be adding one more edge router between border leaf and ISP routers. 

[border-leaf]-------[Edge Router]----------[ISP Router]

In above design I can inject null router from Edge Router right?  But tell me how do i peer border-leaf and edge-router? I can use private AS right? 

[Harold Ritter]

Hi @satish.txt1 ,

> You are suggesting to create a p-2-p link from default VRF to ISP core router and do null route from default VRF?

Yes, that is exactly what I am suggesting.

> How does route exchange will work in that case?

You would run BGP just like if you had an external edge device.

> I was thinking other possible solution would be adding one more edge router between border leaf and ISP routers.

This is the same solution, but adding an additional device. Both are possible. You just need to decide what you prefer.

> In above design I can inject null router from Edge Router right? 

Correct.

> But tell me how do i peer border-leaf and edge-router? I can use private AS right?

The border leaf would peer via BGP to the edge router and the edge router would peer to the ISP router via BGP as well. You can definitely use a private AS to peer between the border leaf and the edge router.

Regards,

[satish.txt1]

In my global VRF I have private AS running and in CUST VRF I have public AS configured so not sure how my ISP will let me peer eBGP with private ASN. Anyway I will create LAB for p-2-p solution to create extra link with ISP to send null route and see how it goes. 

I am also planning to go with extra Edge Router solution for one more reason which is I want to setup iBGP peer to goBGP linux program which will tigger null route using shell script. In current setup I am not able to setup iBGP peer with border-leaf because its running in VRF and I can't use remote/local AS number same. I have tried in LAB and its not letting me do that. 

[Harold Ritter]

Hi @satish.txt1 ,

> In my global VRF I have private AS running and in CUST VRF I have public AS configured so not sure how my ISP will let me peer eBGP with private ASN. Anyway I will create LAB for p-2-p solution to create extra link with ISP to send null route and see how it goes.

I mentioned using the global to peer with the ISP router, but you should probably use another VRF.

> I am also planning to go with extra Edge Router solution for one more reason which is I want to setup iBGP peer to goBGP linux program which will tigger null route using shell script.

It sounds like a plan if this makes it easier for you.

Regards,