BGP RTBH setup using exaBGP

In this post I'll describe a basic setup using Cisco IOS, IOS XR and exaBGP that will function as a BGP remotely triggered blackhole (RTBH) allowing you to null route any source/destination prefix. The example topology will be as below:

exaBGP by a very nice man called Thomas Mangin is an excellent fit for what we are looking to do :)

Unlike BIRD or Quagga, ExaBGP was not designed to transform a general purpose server into a router, but to allow engineers to control their BGP network easily. Think of it as Software Defined Networking for people with "commodity" routers.

Considerations

Before we start just outline some functions/feature our design must consider:

• Must NULL route in our own ASN and automagically with upstream providers
• Must only allow /32 prefixes to be back holed
• Must do some basic sanity checking at each step
• Should allow us to blackhole both source and destination prefixes

Sample configs

exaBGP

Simply installed following the instructions from the exaBGP site then edit the exaBGP.conf file to two sessions configured for our example(IOS and IOS_XR routers from topology above). Note the static section as this is where we configure prefixes to announce with exaBGP.

#################################
#    Example exaBGP config      #
#          m00nie.com 		    #
#################################
#
# Blackhole Communities:
# m00nie co  - 65001:666
# Level4     - 65002:666
# Xayo       - 65003:666
#
# Well-Known communtites
# no-export    -  65535:65281  # do not advertise to any eBGP peers
# no-advertise -  65535:65282  # do not advertise to any BGP peer
# local-as     -  65535:65283  # do not advertise this route to peers outside the local as
#################################

group Core_v4 {
  hold-time 180;
  local-as 65001;
  peer-as 65001;
  router-id 192.168.1.2;
  graceful-restart 1200;

  static {
  }

  neighbor 192.168.1.254 {
    local-address 192.168.1.2;
    description IOS;
  }

  neighbor 192.168.1.253 {
    local-address 192.168.1.2;
    description IOS_XR;
    md5 S00pers3kretz;
  }
}

So that's the sessions configured (obviously needs associated config on the routers) now the interesting bit lets add a single /32 to the exaBGP.conf file under the "static section":

static {
          # Example - Announcing a /32 with one community 
          route 10.10.10.1/32 next-hop 192.0.2.1 community 65001:666;
          #Example - Two communities 
          route 10.10.10.2/32 next-hop 192.0.2.1 community [65001:666 65001:400];
}

There are all the normal BGP attributes available under exaBGP but as you will see below in this example we are only really going to use communities to do what we want :)

Router Config

IOS

The config is pretty much the same for both IOS and XR its just the syntax thats different really. Both will accomplish the same thing. First off we need to add a null route for us to point to as a next-hop. For IPv4 there is a convention to use this "documentation" address but for IPv6 we have a nice 'official' address range to use :)

ip route 192.0.2.1 255.255.255.255 null0
ipv6 route 0100::/64 null0

Now we have somewhere we can set the next hop to. As part of the sanity checks we want to configure a prefix-list containing important IPs that we never want to be null routed or would be painful for us. Imagine null routing the loopbacks of your core routers? So lets configure one that contains the loopbacks of our cores, the exaBGP host IP and the IPs of our transit provider sessions:

ip prefix-list core-ipv4 seq 10 permit 192.168.1.254/32
ip prefix-list core-ipv4 seq 15 permit 192.168.1.253/32
ip prefix-list core-ipv4 seq 20 permit 192.168.1.2/32
ip prefix-list core-ipv4 seq 25 permit 172.16.16.2/32
ip prefix-list core-ipv4 seq 30 permit 172.16.16.4/32

Now we need to configure the blackhole community on the router to match against:

ip community-list standard blackhole permit 65001:666

Next is a prefix list that will only match /32 prefixes:

ip prefix-list only32 seq 5 permit 0.0.0.0/0 ge 32

Now we can create an import route-map to check what we want before accepting the announcement from exaBGP:

route-map exaBGP-in deny 10
 description Dont blackhole our core loopbacks etc
 match ip address prefix-list core-ipv4
!
route-map exaBGP-in permit 20
 description Only accept 32s with comm tagged
 match ip address prefix-list only32
 match community blackhole
 set community 65001:400
 set ip next-hop 192.0.2.1

This route-map will make sure the prefix is not in the core-ipv4 list then only accepts if it is of /32 in length and has the blackhole community tagged on it. At this stage we can NULL route this /32 prefix within our own ASN! :D The next route-map should be added to your transit route-map. Its quite simple and just matches against the blackhole community then sets the transit provider specific RTBH community.

route-map transit-out premit XX
 description Blackhole upstream
 match community blackhole
 set community 65002:666

Now we will blackhole the prefix and ask our transit provider to do the same so we are never even sent the traffic!

IOS XR

This is very much the same as the IOS config so not much description here just the config snippets:

Add the routes:

router static
 address-family ipv4 unicast
  192.0.2.1/32 Null0
 !
 address-family ipv6 unicast
  0100::/64 Null0

Add the community:

community-set blackhole
  65001:666
  end-set
!
community-set Xayo-blackhole
  65003:666
  end-set

Prefix sets:

prefix-set only32
  0.0.0.0/0 ge 32
  end-set
!
prefix-set core-ipv4
 192.168.1.254,
 192.168.1.253,
 192.168.1.2,
 172.16.16.2,
 172.16.16.4
  end-set

The import policy from exaBGP:

route-policy exaBGP-in
  if destination in core-ipv4 then
    drop
  endif
  if destination in only32 and community matches-any blackhole then
    set next-hop 192.0.2.1
  else
    drop
  endif
  set community comm-exaBGP additive
  done
  end-policy

Then we just need to add something similar to the following to our transit provider export rpl

route-policy export-Xayo
  if community matches-any blackhole then
    set community Xayo-blackhole additive
    set community no-export additive
  endif

What does this do?

So following the example topology from the top of the post. We can announce a prefix with exaBGP to all our core routers this prefix will only be accepted if it is tagged with the blackhole community, is a /32 and is NOT an IP from our "protected" list. This affords us some comfort that we cant null route our transit sessions or a core router loopback making BGP b0rk. The core routers will null route the prefix locally while adding appropriate upstream blackhole communities so we never even receive the traffic.

If you add uRPF into the config this will mean we can also blackhole traffic based on the source address! So if the DOS isnt particularly distributed we can mitigate it by blackholing the source IPs and keeping everyone a little happier.

Future fun

So now we can do all this stuff imagine doing it in a more automatic fashion :) Analysing netflow output or output from an NFSEN plugin along with integration into exaBGP to add prefixes after a quick manual review or if your really brave automatically. Then look into flowspec and exaBGP :P

m00nie