BGP FlowSpec on white box switch

BGP FlowSpec is a method of distributing access control lists (ACLs) using the BGP protocol. Distributed denial of service (DDoS) mitigation is an important use case for the technology, allowing a targeted network to push filters to their upstream provider to selectively remove the attack traffic.

Unfortunately, FlowSpec is currently only available on high end routing devices and so experimenting with the technology is expensive. Looking for an alternative, Cumulus Linux is an open Linux platform that allows users to install Linux packages and develop their own software.

This article describes a proof of concept implementation of basic FlowSpec functionality using ExaBGP installed on a free Cumulus VX virtual machine.  The same solution can be run on inexpensive commodity white box hardware to deliver terabit traffic filtering in a production network.

First, install latest version of ExaBGP on the Cumulus Linux switch:

curl -L https://github.com/Exa-Networks/exabgp/archive/4.0.0.tar.gz | tar zx

Now define the handler, acl.py, that will convert BGP FlowSpec updates into standard Linux netfilter/iptables entries used by Cumulus Linux to specify hardware ACLs (see Netfilter - ACLs):

#!/usr/bin/python

import json
import re
from os import listdir,remove
from os.path import isfile
from subprocess import Popen,STDOUT,PIPE
from sys import stdin, stdout, stderr

id = 0
acls = {}

dir = '/etc/cumulus/acl/policy.d/'
priority = '60'
prefix = 'flowspec'
bld = '.bld'
suffix = '.rules'

def commit():
  Popen(["cl-acltool","-i"],stderr=STDOUT,stdout=PIPE).communicate()[0]

def aclfile(name):
  global dir,priority,prefix,suffix
  return dir+priority+prefix+name+suffix

def handleSession(state):
  if "down" == state:
    for key,rec in acls.items():
      fn = aclfile(str(rec['id']))
      if isfile(fn):
        remove(fn)
      del acls[key]
    commit()

def buildACL(flow,action):
  acl = "[iptables]\n-A FORWARD --in-interface swp+"
  if flow.get('protocol'):
    acl = acl + " -p " + re.sub('[!<>=]','',flow['protocol'][0])
  if flow.get('source-ipv4'):
    acl = acl + " -s " + flow['source-ipv4'][0]
  if flow.get('destination-ipv4'):
    acl = acl + " -d " + flow['destination-ipv4'][0]
  if flow.get('source-port'):
    acl = acl + " --sport " + re.sub('[!<>=]','',flow['source-port'][0])
  if flow.get('destination-port'):
    acl = acl + " --dport " + re.sub('[!<>=]','',flow['destination-port'][0])
  acl = acl + " -j DROP\n"
  return acl

def handleFlow(add,flow,action):
  global id
  key = flow['string']
  if add:
    acl = buildACL(flow,action)
    id = id + 1
    acls[key] = {"acl":acl,"id":id}
    fn = aclfile(str(id))
    f = open(fn,'w')
    f.write(acl)
    f.close()
    commit()
  elif key in acls:
    rec = acls[key]
    fn = aclfile(str(rec['id']))
    if isfile(fn):
      remove(fn)
    del acls[key]
    commit()

while True:
  try:
     line = stdin.readline().strip()
     msg = json.loads(line)
     type = msg["type"]
     if "state" == type:
       state = msg["neighbor"]["state"]
       handleSession(state)
     elif "update" == type:
       update = msg["neighbor"]["message"]["update"] 
       if update.get('announce'):
         flow = update["announce"]["ipv4 flow"]["no-nexthop"][0]
         community = update["attribute"]["extended-community"][0]
         handleFlow(True,flow,community)
       elif update.get('withdraw'):
         flow = update["withdraw"]["ipv4 flow"][0]
         handleFlow(False,flow,None)
     else:
       pass
  except IOError:
     pass

Note: This script is a simple demonstration of the concept that has significant limitations: there is no error handling, the only action is to drop traffic, and FlowSpec comparison operators are ignored. The script is is based on the article RESTful control of Cumulus Linux ACLs.

Next, create the exabgp.conf file:

process acl {
   run ./acl.py;
   encoder json;
}

template {
  neighbor controller {
    family {
      ipv4 flow;
    }
    api speaking {
      processes [ acl ];
      neighbor-changes;
      receive {
         parsed;
         update;
      }
    }
  }
}

neighbor 10.0.0.70 {
  inherit controller;
  router-id 10.0.0.140;
  local-as 65140;
  peer-as 65070;
  local-address 0.0.0.0;
  connect 1179;
}

Finally, run ExaBGP:

sudo env exabgp.daemon.user=root exabgp-4.0.0/sbin/exabgp exabgp.conf

This configuration instructs ExaBGP to connect to the controller, 10.0.0.162, and prepare to receive BGP FlowSpec messages. When a BGP message is received, ExaBGP decodes the message and passes it on in the form of a JSON encoded string to the acl.py program. For example, the following FlowSpec message was sent to block an NTP reflection DDoS attack (traffic from UDP port 123) targeting host 192.168.0.1:

{
"exabgp": "4.0.0",
"time": 1498868955.31,
"host": "tor-router",
"pid": 3854,
"ppid": 3849,
"counter": 6,
"type": "update",
"neighbor": {
"address": {
"local": "0.0.0.0",
"peer": "10.0.0.70"
  },
"asn": {
"local": "65140",
"peer": "65070"
  },
"direction": "receive",
"message": {
"update": {
"attribute": {
"origin": "igp",
"as-path": [
      65140
     ],
"confederation-path": [],
"local-preference": 100,
"extended-community": [
      9225060886715040000
     ]
    },
"announce": {
"ipv4 flow": {
"no-nexthop": [
       {
"destination-ipv4": [
"192.168.0.1/32"
        ],
"protocol": [
"=udp"
        ],
"source-port": [
"=123"
        ],
"string": "flow destination-ipv4 192.168.0.1/32 protocol =udp source-port =123"
       }
      ]
     }
    }
   }
  }
 }
}

The acl.py script wrote the file /etc/cumulus/acl/policy.d/60flowspec1.rules with an iptables representation of the FlowSpec message:

[iptables]
-A FORWARD --in-interface swp+ -p udp -d 192.168.0.1/32 --sport 123 -j DROP

The acl.py script also invoked the cl-acltool command to install the new rule, which can be verified using iptables:

sudo iptables --list FORWARD
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DROP       all  --  240.0.0.0/5          anywhere            
DROP       all  --  loopback/8           anywhere            
DROP       all  --  base-address.mcast.net/4  anywhere            
DROP       all  --  255.255.255.255      anywhere            
DROP       udp  --  anywhere             192.168.0.1          udp spt:ntp

The attack traffic is now dropped by the switch. In a hardware switch, this entry would be pushed by Cumulus Linux into the hardware, filtering the traffic at line rate to provide Terabit traffic filtering.