An early version of Flood Protect won the 2014 SDN Idol competition in a joint demonstration with Brocade Networks.

#!/usr/bin/env python

import sys, re, fileinput, requests, json

url = sys.argv[1]
top = {'links':{}}

def dequote(s):
  if (s[0] == s[-1]) and s.startswith(("'", '"')):
    return s[1:-1]
  return s

l = 1
for line in fileinput.input(sys.argv[2:]):
  link = re.search('([\S]+):(\S+)\s*(--|->)\s*(\S+):([^\s;,]+)',line)
  if link:
    s1 = dequote(link.group(1))
    p1 = dequote(link.group(2))
    s2 = dequote(link.group(4))
    p2 = dequote(link.group(5))
    linkname = 'L%d' % (l)
    l += 1
    top['links'][linkname] = {'node1':s1,'port1':p1,'node2':s2,'port2':p2}

requests.put(url,data=json.dumps(top),headers={'content-type':'application/json'})

./ptm.py http://fabricview:8008/script/fabric-view.js/topology/json topology.dot

1. It now runs as a daemon.
2. Exception generated by cl-acltool are caught and handled
3. Rules are compiled asynchronously, reducing response time of REST calls
4. Updates are batched, supporting hundreds of operations per second

server = HTTPServer(('',8080), ACLRequestHandler)

server = HTTPServer(('127.0.0.1',8080), ACLRequestHandler)

root# mv acl_server /etc/init.d/
root# chmod 755 /etc/init.d/acl_server
root# service acl_server start

root#  echo 'deb http://ftp.us.debian.org/debian  wheezy main contrib' \
>>/etc/apt/sources.list.d/deb.list
root# apt-get update
root# apt-get install apache2

root# a2enmod proxy proxy_http

<ifmodule mod_proxy.c>
  ProxyRequests off
  ProxyVia off
  ProxyPass        /acl/ http://127.0.0.1:8080/acl/
  ProxyPassReverse /acl/ http://127.0.0.1:8080/acl/
</ifmodule>

root# service apache2 restart

Create an ACL

curl -H "Content-Type:application/json" -X PUT --data '["[iptables]","-A FORWARD --in-interface swp+ -d 10.10.100.10 -p udp --sport 53 -j DROP"]' http://10.0.0.233/acl/ddos1

[iptables]
-A FORWARD --in-interface swp+ -d 10.10.100.10 -p udp --sport 53 -j DROP

Retrieved an ACL

curl http://10.0.0.233/acl/ddos1

{"result": ["[iptables]", "-A FORWARD --in-interface swp+ -d 10.10.100.10 -p udp --sport 53 -j DROP"]}

List ACLs

curl http://10.0.0.233/acl/

{"result": ["ddos1"]}

Delete an ACL

curl -X DELETE http://10.0.0.233/acl/ddos1

Delete all ACLs

curl -X DELETE http://10.0.0.233/acl/

{"lastError": {"returncode": 255, "lines": [...]}, "result":[...]}

sudo ./leafandspine.py --collector=10.0.0.162 --controller=10.0.0.162 --topofile
=/var/www/html/topology.json

1. While physical networks might have link speeds ranging from 1Gbit/s to 100Gbit/s, the emulation scales link speeds down to 10Mbit/s so that they can be emulated in software.
2. The sFlow sampling rate is scaled proportionally - see Large flow detection
3. A pair of OpenFlow 1.3 tables is used to emulate normal ECMP forwarding and hybrid OpenFlow overrides
4. Linux Traffic Control (tc) commands are used to emulate hardware priority queueing based on Differentiated Services Code Point (DSCP) class, mapping DSCP class 8 to a lower priority or "less than best effort" queue.
5. The script posts the topology as a JSON file under the default Apache document root so that it can be retrieved remotely by an SDN controller
6. In this example the sFlow-RT controller is running on host 10.0.0.162 - change the address to match your setup.

#!/bin/bash
SCALE=1
SCALE=${2-$SCALE}
ping $1 | awk -v SCALE=$SCALE 'BEGIN {FS="[= ]"; } NF==11 { n = $10 * SCALE; bar
 = ""; while(n >= 1) { bar = bar "*"; n--} print bar "" $10 " ms" }

./pingtest 10.0.1.1 10

iperf h2 h3

include('extras/leafandspine-hybrid.js');

// get topology from Mininet
setTopology(JSON.parse(http('http://10.0.0.30/topology.json')));

// Define large flow as greater than 1Mbits/sec for 1 second or longer
var bytes_per_second = 1000000/8, duration_seconds = 1;

// define TCP flow cache
setFlow('tcp',
 {keys:'ipsource,ipdestination,tcpsourceport,tcpdestinationport', filter:'direct
ion=ingress',
  value:'bytes', t:duration_seconds}
);

// set threshold identifying Elephant flows
setThreshold('elephant', {metric:'tcp', value:bytes_per_second, byFlow:true, tim
eout:4});

// set OpenFlow marking rule when Elephant is detected
var idx = 0;
setEventHandler(function(evt) {
 if(topologyInterfaceToLink(evt.agent,evt.dataSource)) return;
 var port = ofInterfaceToPort(evt.agent,evt.dataSource);
 if(port) {
  var dpid = port.dpid;
  var id = "mark" + idx++;
  var k = evt.flowKey.split(',');
  var rule= {
    match:{in_port: port.port, dl_type:2048, ip_proto:6, nw_src:k[0], nw_dst:k[1], tcp_src:k[2], tcp_dst:k[3]},
    actions:["set_ip_dscp=8","output=normal"], priority:1000, idleTimeout:5
  };
  logInfo(JSON.stringify(rule,null,1));
  setOfRule(dpid,id,rule);
 }
},['elephant']);

1. The included leafandspine-hybrid.js script emulates hybrid OpenFlow by rewriting the NORMAL OpenFlow action to jump to the table that contains the ECMP forwarding rules. 
2. The script assumes that Mininet emulation is running on host 10.0.0.30. Modify the address in the setTopology() function for your setup.
3. The setFlow() function instructs the controller build a flow cache to track TCP connections
4. The setThreshold() function defines Elephant flows as TCP connections that exceed 10% of the link's bandwidth (in this case 1Mbit/second) for 1 second or more.
5. The setEventHandler() function processes each Elephant flow notification and applies an OpenFlow marking rules to the ingress port on the edge switch where the traffic enters the fabric.
6. The OpenFlow rules have an idleTimeout of 5 seconds, ensuring that they are automatically deleted by the switch when the flow ends.

-Dopenflow.controller.start=yes 
-Dopenflow.controller.flushRules=no
-Dscript.file=mark.js

./start.sh

• Agent IP address ⟷ OpenFlow switch ID
• SNMP ifIndex ⟷ OpenFlow port ID

Credit: Accelerating Open vSwitch to “Ludicrous Speed”

/* Open vSwitch data path statistics */
/* see datapath/datapath.h */
/* opaque = counter_data; enterprise = 0; format = 2207 */ 
struct ovs_dp_stats { 
  unsigned int hits;                                                
  unsigned int misses; 
  unsigned int lost;
  unsigned int mask_hits;
  unsigned int flows;
  unsigned int masks;
}

• sFlow / OpenFlow integration
• Link aggregation

The Open vSwitch project first added sFlow support five years ago and these recent enhancements build on the detailed visibility into network traffic provided by the core Open vSwitch sFlow implementation and the complementary visibility into hosts, hypervisors, virtual machines and containers provided by the Host sFlow project.

Visibility and the software defined data center

Broad support for the sFlow standard across the cloud data center stack provides simple, efficient, low cost, scaleable, and comprehensive visibility. The standard metrics can be consumed by a broad range of open source and commercial tools, including: sflowtool, sFlow-Trend, sFlow-RT, Ganglia, Graphite, InfluxDB and Grafana.

#!/usr/bin/env python

import requests
import json
import signal
from jsonrpclib import Server

switch_list = ['switch1.example.com','switch2.example.com']
username = "admin"
password = "password"

sflow_collector = "192.168.56.1"
sflow_port = "6343"
sflow_polling = "20"
sflow_sampling = "10000"

for switch_name in switch_list:
  switch = Server("https://%s:%s@%s/command-api" %
                (username, password, switch_name))
  response = switch.runCmds(1,
   ["enable",
"configure",
"sflow source %s" % switch_ip,
"sflow destination %s %s" % (sflow_collector, sflow_port),
"sflow polling-interval %s" % sflow_polling,
"sflow sample output interface",
"sflow sample dangerous %s" % sflow_sampling,
"sflow run"])

#/usr/bin/python 
'''
Copyright (c) 2015, Arista Networks, Inc. All rights reserved.

Redistribution and use in source and binary forms, with or without 
modification, are permitted provided that the following conditions are met:

 * Redistributions of source code must retain the above copyright notice, 
   this list of conditions and the following disclaimer. 

 * Redistributions in binary form must reproduce the above copyright notice, 
   this list of conditions and the following disclaimer in the documentation 
   and/or other materials provided with the distribution. 

 * Neither the name of Arista Networks nor the names of its contributors 
   may be used to endorse or promote products derived from this software 
   without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL ARISTA NETWORKS BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
'''

# v0.5 - initial version of the script to discover network topology using
# Arista eAPI and generate output in json format recognized by sFlow-RT.

from jsonrpclib import Server 
import json 
from pprint import pprint

# define switch in your topology, eapi transport protocol (http or https),
# eapi username and password
switch_list = ['switch1.example.com','switch2.example.com']
eapi_transport = 'https'
eapi_username = 'admin'
eapi_password = 'password'

debug = False

# internal variables used by the script
allports = {}
allswitches = {}
allneighbors = []
alllinks = {}

# method to populate allswitches and allports - called only from processNeighbor()
def addPort(switchname, switchIP, portname, ifindex):
 id = switchname + '>' + portname
 prt = allports.setdefault(id, { "portname": portname, "linked": False })
 if ifindex is not None:
  prt["ifindex"] = ifindex
 sw = allswitches.setdefault(switchname, { "name": switchname, "agent": switchIP, "ports": {} });
 if switchIP is not None:
  sw["agent"] = switchIP
 sw["ports"][portname] = prt

# method to collect neighbor records - called with each LLDP neighbor 
# entry as they are discovered
def processNeighbor(localname,localip,localport,localifindex,remotename,remoteport):
 addPort(localname, localip, localport,localifindex);
 addPort(remotename, None, remoteport, None);
 allneighbors.append({ "localname": localname, "localport": localport,
"remotename": remotename, "remoteport": remoteport });

# method to remove agents that we did not discover properly, or
# that we did not intend to include in the topology.  (If we
# assigned an agent field to the switch then we assume it should stay.)
def pruneAgents():
 for nm,sw in allswitches.items():
  #if not "agent" in sw:
  if sw['agent'] == '0.0.0.0' or not sw['agent']:
   del allswitches[nm]

# method to test for a new link - called only from findLinks()
def testLink(nbor,linkno):
 swname1 = nbor["localname"]
 swname2 = nbor["remotename"]
 # one of the switches might have been pruned out
 if swname1 not in allswitches or swname2 not in allswitches:
  return False
 sw1 = allswitches[swname1]
 sw2 = allswitches[swname2]
 pname1 = nbor["localport"]
 pname2 = nbor["remoteport"]
 port1 = sw1["ports"][pname1];
 port2 = sw2["ports"][pname2];
 if not port1["linked"] and not port2["linked"]:
  # add new link
  linkid = "link" + str(linkno)
  port1["linked"] = True;
  port2["linked"] = True;
  alllinks[linkid] = {
"node1": nbor["localname"],
"port1": nbor["localport"],
"node2": nbor["remotename"],
"port2": nbor["remoteport"]
   }
  return True
 return False

# method to find unique links - call at the end once all the LLDP records have
# been processed from all the switches
def findLinks():
 linkcount = 0
 for nbor in allneighbors:
  if testLink(nbor, linkcount+1):
   linkcount += 1

# method to dump topology in json format recognized by sFlow-RT
def dumpTopology():
 topology = { "nodes": allswitches, "links": alllinks }
 print(json.dumps(topology, indent=4))

# method to get LLDP neighbors of each switch - calls processNeighbor() for each LLDP neighbor found
def getLldpNeighbors(switch_name):
 try:
  switch = Server('%s://%s:%s@%s/command-api' % (eapi_transport, eapi_username, eapi_password, switch_name))

  # Get LLDP neighbors
  commands = ["enable", "show lldp neighbors"]
  response = switch.runCmds(1, commands, 'json')
  neighbors = response[1]['lldpNeighbors']

  # Get local hostname
  commands = ["enable", "show hostname"]
  response = switch.runCmds(1, commands, 'json')
  hostname = response[1]['hostname']

  # Get SNMP ifIndexes
  commands = ["enable", "show snmp mib ifmib ifindex"]
  response = switch.runCmds(1, commands, 'json')
  interfaceIndexes = response[1]['ifIndex']

  # Get sFlow agent source address
  commands = ["enable", "show sflow"]
  response = switch.runCmds(1, commands, 'json')
  sflowAddress = response[1]['ipv4Sources'][0]['ipv4Address']

  # Create 2D array lldp_neighbors where each line has following entries 
  # , , , 
  lldp_neighbors = []
  for neighbor in neighbors:
   lldp_neighbors.append([neighbor['neighborDevice'].split('.')[0], 
        neighbor['port'], neighbor['neighborPort'], interfaceIndexes[neighbor['port']]])

  if (debug): 
   pprint(lldp_neighbors)


  # collect switches, ports and neighbor-relationships
  for row in lldp_neighbors:
   processNeighbor(hostname, 
    sflowAddress,
    row[1], # localport
    row[3], # localifindex
    row[0], # remotename
    row[2]) # remoteport

  # Print list of LLDP neighbors in human friendly format:
  #  neighbor, , connected to local  with remote 
  if debug:
   print "Switch %s has following %d neighbors:" % (hostname[1], len(neighbors))
   for i, neighbor in enumerate(lldp_neighbors):
    print "#%d neighbor, %s, connected to local %s with remote %s" % (i+1, neighbor[0], neighbor[1], neighbor[2])

 except:
  print 'Exception while connecting to %s' % switch_name
  return []


for switch in switch_list:
 getLldpNeighbors(switch)

pruneAgents()
findLinks()
dumpTopology()

{
"nodes": {
"leaf332": {
"name": "leaf332", 
"agent": "10.10.130.142", 
"ports": {
"Management1": {
"portname": "Management1", 
"ifindex": 999001, 
"linked": false
                }, 
"Ethernet50/1": {
"portname": "Ethernet50/1", 
"ifindex": 50001, 
"linked": true
                }, 
"Ethernet36": {
"portname": "Ethernet36", 
"ifindex": 36, 
"linked": true
                }, 
"Ethernet51/1": {
"portname": "Ethernet51/1", 
"ifindex": 51001, 
"linked": true
                }, 
"Ethernet52/1": {
"portname": "Ethernet52/1", 
"ifindex": 52001, 
"linked": true
                }, 
"Ethernet49/1": {
"portname": "Ethernet49/1", 
"ifindex": 49001, 
"linked": true
                }, 
"Ethernet12": {
"portname": "Ethernet12", 
"ifindex": 12, 
"linked": false
                }, 
"Ethernet35": {
"portname": "Ethernet35", 
"ifindex": 35, 
"linked": true
                }
            }
        }, 
"leaf259": {
"name": "leaf259", 
"agent": "10.10.129.220", 
"ports": {
"Management1": {
"portname": "Management1", 
"ifindex": 999001, 
"linked": false
                }, 
"Ethernet5/1": {
"portname": "Ethernet5/1", 
"ifindex": 5001, 
"linked": true
                }, 
"Ethernet29": {
"portname": "Ethernet29", 
"ifindex": 29, 
"linked": true
                }, 
"Ethernet32": {
"portname": "Ethernet32", 
"ifindex": 32, 
"linked": true
                }, 
"Ethernet6/1": {
"portname": "Ethernet6/1", 
"ifindex": 6001, 
"linked": true
                }, 
"Ethernet31": {
"portname": "Ethernet31", 
"ifindex": 31, 
"linked": true
                }, 
"Ethernet30": {
"portname": "Ethernet30", 
"ifindex": 30, 
"linked": true
                }, 
"Ethernet15/1": {
"portname": "Ethernet15/1", 
"ifindex": 15001, 
"linked": false
                }
            }
        }, 
"leaf331": {
"name": "leaf331", 
"agent": "10.10.130.141", 
"ports": {
"Management1": {
"portname": "Management1", 
"ifindex": 999001, 
"linked": false
                }, 
"Ethernet50/1": {
"portname": "Ethernet50/1", 
"ifindex": 50001, 
"linked": true
                }, 
"Ethernet36": {
"portname": "Ethernet36", 
"ifindex": 36, 
"linked": true
                }, 
"Ethernet1": {
"portname": "Ethernet1", 
"ifindex": 1, 
"linked": false
                }, 
"Ethernet51/1": {
"portname": "Ethernet51/1", 
"ifindex": 51001, 
"linked": true
                }, 
"Ethernet52/1": {
"portname": "Ethernet52/1", 
"ifindex": 52001, 
"linked": true
                }, 
"Ethernet49/1": {
"portname": "Ethernet49/1", 
"ifindex": 49001, 
"linked": true
                }, 
"Ethernet11": {
"portname": "Ethernet11", 
"ifindex": 11, 
"linked": false
                }, 
"Ethernet35": {
"portname": "Ethernet35", 
"ifindex": 35, 
"linked": true
                }
            }
        }, 
"leaf260": {
"name": "leaf260", 
"agent": "10.10.129.221", 
"ports": {
"Management1": {
"portname": "Management1", 
"ifindex": 999001, 
"linked": false
                }, 
"Ethernet11/1": {
"portname": "Ethernet11/1", 
"ifindex": 11001, 
"linked": false
                }, 
"Ethernet5/1": {
"portname": "Ethernet5/1", 
"ifindex": 5001, 
"linked": true
                }, 
"Ethernet29": {
"portname": "Ethernet29", 
"ifindex": 29, 
"linked": true
                }, 
"Ethernet32": {
"portname": "Ethernet32", 
"ifindex": 32, 
"linked": true
                }, 
"Ethernet6/1": {
"portname": "Ethernet6/1", 
"ifindex": 6001, 
"linked": true
                }, 
"Ethernet31": {
"portname": "Ethernet31", 
"ifindex": 31, 
"linked": true
                }, 
"Ethernet30": {
"portname": "Ethernet30", 
"ifindex": 30, 
"linked": true
                }
            }
        }, 
"core210": {
"name": "core210", 
"agent": "10.10.129.185", 
"ports": {
"Ethernet3/3/1": {
"portname": "Ethernet3/3/1", 
"ifindex": 3037, 
"linked": false
                }, 
"Ethernet3/6/1": {
"portname": "Ethernet3/6/1", 
"ifindex": 3073, 
"linked": true
                }, 
"Ethernet3/5/1": {
"portname": "Ethernet3/5/1", 
"ifindex": 3061, 
"linked": true
                }, 
"Ethernet3/2/1": {
"portname": "Ethernet3/2/1", 
"ifindex": 3025, 
"linked": false
                }, 
"Ethernet3/8/1": {
"portname": "Ethernet3/8/1", 
"ifindex": 3097, 
"linked": true
                }, 
"Ethernet3/1/1": {
"portname": "Ethernet3/1/1", 
"ifindex": 3013, 
"linked": false
                }, 
"Management1/1": {
"portname": "Management1/1", 
"ifindex": 999011, 
"linked": false
                }, 
"Ethernet3/34/1": {
"portname": "Ethernet3/34/1", 
"ifindex": 3409, 
"linked": false
                }, 
"Ethernet3/31/1": {
"portname": "Ethernet3/31/1", 
"ifindex": 3373, 
"linked": false
                }, 
"Ethernet3/7/1": {
"portname": "Ethernet3/7/1", 
"ifindex": 3085, 
"linked": true
                }
            }
        }, 
"core212": {
"name": "core212", 
"agent": "10.10.129.64", 
"ports": {
"Ethernet3/3/1": {
"portname": "Ethernet3/3/1", 
"ifindex": 3037, 
"linked": false
                }, 
"Ethernet3/12/1": {
"portname": "Ethernet3/12/1", 
"ifindex": 3145, 
"linked": false
                }, 
"Ethernet3/2/1": {
"portname": "Ethernet3/2/1", 
"ifindex": 3025, 
"linked": false
                }, 
"Ethernet3/13/1": {
"portname": "Ethernet3/13/1", 
"ifindex": 3157, 
"linked": false
                }, 
"Ethernet3/31/1": {
"portname": "Ethernet3/31/1", 
"ifindex": 3373, 
"linked": false
                }, 
"Ethernet3/32/1": {
"portname": "Ethernet3/32/1", 
"ifindex": 3385, 
"linked": false
                }, 
"Ethernet3/18/1": {
"portname": "Ethernet3/18/1", 
"ifindex": 3217, 
"linked": true
                }, 
"Ethernet3/28/1": {
"portname": "Ethernet3/28/1", 
"ifindex": 3337, 
"linked": true
                }, 
"Ethernet3/33/1": {
"portname": "Ethernet3/33/1", 
"ifindex": 3397, 
"linked": false
                }, 
"Ethernet3/5/1": {
"portname": "Ethernet3/5/1", 
"ifindex": 3061, 
"linked": true
                }, 
"Ethernet3/8/1": {
"portname": "Ethernet3/8/1", 
"ifindex": 3097, 
"linked": true
                }, 
"Ethernet3/34/1": {
"portname": "Ethernet3/34/1", 
"ifindex": 3409, 
"linked": false
                }, 
"Ethernet3/36/1": {
"portname": "Ethernet3/36/1", 
"ifindex": 3433, 
"linked": false
                }, 
"Ethernet3/35/1": {
"portname": "Ethernet3/35/1", 
"ifindex": 3421, 
"linked": false
                }, 
"Ethernet3/15/1": {
"portname": "Ethernet3/15/1", 
"ifindex": 3181, 
"linked": true
                }, 
"Ethernet3/7/1": {
"portname": "Ethernet3/7/1", 
"ifindex": 3085, 
"linked": true
                }, 
"Ethernet3/16/1": {
"portname": "Ethernet3/16/1", 
"ifindex": 3193, 
"linked": true
                }, 
"Ethernet3/17/1": {
"portname": "Ethernet3/17/1", 
"ifindex": 3205, 
"linked": true
                }, 
"Management1/1": {
"portname": "Management1/1", 
"ifindex": 999011, 
"linked": false
                }, 
"Ethernet3/26/1": {
"portname": "Ethernet3/26/1", 
"ifindex": 3313, 
"linked": true
                }, 
"Ethernet3/25/1": {
"portname": "Ethernet3/25/1", 
"ifindex": 3301, 
"linked": true
                }, 
"Ethernet3/21/1": {
"portname": "Ethernet3/21/1", 
"ifindex": 3253, 
"linked": false
                }, 
"Ethernet3/11/1": {
"portname": "Ethernet3/11/1", 
"ifindex": 3133, 
"linked": false
                }, 
"Ethernet3/6/1": {
"portname": "Ethernet3/6/1", 
"ifindex": 3073, 
"linked": true
                }, 
"Ethernet3/27/1": {
"portname": "Ethernet3/27/1", 
"ifindex": 3325, 
"linked": true
                }, 
"Ethernet3/1/1": {
"portname": "Ethernet3/1/1", 
"ifindex": 3013, 
"linked": false
                }, 
"Ethernet3/23/1": {
"portname": "Ethernet3/23/1", 
"ifindex": 3277, 
"linked": false
                }, 
"Ethernet3/22/1": {
"portname": "Ethernet3/22/1", 
"ifindex": 3265, 
"linked": false
                }
            }
        }
    }, 
"links": {
"link5": {
"node1": "leaf260", 
"node2": "core212", 
"port2": "Ethernet3/15/1", 
"port1": "Ethernet31"
        }, 
"link4": {
"node1": "leaf260", 
"node2": "core212", 
"port2": "Ethernet3/5/1", 
"port1": "Ethernet30"
        }, 
"link7": {
"node1": "leaf259", 
"node2": "core210", 
"port2": "Ethernet3/6/1", 
"port1": "Ethernet29"
        }, 
"link6": {
"node1": "leaf260", 
"node2": "core212", 
"port2": "Ethernet3/25/1", 
"port1": "Ethernet32"
        }, 
"link1": {
"node1": "leaf260", 
"node2": "leaf259", 
"port2": "Ethernet5/1", 
"port1": "Ethernet5/1"
        }, 
"link3": {
"node1": "leaf260", 
"node2": "core210", 
"port2": "Ethernet3/5/1", 
"port1": "Ethernet29"
        }, 
"link2": {
"node1": "leaf260", 
"node2": "leaf259", 
"port2": "Ethernet6/1", 
"port1": "Ethernet6/1"
        }, 
"link9": {
"node1": "leaf259", 
"node2": "core212", 
"port2": "Ethernet3/16/1", 
"port1": "Ethernet31"
        }, 
"link8": {
"node1": "leaf259", 
"node2": "core212", 
"port2": "Ethernet3/6/1", 
"port1": "Ethernet30"
        }, 
"link15": {
"node1": "leaf331", 
"node2": "core212", 
"port2": "Ethernet3/17/1", 
"port1": "Ethernet51/1"
        }, 
"link14": {
"node1": "leaf331", 
"node2": "core212", 
"port2": "Ethernet3/7/1", 
"port1": "Ethernet50/1"
        }, 
"link17": {
"node1": "leaf332", 
"node2": "core210", 
"port2": "Ethernet3/8/1", 
"port1": "Ethernet49/1"
        }, 
"link16": {
"node1": "leaf331", 
"node2": "core212", 
"port2": "Ethernet3/27/1", 
"port1": "Ethernet52/1"
        }, 
"link11": {
"node1": "leaf331", 
"node2": "leaf332", 
"port2": "Ethernet35", 
"port1": "Ethernet35"
        }, 
"link10": {
"node1": "leaf259", 
"node2": "core212", 
"port2": "Ethernet3/26/1", 
"port1": "Ethernet32"
        }, 
"link13": {
"node1": "leaf331", 
"node2": "core210", 
"port2": "Ethernet3/7/1", 
"port1": "Ethernet49/1"
        }, 
"link12": {
"node1": "leaf331", 
"node2": "leaf332", 
"port2": "Ethernet36", 
"port1": "Ethernet36"
        }, 
"link20": {
"node1": "leaf332", 
"node2": "core212", 
"port2": "Ethernet3/28/1", 
"port1": "Ethernet52/1"
        }, 
"link19": {
"node1": "leaf332", 
"node2": "core212", 
"port2": "Ethernet3/18/1", 
"port1": "Ethernet51/1"
        }, 
"link18": {
"node1": "leaf332", 
"node2": "core212", 
"port2": "Ethernet3/8/1", 
"port1": "Ethernet50/1"
        }
    }
}

There are important scaleability and cost advantages to placing the sFlow-RT analytics engine in front of the metrics collection service. For example, in large scale cloud environments the metrics for each member of a dynamic pool isn't necessarily worth trending since virtual machines are frequently added and removed. Instead, sFlow-RT tracks all the members of the pool, calculates summary statistics for the pool, and logs the summary statistics. This pre-processing can significantly reduce storage requirements, reducing costs and increasing query performance. The sFlow-RT analytics software also calculates traffic flow metrics, hot/missed Memcache keys, top URLs, exports events via syslog to Splunk, Logstash etc. and provides access to detailed metrics through its REST API.

wget http://www.inmon.com/products/sFlow-RT/sflow-rt.tar.gz
tar -xvzf sflow-rt.tar.gz
cd sflow-rt

var url = "https://metrics-api.librato.com/v1/metrics";
var user = "first.last@mycompany.com";
var token = "55add91c806fb5f634ad1a334789a32e8d10a597815e6865aa84f0749324450e";

setIntervalHandler(function() {
  var metrics = ['min:load_one','q1:load_one','med:load_one',
'q3:load_one','max:load_one'];
  var vals = metric('ALL',metrics,{os_name:['linux']});
  var gauges = {};
  for each (var val in vals) {
     gauges[val.metricName] = {
"value": val.metricValue,
"source": "Linux_Pool"
     };
  }
  var body = {"gauges":gauges};
  http(url,'post', 'application/json', JSON.stringify(body), user, token);
} , 15); 

./start.sh

Figure 1: Two-Level Folded CLOS Network Topology Example

Figure 2: OF-DPA Programming Pipeline for ECMP

The speed with which this new features can be delivered on hardware from the wide range of vendors supporting Cumulus Linux is a powerful illustration of the power of open networking. While support for the Broadcom ASIC table extension has been checking into the Host sFlow trunk it hasn't yet made it into the Cumulus Networks binary repositories. However, Cumulus Linux is an open platform, so users are free to download sources, compile and install the latest software version direct from SourceForge.

bcm_asic_host_entries 4
bcm_host_entries_max 8192
bcm_ipv4_entries 0
bcm_ipv4_entries_max 0
bcm_ipv6_entries 0
bcm_ipv6_entries_max 0
bcm_ipv4_ipv6_entries 9
bcm_ipv4_ipv6_entries_max 16284
bcm_long_ipv6_entries 3
bcm_long_ipv6_entries_max 256
bcm_total_routes 10
bcm_total_routes_max 32768
bcm_ecmp_nexthops 0
bcm_ecmp_nexthops_max 2016
bcm_mac_entries 3
bcm_mac_entries_max 32768
bcm_ipv4_neighbors 4
bcm_ipv6_neighbors 0
bcm_ipv4_routes 0
bcm_ipv6_routes 0
bcm_acl_ingress_entries 842
bcm_acl_ingress_entries_max 4096
bcm_acl_ingress_counters 68
bcm_acl_ingress_counters_max 4096
bcm_acl_ingress_meters 18
bcm_acl_ingress_meters_max 8192
bcm_acl_ingress_slices 3
bcm_acl_ingress_slices_max 8
bcm_acl_egress_entries 36
bcm_acl_egress_entries_max 512
bcm_acl_egress_counters 36
bcm_acl_egress_counters_max 1024
bcm_acl_egress_meters 18
bcm_acl_egress_meters_max 512
bcm_acl_egress_slices 2
bcm_acl_egress_slices_max 2

DevOps

var network_wide_metrics = [
'max:bcm_host_utilization',
'max:bcm_mac_utilization',
'max:bcm_ipv4_ipv6_utilization',
'max:bcm_total_routes_utilization',
'max:bcm_ecmp_nexthops_utilization',
'max:bcm_acl_ingress_utilization',
'max:bcm_acl_ingress_meters_utilization',
'max:bcm_acl_ingress_counters_utilization',
'max:bcm_acl_egress_utilization',
'max:bcm_acl_egress_meters_utilization',
'max:bcm_acl_egress_counters_utilization'
];

var max_utilization = 80;

setIntervalHandler(function() {
  var vals = metric('ALL',network_wide_metrics);
  var graphite_metrics = {};
  for each (var val in vals) {
    if(!val.hasOwnProperty('metricValue')) continue;

    // generate syslog events for over utilized tables
    if(val.metricValue >= max_utilization) {
       var event = {
"asic_table":val.metricName,
"utilization":val.metricValue,
"switchIP":val.agent
       };
       try {
         syslog(
'10.0.0.1', // syslog collector: splunk>, logstash, etc.
           514,        // syslog port
           16,         // facility = local0
           5,          // severity = notice
           event
        );
      } catch(e) { logWarning("syslog() failed " + e); }
    }

    // add metric to graphite set
    graphite_metrics["network.podA."+val.metricName] = val.metricValue;
  }

  // sent metrics to graphite
  try {
    graphite(
'10.0.0.151',  // graphite server
      2003,          // graphite carbon UDP port
      graphite_metrics
    );
  } catch(e) { logWarning("graphite() failed " + e); }
},15);

Real-time traffic analytics

Scaleable traffic measurement is possible because Broadcom ASICs implement hardware support for sFlow monitoring, providing cost effective, line rate visibility that is build into the switches and scales to all port speeds (1G, 10G, 25G, 40G, 50G, 100G, ...) and the high port counts found in large leaf and spine networks.

SDN

The ability to install software on the switches is transformative, allowing third party developers and network operators transparent access to the full capabilities of the switch and build solutions that efficiently handle automation challenges.

• DDoS flood attack mitigation
• Large "Elephant" flow marking
• Large flow load balancing

Prescriptive Topology Manager

cumulus@wbench:~$ curl http://leaf1:8080/ptm

{
"links": {
"L1": {
"node1": "leaf1", 
"node2": "spine1", 
"port1": "swp1s0", 
"port2": "swp49"
  },
  ...
 }
}

LLDP

cumulus@wbench:~$ curl http://leaf1:8080/hostname

"leaf1"

cumulus@wbench:~$ curl http://leaf1:8080/lldp/neighbors

{
"lldp": [
     {
"interface": [
         {
"name": "eth0",
"via": "LLDP",
"chassis": [
             {
"id": [
                 {
"type": "mac",
"value": "6c:64:1a:00:2e:7f"
                 }
               ],
"name": [
                 {
"value": "colo-tor-3"
                 }
               ]
             }
           ],
"port": [
             {
"id": [
                 {
"type": "ifname",
"value": "swp10"
                 }
               ],
"descr": [
                 {
"value": "swp10"
                 }
               ]
             }
           ]
         },
         ...
     }
   ]
 }

cumulus@wbench:~$ curl http://leaf1:8080/lldp/configuration

{
"configuration": [
     {
"config": [
         {
"tx-delay": [
             {
"value": "30"
             }
           ],
           ...
         }
       ]
     }
   ]
 }

Topology discovery with LLDP

#!/usr/bin/env python

import sys, re, fileinput, json, requests

switch_list = ['leaf1','leaf2','spine1','spine2']

l = 0
linkdb = {}
links = {}
for switch_name in switch_list:
  # verify that lldp configuration exports hostname,ifname information
  r = requests.get("http://%s:8080/lldp/configuration" % (switch_name));
  if r.status_code != 200: continue
  config = r.json()
  lldp_hostname = config['configuration'][0]['config'][0]['hostname'][0]['value']
  if lldp_hostname != '(none)': continue
  lldp_porttype = config['configuration'][0]['config'][0]['lldp_portid-type'][0]['value']
  if lldp_porttype != 'ifname': continue
  # local hostname 
  r = requests.get("http://%s:8080/hostname" % (switch_name));
  if r.status_code != 200: continue
  host = r.json()
  # get neighbors
  r = requests.get("http://%s:8080/lldp/neighbors" % (switch_name));
  if r.status_code != 200: continue
  neighbors = r.json()
  interfaces = neighbors['lldp'][0]['interface']
  for i in interfaces:
    # local port name
    port = i['name']
    # neighboring hostname
    nhost = i['chassis'][0]['name'][0]['value']
    # neighboring port name
    nport = i['port'][0]['descr'][0]['value']
    if not host or not port or not nhost or not nport: continue
    if host < nhost:
      link = {'node1':host,'port1':port,'node2':nhost,'port2':nport}
    else:
      link = {'node1':nhost,'port1':nport,'node2':host,'port2':port}
    keystr = "%s %s -- %s %s" % (link['node1'],link['port1'],link['node2'],link['port2'])
    if keystr in linkdb:
       # check consistency
       prev = linkdb[keystr]
       if (link['node1'] != prev['node1'] 
           or link['port1'] != prev['port1']
           or link['node2'] != prev['node2']
           or link['port2'] != prev['port2']): raise Exception('Mismatched LLDP', keystr)
    else:
       linkdb[keystr] = link
       linkname = 'L%d' % (l)
       links[linkname] = link
       l += 1

top = {'links':links}               
print json.dumps(top,sort_keys=True, indent=1)

cumulus@wbench:~$ ./lldp.py 
{
"links": {
"L0": {
"node1": "colo-tor-3", 
"node2": "leaf1", 
"port1": "swp10", 
"port2": "eth0"
  }, 
  ...
 }
}

Demonstration

_sflow._udp     30      SRV     0 0 6343 wbench

This workbench example automatically provisions an OpenStack cluster on the two servers along with the network to connect them. In much the same way OpenStack provides access to virtual resources, Cumulus' Remote Lab leverages the automation capabilities of open hardware to provide multi-tenant access to physical servers and networks.

cumulus@server1:~$ while true; do iperf -c 10.4.2.2 -t 20; sleep 20; done
------------------------------------------------------------
Client connecting to 10.4.2.2, TCP port 5001
TCP window size: 1.06 MByte (default)
------------------------------------------------------------
[  3] local 10.4.1.2 port 57234 connected with 10.4.2.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-20.0 sec  21.9 GBytes  9.41 Gbits/sec
------------------------------------------------------------
Client connecting to 10.4.2.2, TCP port 5001
TCP window size: 1.06 MByte (default)
------------------------------------------------------------
[  3] local 10.4.1.2 port 57240 connected with 10.4.2.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-20.0 sec  10.1 GBytes  4.34 Gbits/sec
------------------------------------------------------------
Client connecting to 10.4.2.2, TCP port 5001
TCP window size: 1.06 MByte (default)
------------------------------------------------------------
[  3] local 10.4.1.2 port 57241 connected with 10.4.2.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-20.0 sec  21.9 GBytes  9.41 Gbits/sec
------------------------------------------------------------

Readers might be surprised by the frequency of collisions given the number of links in the network. Packets take two hops to go from leaf1 to leaf2 - routed via spine1 or spine2. In addition, the links between switches are paired, so there are 8 possible two hop paths from leaf1 to leaf2. The explanation involves looking at the conditional probability that the second flow with overlap with the first. Suppose the first flow takes is routed to spine1 via port swp1s0 and that spine1 routes the flow to leaf2 via port swp51. If the second flow is routed via any of the 4 paths through spine2, there is no collision. However, if it is routed via spine1, there is only 1 path that avoids collisions (leaf1 port swp1s1 to spine1 port swp52). This means that there is a 5 / 8 chance of avoiding a collision, or a 3/8 (37.5%) chance that the two flow will collide. The probability of flow collisions is surprisingly high even on very large networks with many spine switches and paths (see Birthday Paradox). 

The Discard trend lags the Collision trend because discards are reported using sFlow counters and the Collision metric are based on packet samples - see Measurement delay, counters vs. packet samples

This small four switch leaf and spine network is composed of 12 x 10 Gbits/sec links which would require 24 x 10 Gbits/sec taps with associated probes and collector to fully monitor using traditional tools used to monitor legacy data center networks. The cost and complexity of tapping leaf and spine topologies is prohibitive. However, leaf and spine switches typically include hardware support for the sFlow measurement standard, embedding line rate visibility into every switch port for network wide coverage at no extra cost. In this example, the Fabric View analytics software is running on a commodity physical or virtual server consuming 1% CPU and 200 MBytes RAM.

• SDN fabric controller for commodity data center switches
• ECMP visibility with Cumulus Linux
• Open vSwitch 2014 Fall Conference
• SDN fabric controllers

• 0:00 Introduction to Big Tap
• 7:00 sFlow generation and use cases
• 12:30 Demonstration of real-time tap triggering based on sFlow

• Traffic engineering
• Quality of service
• DDoS mitigation
• Security

while true; do iperf -B 10.200.3.32 -c 10.200.3.42 -n 10000M; done

while true; do iperf -B 10.200.3.33 -c 10.200.3.43 -n 10000M; done

index = hash(packet fields) % group.size
selected_physical_port = group[index]

curl -H "Content-Type: application/json" -X POST http://localhost:8181/onos/segmentrouting/tunnel -d '{"tunnel_id":"t1", "label_path":[103,106,104]}'

curl -H "Content-Type: application/json" -X POST http://localhost:8181/onos/segmentrouting/policy -d '{"policy_id":"p1", "priority":1000, "src_ip":"10.200.3.33/32", "dst_ip":"10.200.4.43/32", "proto_type":"TCP", "src_tp_port":53163, "dst_tp_port":5001, "policy_type":"TUNNEL_FLOW", "tunnel_id":"t1"}'

In Overall Data Center Costs James Hamilton breaks down the monthly costs of running a data center and puts the cost of network equipment at 8% of the overall cost.

• Leaf and Spine (Clos) topology
• Merchant silicon based switches (white box / brite box / bare metal)
• Centralized control (SDN)

ONS2015 Solutions Showcase: Open-source spine-leaf Fabric

In this example, the Bird routing daemon is running on the Internet Router and the TopN prefixes are written into a filter file that restricts prefixes that can be installed in the hardware. 

Route caching is not a new idea, the paper, Revisiting Route Caching: The World Should Be Flat, discusses the history of route caching and discusses applications to contemporary workloads and requirements.

wget http://www.inmon.com/products/sFlow-RT/sflow-rt.tar.gz
tar -xvzf sflow-rt.tar.gz
cd sflow-rt

-Dbgp.start=yes -Dbgp.port=1179

bgpAddNeighbor('10.0.0.254',NNNN);
bgpAddSource('10.0.0.253','10.0.0.254',10,'bytes');

router bgp NNNN
 bgp router-id 10.0.0.254
 neighbor 10.0.0.252 remote-as NNNN
 neighbor 10.0.0.252 port 1179
 neighbor 10.0.0.252 route-reflector-client

./start.sh

curl "http://10.0.0.162:8008/bgp/topprefixes/10.0.0.30/json?direction=destination&maxPrefixes=5&minValue=1"
{
"as": NNNN,
"direction": "destination",
"id": "N.N.N.N",
"learnedPrefixesAdded": 568313,
"learnedPrefixesRemoved": 9,
"nPrefixes": 567963,
"pushedPrefixesAdded": 0,
"pushedPrefixesRemoved": 0,
"startTime": 1436830843625,
"state": "established",
"topPrefixes": [
  {
"aspath": "NNNN",
"localpref": 888,
"nexthop": "N.N.N.N",
"origin": "IGP",
"prefix": "0.0.0.0/0",
"value": 680740.5504781345
  },
  {
"aspath": "NNNN-NNNN",
"localpref": 100,
"nexthop": "N.N.N.N",
"origin": "IGP",
"prefix": "N.N.0.0/14",
"value": 58996.251739893225
  },
  {
"aspath": "NNNN-NNNNN",
"localpref": 130,
"nexthop": "N.N.N.N",
"origin": "IGP",
"prefix": "N.N.0.0/13",
"value": 7966.802831354894
  },
  {
"localpref": 100,
"med": 2,
"nexthop": "N.N.N.N",
"origin": "IGP",
"prefix": "N.N.N.0/18",
"value": 3059.8853014045844
  },
  {
"aspath": "NNNN",
"localpref": 1010,
"med": 0,
"nexthop": "N.N.N.N",
"origin": "IGP",
"prefix": "N.N.N.0/24",
"value": 1635.0250535959976
  }
 ],
"valuePercentCoverage": 99.67670497397555,
"valueTopPrefixes": 752398.5154043833,
"valueTotal": 754838.871931838
}

Try running this query on your own network to find out how many prefixes are required to cover 90%, 95%, and 99% of the traffic. If you have results you can share, please post them as comments to this article.

bgpAddNeighbor('10.0.0.254',65000);
bgpAddSource('10.0.0.253','10.0.0.254',10,'bytes');
bgpAddNeighbor('10.0.0.253',65000);

var installed = {};
setIntervalHandler(function() {
  let now = Date.now();
  let top = bgpTopPrefixes('10.0.0.254',100,1,'destination');
  if(!top || !top.hasOwnProperty('topPrefixes')) return;

  let tgt = bgpTopPrefixes('10.0.0.253',0);
  if(!tgt || 'established' != tgt.state) return;

  for(let i = 0; i < top.topPrefixes.length; i++) {
     let entry = top.topPrefixes[i];
     if(bgpAddRoute('10.0.0.253',entry)) {
       installed[entry.prefix] = now; 
     }
  }
  for(let prefix in installed) {
     let time = installed[prefix];
     if(time === now) continue;
     if(bgpRemoveRoute('10.0.0.253',prefix)) {
        delete installed[prefix];
     } 
  }
}, 5);

1. setIntervalHandler registers a function that is called every 5 seconds
2. The interval handler queries for the top 100 destination prefixes
3. Active prefixes are pushed to the switch using bgpAddRoute
4. Inactive prefixes are withdrawn using bgpRemoveRoute
5. The bgpAddRoute/bgpRemoveRoute functions are BGP session state aware and will only forward changes

[root@peer1 ~]# traceroute -s 192.168.250.1 192.168.251.1
traceroute to 192.168.251.1 (192.168.251.1), 30 hops max, 40 byte packets
 1  192.168.152.2 (192.168.152.2)  3.090 ms  3.014 ms  2.927 ms
 2  192.168.150.3 (192.168.150.3)  3.377 ms  3.161 ms  3.099 ms
 3  192.168.251.1 (192.168.251.1)  6.440 ms  6.228 ms  3.217 ms

Peer 1

router bgp 150
 bgp router-id 192.168.150.1
 network 192.168.250.0/24
 neighbor 192.168.152.3 remote-as 152
 neighbor 192.168.152.3 ebgp-multihop 2

Peer 2

router bgp 151
 bgp router-id 192.168.151.1
 network 192.168.251.0/24
 neighbor 192.168.152.3 remote-as 152
 neighbor 192.168.152.3 ebgp-multihop 2

Software Router

interface lo
  ip address 192.168.152.3/32

router bgp 152
 bgp router-id 192.168.152.3
 neighbor 192.168.150.1 remote-as 150
 neighbor 192.168.150.1 update-source 192.168.152.3
 neighbor 192.168.150.1 passive
 neighbor 192.168.151.1 remote-as 151
 neighbor 192.168.151.1 update-source 192.168.152.3
 neighbor 192.168.151.1 passive
 neighbor 10.0.0.162 remote-as 152
 neighbor 10.0.0.162 port 1179
 neighbor 10.0.0.162 timers connect 30
 neighbor 10.0.0.162 route-reflector-client

Hardware Switch

router bgp 65000
 bgp router-id 0.0.0.1
 neighbor 10.0.0.162 remote-as 65000
 neighbor 10.0.0.162 port 1179
 neighbor 10.0.0.162 timers connect 30

auto br_xen
iface br_xen
  bridge-ports swp1 swp2 swp3
  address 192.168.150.2/24
  address 192.168.151.2/24
  address 192.168.152.2/24

SDN Routing Application

bgpAddNeighbor('10.0.0.152',152,false);
bgpAddNeighbor('10.0.0.233',65000,true);
bgpAddSource('10.0.0.233','10.0.0.152',10);

var installed = {};
setIntervalHandler(function() {
  let now = Date.now();
  let top = bgpTopPrefixes('10.0.0.152',20000,1);
  if(!top || !top.hasOwnProperty('topPrefixes')) return;

  let tgt = bgpTopPrefixes('10.0.0.233',0);
  if(!tgt || 'established' != tgt.state) return;

  for(let i = 0; i < top.topPrefixes.length; i++) {
     let entry = top.topPrefixes[i];
     if(bgpAddRoute('10.0.0.233',entry)) {
       installed[entry.prefix] = now; 
     }
  }
  for(let prefix in installed) {
     let time = installed[prefix];
     if(time === now) continue;
     if(bgpRemoveRoute('10.0.0.233',prefix)) {
        delete installed[prefix];
     } 
  }
}, 1);

$ ./start.sh 
2015-07-21T19:36:52-0700 INFO: Listening, BGP port 1179
2015-07-21T19:36:52-0700 INFO: Listening, sFlow port 6343
2015-07-21T19:36:53-0700 INFO: Starting the Jetty [HTTP/1.1] server on port 8008
2015-07-21T19:36:53-0700 INFO: Starting com.sflow.rt.rest.SFlowApplication application
2015-07-21T19:36:53-0700 INFO: Listening, http://localhost:8008
2015-07-21T19:36:53-0700 INFO: bgp.js started
2015-07-21T19:36:57-0700 INFO: BGP open /10.0.0.152:50010
2015-07-21T19:37:23-0700 INFO: BGP open /10.0.0.233:55097

cumulus@cumulus$ ip route
default via 192.168.152.1 dev br_xen 
10.0.0.0/24 dev eth0  proto kernel  scope link  src 10.0.0.233 
192.168.150.0/24 dev br_xen  proto kernel  scope link  src 192.168.150.2 
192.168.151.0/24 dev br_xen  proto kernel  scope link  src 192.168.151.2 
192.168.152.0/24 dev br_xen  proto kernel  scope link  src 192.168.152.2

iperf -s -B 192.168.251.1

iperf -c 192.168.251.1 -B 192.168.250.1

cumulus@cumulus$ ip route
default via 192.168.152.1 dev br_xen 
10.0.0.0/24 dev eth0  proto kernel  scope link  src 10.0.0.233  
192.168.150.0/24 dev br_xen  proto kernel  scope link  src 192.168.150.2 
192.168.151.0/24 dev br_xen  proto kernel  scope link  src 192.168.151.2 
192.168.152.0/24 dev br_xen  proto kernel  scope link  src 192.168.152.2 
192.168.250.0/24 via 192.168.150.1 dev br_xen  proto zebra  metric 20 
192.168.251.0/24 via 192.168.151.1 dev br_xen  proto zebra  metric 20

[root@peer1 ~]# traceroute -s 192.168.250.1 192.168.251.1
traceroute to 192.168.251.1 (192.168.251.1), 30 hops max, 40 byte packets
 1  192.168.150.2 (192.168.150.2)  3.260 ms  3.151 ms  3.014 ms
 2  192.168.251.1 (192.168.251.1)  4.418 ms  4.351 ms  4.260 ms

$ curl http://10.0.0.162:8008/bgp/topprefixes/10.0.0.52/json
{
"as": 152,
"direction": "destination",
"id": "192.168.152.3",
"learnedPrefixesAdded": 2,
"learnedPrefixesRemoved": 0,
"nPrefixes": 2,
"pushedPrefixesAdded": 0,
"pushedPrefixesRemoved": 0,
"startTime": 1437535255553,
"state": "established",
"topPrefixes": [
  {
"aspath": "150",
"localpref": 100,
"med": 0,
"nexthop": "192.168.150.1",
"origin": "IGP",
"prefix": "192.168.250.0/24",
"value": 1.4462334178258518E7
  },
  {
"aspath": "151",
"localpref": 100,
"med": 0,
"nexthop": "192.168.151.1",
"origin": "IGP",
"prefix": "192.168.251.0/24",
"value": 391390.33359066787
  }
 ],
"valuePercentCoverage": 100,
"valueTopPrefixes": 1.4853724511849185E7,
"valueTotal": 1.4853724511849185E7
}

The test setup was configured to quickly test the concept using limited hardware at hand and can be improved for production deployment (using smaller CIDRs and VLANs to separate peer traffic).