Fabric View

The Fabric View application has been released on Github, https://github.com/sflow-rt/fabric-view. Fabric View provides real-time visibility into the performance of leaf and spine ECMP fabrics.
A leaf and spine fabric is challenging to monitor. The fabric spreads traffic across all the switches and links in order to maximize bandwidth. Unlike traditional hierarchical network designs, where a small number of links can be monitored to provide visibility, a leaf and spine network has no special links or switches where running CLI commands or attaching a probe would provide visibility. Even if it were possible to attach probes, the effective bandwidth of a leaf and spine network can be as high as a Petabit/second, well beyond the capabilities of current generation monitoring tools.

Fabric View solves the visibility challenge by using the industry standard sFlow instrumentation built into most data center switches. Fabric View represents the fabric as if it were a single large chassis switch, treating each leaf switch as a line card and the spine switches as the backplane. The result is an intuitive tool that is easily understood by anyone familiar with traditional networks.

Fabric View provides real-time, second-by-second visibility to traffic, identifying top talkers, protocols, tenants, tunneled traffic, etc. In addition, Fabric View reveals key fabric performance indicators such as number of congested spine links, links with colliding Elephant flows, discards and errors.

If you have a leaf / spine network, download the software and try it out. Community support is available on sFlow-RT.com.

Demo

If you don't have access to a live network, Fabric View includes data captured from the Cumulus Networks workbench network shown above, a 2 leaf / 2 spine 10Gbit/s network.

First, download sFlow-RT

wget http://www.inmon.com/products/sFlow-RT/sflow-rt.tar.gz
tar -xvzf sflow-rt.tar.gz
cd sflow-rt

Next, install Fabric View:

cd app
wget http://github.com/sflow-rt/fabric-view/archive/master.zip
unzip master
mv fabric-view-master dashboard-example
rm master
cd ..

Now edit the start.sh script to playback the captured packet trace:

#!/bin/sh

HOME=`dirname $0`
cd $HOME

JAR="./lib/sflowrt.jar"
JVM_OPTS="-Xincgc -Xmx200m -Dsflow.file=app/fabric-view/demo/ecmp.pcap"
RT_OPTS="-Dsflow.port=6343 -Dhttp.port=8008"
SCRIPTS="-Dscript.file=init.js"

exec java ${JVM_OPTS} ${RT_OPTS} ${SCRIPTS} -jar ${JAR}

Start sFlow-RT:

[user@server sflow-rt]$ ./start.sh 
2015-10-09T11:08:25-0700 INFO: Reading PCAP file, app/fabric-view/demo/ecmp.pcap
2015-10-09T11:08:26-0700 INFO: Starting the Jetty [HTTP/1.1] server on port 8008
2015-10-09T11:08:26-0700 INFO: Starting com.sflow.rt.rest.SFlowApplication application
2015-10-09T11:08:26-0700 INFO: Listening, http://localhost:8008
2015-10-09T11:08:26-0700 INFO: init.js started
2015-10-09T11:08:26-0700 INFO: app/fabric-view/scripts/fabric-view-stats.js started
2015-10-09T11:08:26-0700 INFO: app/fabric-view/scripts/fabric-view.js started
2015-10-09T11:08:26-0700 INFO: app/fabric-view/scripts/fabric-view-elephants.js started
2015-10-09T11:08:26-0700 INFO: app/fabric-view/scripts/fabric-view-usr.js started
2015-10-09T11:08:27-0700 INFO: init.js stopped

Now install the network topology and address groupings:

[user@server sflow-rt]$ curl -H 'Content-Type:application/json' -X PUT --data @app/fabric-view/demo/topology.json http://127.0.0.1:8008/app/fabric-view/scripts/fabric-view.js/topology/json
[user@server sflow-rt]$ curl -H 'Content-Type:application/json' -X PUT --data @app/fabric-view/demo/groups.json http://127.0.0.1:8008/app/fabric-view/scripts/fabric-view.js/groups/json

Finally, access the web interface http://server:8008/app/fabric-view/html/ and you should see the screen shown at the top of this article.

Real-time control

Visibility is just the starting point. Real-time detection of congested links and Elephant flows can be used to drive software defined networking (SDN) control actions to improve performance, for example, by marking and/or steering Elephant flows.

Leaf and spine traffic engineering using segment routing and SDN describes a demonstration shown at the 2015 Open Network Summit that integrated Fabric View with the ONOS controller to load balance Elephant flows.

The fabric-view-elephants.js script has three dummy function that are place holders for control actions:

function elephantStart(flowKey, rec) {
  // place holder to mark elephant flows
}

function linkBusy(linkDs, linkRec, now) {
  // place holder to steer elephant flows
}

function elephantEnd(flowKey, rec) {
  // place holder to remove marking
}

RESTful control of Cumulus Linux ACLs demonstrated how flows could be marked. Adding the following code to the fabric-view-elephants.js script demonstrates how the large flow marking function can be integrated in Fabric View:

var aclProtocol = "http";
var aclPort = 8080;
var aclRoot = '/acl/';
var mark_dscp = 10;
var mark_cos  = 5;
var id = 0;

var marking_enabled = true;

function newRequest(host, path) {
    var url = aclProtocol + "://" + host;
    if(aclPort) url += ":" + aclPort;
    url += aclRoot + path;
    var req = {"url":url};
    if(aclUser) req.user = aclUser;
    if(aclPasswd) req.password = aclPasswd;
    return req;
}

function submitRequest(req) {
    if(!req.error) req.error =  function(err) { logWarning("request error " + req.url + " error: " + err); };
    try { httpAsync(req); }
    catch(e) { logWarning('bad request, ' + req.url + "" + e); }
}

function elephantStart(flowKey, rec) {
   if(!marking_enabled) return;

   var [ipsrc,ipdst,proto,srcprt,dstprt] = flowKey.split(',');

   // only interested in TCP flows
   if(proto !== 6) return;

   var acl = [
'[iptables]',
'# marking Elephant flow',
'-t mangle -A FORWARD --in-interface swp+ '
     + ' -s ' + ipsrc + ' -d ' + ipdst 
     + ' -p tcp --sport ' + srcprt + ' --dport ' + dstprt
     + ' -j SETQOS --set-dscp ' + mark_dscp + ' --set-cos ' + mark_cos
   ];

   var rulename = 'mark' + id++;
   var req = newRequest(rec.agent, '/acl/'+rulename);
   req.operation = 'PUT';
   req.headers = {
"Content-Type":"application/json; charset=utf-8",
"Accept":"application/json"
   };
   req.body = JSON.stringify(acl);
   req.error= function(res) {
      logWarning('mark failed=' + res + ' agent=' + rec.agent);
   };
   req.success = function(res) {
      logInfo('mark rule=' + rulename);
   };
   rec.rulename=rulename; 
   submitRequest(req);
}

function elephantEnd(flowKey, rec) {
   if(!rec.rulename) return;

   var req = newRequest(rec.agent, '/acl/'+rec.rulename);
   req.operation = 'DELETE';
   req.error = function(res) {
      logInfo('delete failed='+res);
   };
   req.success = function(res) {
      logInfo('delete rule='+rec.rulename);
   };
   submitRequest(req);
}

This example doesn't only apply to Cumulus Linux. It can easily be modified to interact with other vendor's switch API's, for example NX-API for Cisco Nexus 9k/3k switches, eAPI for Arista switches, or with SDN controller REST APIs, such as Floodlight, OpenDaylight, ONOS etc.