OF Actions using eBPF programs

I was just thinking about this and searched in Internet and I found that this is already implemented and part of OVS.  Please see the series of patches here:

http://openvswitch.org/pipermail/dev/2015-February/051002.html

Also, if you want to look at the patched OVS source code,  check this Linux distribution :

https://github.com/azhou-nicira/net-next

You can check the net/openvswitch directory to understand BPF changes.

Essentially, above patches are addressing one of the concerns of OF critics.  OF specifications today define actions.  Any new action requirement has to go through the standardization process in ONF, which can take few months, if not years. Many companies, like NICIRA, Freescale defined their own proprietary extensions.  But the acceptability is low as these are not part of the OF specifications. Now with BPF integration in the Kernel OVS data path,  one can define proprietary actions as BPF set of BPF programs.   As long as this generic BPF program action is standardized in OF specifications,  then new actions can be implemented as BPF programs without having to go through standardization process.

As of now, Linux provides hooking the BPF programs to three areas -  system call level,  CLS scheduling level and at the "dev" layer.   Once above patches are accepted in Linux Kernel, OVS datapath becomes fourth area where BPF programs can be associated.

I am hoping that, BPF programs can be executed not only in the Kernel, but also in DPDK accelerated OVS as well as smart-NIC (FPGA, Network processors, AIOP etc..).

Thanks
Srini

virtio based flow API

Hi,

OPNFV DPACC team is trying to standardize the flow API, that would be called from VNFs to talk to flow processing modules in VMM or smart-NIC.

One discussion group was started.  You are welcome to join this group here:

https://app.pivot-it.com/Public/Open-flow-based-distributed-data-path-for-virtualised-data-centers-openflowdatapath/enter/

Why flow flow API from VNFs :  NFV success depends on the performance VNFs can achieve. Many are feeling that NFV flexibility is great, but to achieve similar performance as physical appliances, one needs to have 2x to 10x hardware.  Even with lot more hardware, one can achieve throughput, but issues remain in achieving low packet latency and low packet jitter.  Since, NFV uses generic hardware, cost may not go up (in comparison with physical appliances), but the electricity charges related to powering more hardware and cooling would go up.  Also, space requirements also go up which can add to the cost of operations.

OVS acceleration using DPDK and smart-NICs are addressing this issue by accelerating or offloading virtual switching functionality.  1/2 of the problem is take care by this. Another 1/2 of the problem is packet processing by VNFs.  There are many VNFs, that follow CP-DP model where DP can be offloaded. Many smart-NIC solution want to take care of that too.  But lack of standards is dampening these efforts.  In NFV environments, vNFs and smart-NICs are coming from two different parties.  VNFs can be placed in many servers with smart-NICs  of various vendors. VNF vendors would not like to have drivers for various smart-NICs.  Also, smart-NIC vendors would not like to provide drivers for various VNF operating systems and their versions.  Operators, who are procuring these also would not want to worry about any inter-dependencies. That is where, standardization helps.

Another bigger issue in NFV markets is that type of VNFs that are going to be placed on a given server.  The type of VNFs on a server  keep changing with the time.  Since the type of VNFs are not known a priori,

• One way to address is to provide all kinds of fast path functions in smart-NICs. That is very complicated, not only with respect to number of functions, but also with respect to keeping it up with newer VNF and related fast path functions. 
• Second way is to let the VNFs install their own fast paths in smart-NIC.  That is also challenging as there are multiple  smart-NIC vendors. It is a complexity for vNF vendors to provide fast path functions for multiple  smart-NIC vendors.  It is also a complexity for smart-NIC vendors to ensure that malicious VNF fast path does not hinder its operations.  Few people are exploring the ways to use eBPF based fast path functions.  Since eBPF comes with some kind of verification of program to ensure that it does not hinder other operations,  many think that this could be one approach.  
• Third way is to define generic pipeline of packet stages with dynamic flow programming. Similar to Openflow, but with reduced complexity of Openflow.  Smart-NIC vendors only need to provide flow processing entity.  All the intelligence of using flow processing entity is up to the VNFs.  VNFs don't push any software entity in smart-NIC.  With this mechanism, smart-NIC can be a network processors, AIOP,  FPGA etc.. 

Above discussion forum is trying to create generic flow API that can be used by VNFs to program tables, flows with actions, objects etc.. 

Please join the group and let us discuss technical items there. Once we have some clarity, then we intend to present this to OPNFV DPACC forum.

Thanks

Srini

Openflow based Local fast path for distributed Security

OPNFV DPACC working group is trying to standardize the accelerator interface for vNFs to accelerate the workloads.

Accelerator standardization is required to ensure that  the virtual appliances can work on different types of compute nodes - Compute nodes having no acceleration hardware,  compute nodes having accelerator hardware from vendor 1,  compute nodes having accelerator hardware from vendor 2 etc..
Above becomes critical in cloud environments as compute node hardware is procured independent of procurement of virtual appliances.  Virtual appliance vendors like to have their virtual appliances' images work on multiple compute nodes.  Operators likes to ensure that any virtual appliances they purchase continue to work with future compute node hardware.

OPNFV DPACC team goal is to identify  various acceleration types. And then define software based interface for each acceleration type.

VMM which is typically bound to the compute node hardware is expected to have conversion module from the software interface to local hardware interface.

OPNFV DPACC team is choosing virtio and vring transport to implement software interface.

Accelerations normally fall into two categories -  Look-aside model and Inline/fast-path model.  In Look-aside model,  virtual appliance software sends command to the accelerator and expect response for each command.  The response can come back to virtual appliance in synchronous fashion or asynchronous fashion.  Inline/fast-path model typically is for packet processing kind of vNFs.  Any network function that does not need to inspect all the packets can take advantage of inline/fast-path acceleration model.

In inline/fast-path acceleration model,  network function in virtual appliance establishes a session state (either pro-actively or re-actively to the packets) and then it expects the fast-path accelerator to process further packets.

Many smart-NIC vendors provide facilities to their customers to create fast-path functions in the smart-NIC.  In physical appliances, typically this works best.  Physical appliances are fixed functions and the entire software and hardware comes from one vendor.  This vendor would ensure that both normal network function and fast-path functions work together.

In virtual world,  smart-NIC comes from one vendor and virtual appliances come from separate vendors.  Also, it is unknown at smart-NIC build time, the network functions that would be hosted on the compute node.  It may seem that smart-NIC vendors need to have all types of fast-path functions implemented in the smart-NIC.  It is obvious that it is not feasible and the smart-NIC may not have too much of code space to put all kinds of fast-path functions.  Note that, smart-NIC could be based on FPGA or constrained network processor.

Another model could be that smart-NIC is populated dynamically with fast-path functions based on the type of virtual appliances are being brought up on that node.  This also could be a problematic as there could be multiple smart-NIC vendors using various processors/network-processors/FPGA etc..  Hence, one may need to create similar fast path functions for many smart-NIC types.  There is always security & reliability issues as these functions may not co-exist s they come from various vendors.  Some function may misbehave or some functions might crash other functions. In addition, there is always a concern on amount of time it adds to bringing up and bringing down the virtual appliance.

Hence, some in OPNFV DPACC team believe that smart-NIC must implement flow processor such as openflow.  Openflow has very good advantages, Some of them are :

• Only one logic module.
• One could create multiple instances.
• Each instance can be controlled by separate entity.
• Multiple tables in each instance.
• Flows in the table can be pipelined.
• Flows can be programmed with pre-defined actions.

If smart-NIC implements OF,  orchestration module can create an instance for each vNF and assign the instance ownership for that vNF.  vNF at run time can program the flows with appropriate actions to create the fast path for those flows.  

It is already proven in Industry that OF and some proprietary extensions can be used to realize IPSec,  Firewall, NAT, forwarding,  SLB fast paths.  

Please see this link on the presentation slides made to OPNFV DPACC :  https://wiki.opnfv.org/_media/opnfv-dpacc-fast-path-and-inline-acceleration-flow-processor_v1.pdf.  This slide deck provides graphical picture of above description.

Comments?  Please post your comments here or  at https://app.pivot-it.com/public-583/Open-flow-based-distributed-data-path-for-virtualised-data-centers-openflowdatapath/enter/

Openflow switches - flow setup & termination rate

I attended ONS yesterday.  This is the 1st time, I see many OF hardware switch vendors (either based on NPUs,  FPGAs or ASICs) advertising the performance numbers.  Though the throughput numbers are impressive, flow setup/termination rates are, in my view, are disappointing.  I see the flow setup rate claims are any where between 100 to 2000/sec.

Software based virtual switches support flow setup rate, up to10K/sec for every core. If 4 core part is used, one can get easily 40K flow setups/sec.

In my view,  unless the flow setup rate is improved, it is very unlikely the hardware based OF switches would be popular as the market addressability is limited.

By talking to few, I understand that poor flow setup rate is mainly due to the way TCAMs are used.  As I understand, every flow update (add/delete) requires rearranging the existing entries and that leads to bad performance. I also understand from many of these vendors that they intend to support algorithmic search accelerators to improve the performance of flow insertions and deletions. I also understand from them that this could improve the performance to hundreds of thousands of flow setups/sec.

Unless following are taken care,  hardware based OF switch adoption would be limited.

• Higher flow setup rate. (Must be better than software based switches)
• Ability to maintain millions of flows.
• Support for multiple tables (All 256 tables)
• VxLAN Support
• VxLAN with IPSec

Throughput performance is very important and hardware based OF switches are very good at that.  In my view, all of above are also important for any deployments to seriously consider hardware based OF switches in place of software based switches.

OVS Acceleration - Are you reading between the lines?

OVS, part of Linux distributions, is openflow based software switch implementation.  OVS has two components - User space component and kernel space component.  User space component consists of   OVSDB n(configuration) sub-component,  Openflow agent that communicates with openflow controllers and OF normal path processing logic.  Kernel component implements the fastpath.  OVS calls the kernel component 'datapath'.  OVS datapath maintains one table.  When the packet comes into the datapath logic, it determines the matching flow in the table.  If no entry, then the packet is sent to the user space 'normal path processing' module.  OVS, in user space, finds out if there are all matching entries across all OF tables in that pipeline. If there are all matching entries,  then it pushes the aggregate flow to the kernel datapath.  Further packets of the aggregate flow gets processed in the kernel itself.  Just to complete this discussion,  if OVS does not find any matching flow in the OF pipeline, then it, based on the miss entry,  sends the packet to the controller. Controller, then pushes the flow mod entries in the OVS.

It is believed that Linux Kernel based datapath is not efficient for following reasons.

• Linux Kernel overhead before packet is handed over to the datapath.  Before packet is handed over to the OVS datapath, following processing occurs on packets.
• Interrupt processing
• Softirq processing
• Dev Layer processing
•  Interrupt overhead 
• Fast Path to Normal path overhead for first few packets of a flow.

Few software/networking vendors have implemented user space datapath using OVS 'dpif' provider interface. Many of these vendors have implemented user space  datapath using Intel DPDK in poll mode.  Poll mode dedicates few cores to receive packets from the Ethernet controller directly, eliminating interrupts and  thereby avoiding overhead associated with interrupt processing. Also, since the Ethernet ports are owned by this custom user space process, there is no need for any complex hook processing as needed in Linux kernel.  This user space datapath can start OF processing on the packets immediately upon reception  from Ethernet controllers, thereby avoiding any intermediate processing overhead. There is no doubt that these two features by themselves provide good performance boost.

Many of the OVS acceleration implementations, as I know as on today, don't perform well  in Openstack environment.  Openstack orchestration with KVM not only uses OVS to implement virtual switch, but also it uses OVS to realize network virtualization (using VxLAN, NVGRE). In addition, it also uses Linux IPTables to implement isolation among VMs using security groups configuration.  Moreover, some deployments even use IPSec Over VxLAN to protect the traffic from eaves dropping.  Based on some literature I have read,  user space datapath packages also don't perform well if VMs are connected to OVS via tuntap interfaces.   They work well if there is no need for  isolation among VMs  and where there is no need for overlay based network virtualization.  That is, these acceleration packages work if VMs are trusted and when VLAN interfaces are good enough.  But, we know that Openstack integrated OVS requires VxLANs,  isolation among VMs,  usage of tuntap interfaces and even VxLAN over IPSec.

Note that isolation using firewall,  tuntap interfaces & VxLAN-over-IPsec today use Linux Kernel capabilities.  If OVS datapath is in user space, then packets anyway have to traverse through the Kernel (some times twice) to avail these capabilities, which even may have lot more performance issues over having OVS datapath in Kernel.  I will not be surprised, if the performance is lot lower than native OVS kernel datapath.

Hence, while evaluating OVS acceleration, one should ask for following questions:

• What features of OVS applied in measuring the performance?
• How many OF tables are used are hitting in this performance test?  Make sure that at least 5 tables are used in the OVS as this is very realistic number. 
• Does it use VxLAN?
• Does it implement firewall?
• Does it implement VxLAN over IPSec?
• Does it assume that VMs are untrusted?

If there is a performance boost with all features enabled, then that implementation is good. Otherwise, one should be very careful.

SDN Controller - What to look for?

Introduction

ONF is making Openflow specification as one of the standards enabling non-proprietary communication between central control plane entity & distribute data plane entities. SDN Controllers are the ones which implement control plane for various data path entities.  OVS, being part of the Linux distributions,  is becoming a defacto virtual switch entity in data centers and service provider market segments.  OVS virtual switch, sitting in the Linux host acts as a switch (data path entity) between virtual machines on the Linux host and  rest of the network.

As with virtualization of compute and storage,  networks are also being virtualized. VLAN used to be the one of the techniques  to realize virtual networks. With the limitations of number of VLANs and inability of extending virtual networks using VLANs over L3 networks,  overlay based virtual network technology is replacing VLAN technology.   VxLAN overlay protocol is becoming a choice of virtual network technology.  Today virtual switches (such as OVS) are supporting VxLAN and becoming defactor overlay protocol in data center and service provider networks.

Another important technology that is becoming popular is Openstack.  Openstack is virtual resource orchestration technology to manage virtualization of compute, storage and network resources.  Neutron component of openstack takes care of configuration & management of virtual networks,  network services such as router,  DHCP, Firewall, IPSec VPN and Load balancers.  Neutron provides API to configure these network resources.  Horizon, which is GUI of openstack provides user interface for these services.

Network Nodes (A term used by Openstack community) are the ones which normally sit at the edge of the data centers. They provide firewall capability between Internet & data center networks,  IPSec capability to terminate IPSec tunnels with the  peer networks, SSL offload capability and load balancing capability to distribute the incoming connections to various servers.  Network nodes also acts as routers between external networks & internal virtual networks.  In today networks,  network nodes are self-contained devices.  They have both control plane and data plane  in each node.  Increasingly, it is being felt that SDN concepts can be used to separate out control plane & normal path software from data plane & fast path software.

Network nodes are also being used as routers across virtual networks within data centers for east-west traffic.  Some even  use them as firewall and load balancers for east-west traffic.  Increasingly,  it is being realized that network nodes should not be burdened with the east-west traffic and rather use virtual switches within each compute node to do this job.  That is, virtual switches are being thought to be used as distributed router, firewall and load balancer for east-west traffic.

Advanced network services, which do deep inspection of packets and data, such as Intrusion Prevention,  Web application firewalls,  SSL offload are being deployed in L2 transparent mode to avoid reconfiguration of networks and also to enable vmotion easily.  When deployed as virtual appliances, it also provides agility and scale-out functions.  It requires traffic steering capability to steer the traffic across required virtual appliances.  Though most of the network services are required for north-south traffic, some of them (such as IPS) are equally needed for east-west traffic.

Requirements

As one see from above introduction,  operators would like to see following supported by centralized control plane entity (SDN Controllers)

• Realization of virtual networks
• Control plane for network nodes 
• Normal path software for network nodes.
• Traffic Steering capability to steer the traffic across advanced network services
• Distributed routing, firewall & Load balancing capability for east-west traffic.
• Integration with Openstack Neutron

At no time, centralized entity should  become a bottleneck, hence following additional requirements come in play.

• Scale-out of control plane entity (Clustered Controllers) - Controller Manager.
• Performance of each control plane entity.
• Capacity of each control plane entity.
• Security of control plane entity

Let us dig through each one of the above.

Realization of Virtual Networks:

SDN Controller  is expected to provide following:

Functionality

• Ability to program the virtual switches in compute nodes.
• No special agent in compute nodes.
• Ability to use OVS  using Openflow 1.3+ 
• Ability to realize VxLAN based virtual networks using flow  based tunneling mechanism provided by OVS.
• Ability to realize broadcast & unicast traffic using OF groups.
• Ability to  integrate with Openstack to come to know about VM MAC addresses and the compute nodes on which they are present.
• Ability to use above repository to program the flow entries in virtual switches without resorting broadcasting the traffic to all peer compute nodes.
• Ability to auto-learn VTEP entries.
• Ability to avoid multiple data path entities in a compute nodes - One single data path for each compute node.
• Ability to honor security groups configured in Openstack Nova. That is, ability to program flows based on security groups configuration without using 'IP tables" in the compute node. 
• Ability to use 'Connection tracking" feature to enable stateful firewall functionality.
• Ability to support IPSec in virtual networks across compute nodes.

Capacity: 

Capacity is entirely based on deployment scenario.  Based on best of my knowledge, I believe these parameters are realistic from deployment perspective and also based on capability of hardware.

• Ability to support 256 compute nodes by one controller entity.  if there are more  256 compute nodes, then more controllers in the cluster should be able to take care of rest.
• Ability to support multiple controllers - Ability to distribute controllers across the virtual switches.
• Support for 16K Virtual networks.
• Support for 128K Virtual ports
• Support for 256K VTEP entries.
• Support for 16K IPSec transport mode tunnels

 Performance

• 100K Connections/sec per SDN Controller node (Due to firewall being taken care in the controllers).  With new feature, that is being thought in ONF, called connection template,  this requirement of 100K connections/sec can go down dramatically.  I think 50K connections/sec or connection templates/sec would be good enough.
• 512 IPSec tunnels/sec.

Control Plane & Normal Path software for network nodes

Functionality such as router control plane,  Firewall normal path,  Load balancer normal path & control plane for IPSec (IKE) are the requirements to implement control plane for network nodes.

Functionality

• Ability to integrate with Neutron configuration of routers,  firewalls,  load balancers & IPSec.
• Support for IPv4 & IPv6 unicast routing protocols - OSPF, BGP, RIP and IS-IS.
• Support for IPv4 & IPv6 Multicast routing protocols - PIM-SM
• Support for IP-tables kind of firewall normal path software.
• Support for IKE with public key based authentication.
• Support for LVS kind of L4 load balancing software.
• Ability to support multiple routes, firewall, load balancer instances.
• Ability to support multiple Openflow switches that implement datapath/fastpath functionality of network nodes.
• Ability to receive exception packets from Openflow switches, process them through control plane/normal-path software & programming the resulting flows in openflow switches.
• Ability to support various extensions to Openflow specifications such as
• Bind Objects 
• To bind client-to-server & Server-to-client flows.
• To realize IPSec SAs
• To bind multiple flow together for easy revalidation in case of firewalls.
• Multiple actions/instructions to support:
• IPSec outbound/inbound SA processing.
• Attack checks - Sequence number checks.
• TCP sequence number NAT with delta history table.
• Generation of ICMP error messages.
• Big Metadata
• LPM table support
• IP Fragmentation
• IP Reassembly on per table basis.
• Ability to go back to the tables whose ID is less than the current table ID.
• Ability to receive all pipe line fields via packet-in and sending them back via packet-out.
• Ability for controller to set the starting table ID along with the packet-out.
• Ability to define actions when the flow is created or bind object is created.
• Ability to define actions when the flow is  being deleted or bind object is being deleted.
• Connection template support to auto-create the flows within the virtual switches.

 Capacity

• Ability to support multiple network node switches - Minimum 32.
• Ability to support multiple routers -  256 per controller node,  that is, 256 name spaces per controller node.
• Ability to support 10K Firewall rules on per router.
• Ability to support 256 IPSec policy rules on per router.
• Ability to support 1K pools in LVS on per router basis.
• Ability to support 4M firewall/Load balancer sessions.
• Ability to support 100K IPSec SAs. (If you need to support mobile users coming in via from IPSec)

Performance

• 100K Connections or Connection templates/sec on per controller node basis.
• 10K IPSec SAs/sec on per controller node basis.

Traffic Steering 

Functionality

• Ability to support network service chains
• Ability to define multiple network services in a chain.
• Ability to define bypass rules - to bypass some services for various traffic types.
• Ability to associate multiple network service chains to a virtual network.
• Ability to define service chain selection rules - Select a service chain based on the the type of traffic.
• Ability to support multiple virtual networks.
• Ability to establish rules in virtual switches that are part of the chain.
• Ability to support scale-out of network services.

Capacity:

• Support for 4K virtual networks.
• Support for 8 network services in each chain.
• Support for 4K chains.
• Support for 32M flows.

Performance

• 256K Connections Or connection templates/sec.

Distributed Routing/Firewall/Load balancing for East-West traffic

As indicated before, virtual switches in the compute nodes should be used as data plane entity for these functions. As a controller, in addition to programming the flows to realize virtual networks and traffic steering capabilities,  it should also program flows to control the traffic based on firewall rules and direct the east-west traffic based on the routing information and load balancing decisions.

Functionality

• Ability to integrate with Openstack to get to know the routers, firewall & LB configurations.
• Ability to act as control plane/normal-path entity for firewall & LB (Similar to network node except that it programs the virtual switches of compute nodes).
• Ability to add routes in multiple virtual switches (Unlike in network node where the routes are added to only corresponding data plane switch).
• Ability to support many extensions (as specified in network node section).
• Ability to collect internal server load (For load balancing decisions).

 Capacity

•  Support for 512 virtual switches.
•  8M+ firewall/SLB entries.

 Performance

• 100K Connections/sec by one SDN controller node.

SDN Controller Manager

When there are multiple controller nodes in a cluster or multiple clusters of controllers,  I believe there is a need for a manager to manage these controller nodes.

Functionality

• Ability to on-board new clusters 
• Ability to on-board new controller nodes and assigning them to clusters.
• Ability to recognize virtual switches - Automatically wherever possible.  Where not possible, via on-boarding.
• Ability to associate virtual switches to controller nodes and ability  to inform controller nodes on the virtual switches that would be connected to it.
• Ability to schedule virtual switches to controller nodes based on controller node capabilities to take in more virtual switches.
• Ability to act as a bridge between Openstack Neutron & SDN controller nodes in synchronizing the resources & configuration of Neutron with all SDN controller nodes.  Configuration includes:
• Ports & Networks.
• Routers
• Firewall, SLB & IPSec VPN configuration.
• Ensuring that configuration in appropriate controller node is set to avoid any race conditions.
• Ability to set backup relations.

Securing the SDN Controller

Since SDN Controller is brain behind realization of virtual networks and network services, it is required that it is highly available and not prone to attacks. Some of the security features it should implement in my view are:

• Support SSL/TLS based OF connections.
• Accept connections only from authorized virtual switches.
• Always work with  backup controller.
• Synchronize state information with backup controller.
• DDOS Prevention 
• Enable Syn-Cookie mechanism.
• Enable host based Firewall
• Allow traffic that is of interest to SDN Controller. Drop all other traffic.
• Enable rate limiting of the traffic.
• Enable rate  limiting on the exception packets from virtual switches.
• Control number of flow setups/sec.
• Constant vulnerability asssesment.
• Running fragroute tools and isic tools to ensure that no known vulnerabilities are present.
• Always authenticate the configuration users.
• Provide higher priority to the configuration traffic.

Note: If one SDN controller node is implementing all the functions listed above,  it is required to combine all performance and capacity requirements.

Check SDN controller from Freescale consisting of comprehensive feature set,  takes advantage of multiple cores to provide very high performance system. Check the details here:  http://www.freescale.com/webapp/sps/site/prod_summary.jsp?code=VORTIQA_OND

Linux OVS & VxLAN based virtual networks

OVS (www.openvswitch.org) is the Openflow switch implementation in Linux. It implements various OF versions including 1.3.   OVS has support to realize virtual networks using VLAN and GRE for a long time.  In recent past,   OVS was enhanced to support overlay based virtual networks.  In this post, I give some commands that can be used to realize virtual networks using VxLAN.

For more information about VxLAN,  please see this tutorial.

Recently, there was a very good development in OVS on overlays.  It is no longer required to have as many 'vports' as number of compute servers to realize a virtual network across multiple compute servers.  OVS now implements the concept of flow based overlay protocol values selection.  Due to this, one VxLAN port is good enough in OVS OF switch irrespective number of remote compute nodes and irrespective of number of virtual networks.

OVS introduced new extensions (an action and set of OXM fields that can be set using set_field action)  to Openflow protocol where OF controller specifies the flow with tunnel/overlay specific information.

VxLAN protocol layer adds overlay header and it needs following information - Source IP address and Destination IP address of outer IP header,  source port and destination ports of UDP header and VNI for VxLAN header.  OVS provides facilities for Openflow controller to set the source IP, destination IP and VNI using set_field action.  OVS introduced following NXM fields

NXM_NX_TUN_ID :  To specify VNI (VxLAN Network Identifier).
NXM_NX_TUN_IPV4_SRC :  To specify source IP of the outer IP header.
NXM_NX_TUN_IPV4_DST :  To specify the destination IP of the outer IP header.

VxLAN protocol layer knows the UDP destination port from the 'vport'.   ovs-vsctl command can be used to create VxLAN ports.  ovs-vsctl command can be used to create many VxLAN ports on the same VNI with different destination port on each one of them.  VxLAN protocol layer gets rest of information required to frame outer IP, UDP headers by itself and with the help of Linux TCP/IP stack.

Similarly,  VxLAN protocol layer informs the OVS OF switches by filling up above fields after decapsulating the packets.  Due to this,  Openflow controller can use above fields as match fields.

Essentially, OVS provided mechansims to set the tunnel field values for outgoing packets in the Openflow flows and also provided mechanisms to use these tunnel fields as match fields in OF tables for incoming packets.

Following commands can be used to create VxLAN ports using 'ovs-vsctl' without explicitly mentioning the tunnel destination and tunnel ID, letting Openflow controller to specify these field values in OF flows.

Creation of VxLAN port with default UDP service port:

  ovs-vsctl add-port br-tun vxlan0 -- set Interface vxlan0   type=vxlan  options:remote_ip=flow options:key=flow

Above command is used to create VxLAN port 'vxlan0' on OF switch 'br-tun' and specifying this port to get the tunnel ID (VNI)  and tunnel remote IP from the OF flow.  "key=flow" is meant to get the tunnel ID from the flow and "remote_ip=flow" is meant to get the tunnel destination IP address from the flow.

Small variation of above command to create the VxLAN port with different UDP destination port, 5000.

ovs-vsctl add-port br-tun vxlan1 -- set Interface vxlan1 type=vxlan options:remote_ip=flow options:key=flow options:dst_port=5000

OVS provides a mechanism to create Openflow flows without having to have external Openflow controller.  'ovs-ofctl' is the machanism provided by OVS to do this.

Following command can be used to create the
ovs-ofctl add-flow br-tun "in_port=LOCAL actions=set_tunnel:1,set_field:172.16.2.3->tun_dst,output:1"  (OR)
ovs-ofctl add-flow br-tun "in_port=LOCAL actions=set_field:172.16.2.3->tun_dst, set_field:1->tun_id, output:1

"set_tunnel" is used to specify the VNI.  "set_field" to specify the tunnel destination.

Other commands of interest are:

To see the openflow port numbers:
        ovs-ofctl show br-tun
To dump flows:
        ovs-ofctl dump-flows br-tun

L4 Switching & IPSec Requirements & Openflow extensions

 

L4 Switching - Introduction

L4 switching typically involves connection tracking,  NAT and common attack checks.  Stateful inspection firewalls,  NAT, URL filtering  and SLB (Server Load Balancing) are some of the middle-box functions that take advantage of L4 switching.  These middle-box functions inspect first few packets of every connection (TCP/UDP/ICMP etc..) and offload the connection processing to some fast path entity to do L4 switching.  Normally both fastpath and normal path functionality reside in the same box/device.

Openflow, promoted by industry big names,  is one of the protocols that separates control plane from data plane where control planes of multiple switches implemented in one logical central controller and leaving the data plane alone at devices, which are programmable to work in a specified way,  thereby making devices simple.

Middle-box functions described above can also be separated - Normal path (service plane) and fast path (L4 switchiing).  Normal path works on first packet or at the most first few packets and fast path works on the connection for rest of the packets in the connection. By implementing normal path (Service plane) in centralized logical controllers and leaving the fastpath (L4 switching) at the physical/logical device level,  similar benefits of CP/DP separation can be achieved.  Benefits include

• Programmable devices where device personality can be changed by controller applications.  Now,  Openflow switch can be made as L2 switch, L3 switch and/or L4 switch.
• By removing major software logic (in this case, it is service plane) from device to the central location, cost of ownership goes down for end customers.
• By centralizing the service plane,  operation efficiency can be improved significantly
• Ease-of software upgrades
• Configure/Manage from a central location.
• Granular flow control.
• Visibility across all devices.
• Comprehensive traffic visualization

L4 Switching - Use cases:

• Branch office connectivity with corporate head quarters :   Instead of having firewall, URL filtering and Policy control on every branch office device,  it can be centralized at corporate office. First few packets of every connection could go to main office.  Main office controller decides on the connection. If the connection is allowed,  it lets the openflow switch in the branch office to forward rest of the traffic on that connection.  This method only requires simple openflow switches in the branch office and all intelligence can be centralized at one location, that is, at main office. Thus, it reduces the need for skilled administrator at every branch office.
• Server Load Balancing in Data Centers:   Today data-centers have expensive server load balancers to distribute the incoming connections to multiple servers.  SLB devices are big boxes today for two reasons -  Normal path processing that selects the best server for every new connection and  fastpath packet processing are combined into one box/device.   By offloading packet processing to inexpensive openflow switches,  SLB devices only need to worry about normal path processing, which could be done in lesser expensive boxes and even using commodity PC hardware.  
• Managed Service providers providing URL filtering capability to home & SME offices :  Many homes today use PC based software to do URL filtering  to provide safe internet experience to kids.  SME administrators require URL filtering service to increase the productivity of their employees by preventing them to access recreational sites and also to prevent any malware contamination.  Again, instead of deploying URL filtering service at customer office premises,   service providers like to host this service centrally and manage centrally across multiple of their customers for operational efficiency.  Openflow controller implementing URL filtering service can get hold of packets until URL is fetched, find the category of URL, apply policy and decide on whether to continue the connection.  If the connection is to  be continued, then it can program the openflow switch in customer premises to forward rest of the packets.  
• Though URL filtering service is one use case given,  this concept of centralizing the intelligence at one central place and programming the openflow switches for fast path is equally applicable for "Session Border Controller" and "Application Detection using DPI" and other services. Whichever service needs to inspect first few packets of the connection is candidate for Service Plane/L4 Switching  separation using Openflow.

L4 Switching - Processing

L4 switching or fastpath entity are used to do connection level processing. Any connection consists of two flows -  Client-to-Server (C-S) flow and Server-to-Client (S-C) flow.  

Connections are typically 5-tuple based.  

Functionality of L4 switching:

• Connection Management :  
• Acts on the connection Add/Remove/Query/Modify requests from the service plane.
• Inactivity timeout :  Activity is observed on the connection. If no activity for some time (programmed during the connection entry creation),   connection entry is removed and notification is sent to the normal path.
• TCP Termination : L4 switching entity can remove the connection entry if it observes the both endpoints of TCP connections exchange FINs and ACKed. It can also remove the entry if TCP packet with RST is seen.
• Periodic notifications of the state collected to Service plane.
•  Packet processing logic involves
• Pre-flow lookup actions typically performed:
• Packet integrity checks :  Checksum verification of both IP and transport headers,  Ensuring that the length field values are consistent with packet sizes etc..
• IP Reassembly:  As the flows are identified by 5-tuples, it is required that packets are reassembled to determine the flow across fragments. 
• IP Reassembly related attack checks and remediation
• Ping-of-Death attack checks (IP fragment Overrun) - Checking for full IP packet never exceeds 64K.
• Normalization of the IP fragments to remove overlaps to protect from teardrop related vulnerabilities.
• Too many overlap fragments - Remove the IP reassembly context when too many overlaps observed.
• Too many IP fragments  & Very small IP fragment size:  Drop the reassembly context and associated IP fragments.
• Incomplete IP fragments resulting Denial of Service attacks - Limit the number of reassembly contexts on per IP address pair,  IP address etc..
• TCP Syn flood protection using TCP Syn Cookie mechanism
• Flow & Connection match:  Find the flow entry and corresponding connection entry.
• Post lookup actions
• Network Address Translation :  Translation of SIP, DIP, SP, DP.
• Sequence number translation in TCP connections :  This is normally needed when the service plane updates the TCP payload that decreases or increases the size or when the TCP syn cookie processing is applied on the TCP connection.
• Delta Sequence number updates in TCP connections :  This is normally required to ensure that right sequence number translation is applied, especially for retransmitted TCP packets.
• Sequence number attack checks :  Ensuring that sequence numbers of the packets going in one direction of the connection are within some particular range. This is to ensure that any attacker injecting the traffic is not honored.  This check is mainly required to stop TCP RST packets that are generated and sent by attackers.  As RST packet terminates the connection and thereby creating DoS attack, it is required that this check is done.
• Forwarding action :  To send the packet out by referring to routing tables and ARP tables.

IPSec - Introduction

IPSec is used to secure the IP packets. IP packets are encrypted to avoid leakage of data on the wire and also authenticated to ensure that the packets are indeed sent by the trusted party.  It runs in two modes - Tunnel mode and transport mode.   IP packets are encapsulated in another IP packet in tunnel mode.  No additional IP header in involved in transport mode. IPSec applies the security on per packet basis - Hence it is datagram oriented protocol.  Multiple encryption and authentication algorithms can be used to secure the packets.   Encryption keys and authentication keys are established on per tunnel basis by the Internet Key Exchange Protocol (IKE).   Essentially,  IPSec protocol suite contains IKE protocol and IPSec-PP (Packet processing).  

Traditionally,  implementations use proprietary mechanism between IKE and IPSec-PP as both of them sit in the same box.  New way of networking (SDN) centralizes control plane with distributed data paths.  IPSec is one good candidate for this.  When the control plane (IKE) is separated from the data path(IPSec-PP),   then there is a need for some standardization for communication between IKE and IPSec-PP.  Openflow is thought to be one of the protocols to separate out control plane and data plane.   But Openflow as defined in OF 1.3.x specification did not keep IPSec CP-DP separation in mind.  This blog post tries to provide extensions required in OF specification to enable IPsec CP-DP separation.

Openflow

ONF (Open Networking Foundataion) is standardizing Openflow protocol.  Controllers using OF protocol programs the OF switches with flows and instructions/actions to be performed on packets.  Openflow 1.3+ specification supports multiple tables which simplifies the controller programming as each table in the switch can be used for a purpose.

Contrary to what everybody says,  Openflow protocol is not really friendly let alone to SP/FP separation, but also to L3 CP/DP separation.  The way I look at OF today is that it is good for L2 CP/DP separation and also good for traffic steering kind of use cases, but it is not sufficient for L3 and L4 switching.  This post tries to give some of the features that are required in OF specifications to support L4 switching in particular and L3 switching to some extent.

Extensions Required to realize L4 switching & IPSec CP-DP separation

Generic extensions:

Table selection on packet-out message:

Problem Description
Current OF 1.3 specification does not give provision for controllers to start the processing of packet-out message from a specific table in the switch.  Currently,  controller can send the packet-out message with set of actions.    These actions are expected to be executed in the switch.  Typically 'OUTPUT' action is expected to be specified by the controller in the action list.  This action sends the packets out on the port given in "OUTPUT" action.  One facility is provided to run through OF pipeline though.  If the port in 'OUTPUT' action is specified to be a reserved port OFPP_TABLE,  then the packet-out message execution starts from table 0.

Normally controllers takes advantage of multiple tables in the OF switch by dedicating tables to purposes.  For example, when L4 switching  with L3 forwarding are used, multiple tables are used - A table to classify packets,  A table for Traffic Policing entries,  A table for PBR and few tables for routing tables,  a table for L4 connections and a table for next hop entries and a table for traffic shaping rules.   Some tables are populated pro-actively by the OF controller , hence miss packets are not expected by controllers.  Entries in some tables are populated re-actively.  Tables such as 'L4 connections',  'Next hop' entries are created reactively by the controllers.   Processing of the miss packets (packet-in) can result into a flow being created in the table which was the cause of miss packet. Then controllers likes to send back the packet to the OF switch and would like OF switch to start processing the packet as OF switch does when there is a matching entry. Controller programming does not get complicated  if that was possible.  As indicated before,  due to limitations of the specification,  controller can't ask switch to start from a specific table.   Due to lack of this feature in the switches,   controllers are forced to figure out the actions that would need to be performed and then program the action list in the packet-out message.  In our view,  that is quite complex task for the controller programmers.  As I understand,  many controller applications simply using packet-in message to create the fow, but drop the packet-in message.  This is not good at all.  Think of SYN packet getting lost.  It would take some time for client to retransmit the TCP SYN packet and that delay results into very bad user experience.

Also, in few cases,  applications (CP protocols such as OSPF, BGP)  generate packets that needs to be sent out on the data network.  Controllers typically sit in management network, not on the data network.  Hence these packets are to be sent to the data network via OF switches.  This can only be achieved by sending these packets as packet-out messages to the switch.  Yet times, these packets need to be exposed to only part of the table pipeline.  In these cases, it would be required to have ability for controllers to start the packet processing in the switch for these packets from a specific table given by controller.

Solution

Freescale Extension :

struct ofp_action_fsl_experimenter_goto_table {
      uint16_t type; /* OFPAT_EXPERIMENTER. */
      uint16_t len; /* Length is a multiple of 8. */
      uint32_t experimenter; /** FSL Experimenter ID **/
      uint16_t fsl_action_type; /** OFPAT_FSL_GOTO_TABLE **/
      uint16_t table_id;
}
OFP_ASSERT(sizeof(struct ofp_action_fsl_experimenter_goto_table == 8);

table_id :  When switch encounters this action,  switch starts processing the packet from the table specified by 'table_id'.

Though this action is normally present in the action_list of e packet-out messages, this action can be used in flow_mod actions too.  When there is GOTO_TABLE instruction and GOTO_TABLE action,  then the GOTO_TABLE action takes precedence and GOTO_TABLE instruction is ignored.

Notice that 'table_id' size is 2 bytes.  This was intentional.  We believe that OF specifications in future would need to support more than 256 tables.

L2 Network Virtualization & Is there a role for Openflow controllers?

 

Introduction

 

Current Method of Network Virtualization

IaaS (Cloud Service Providers) providers do provide network isolation among their tenants. Even Enterprise private cloud operators are increasingly expected to provide network isolation among tenants - tenants being departments,  divisions,  test networks, lab networks etc..  This allows tenants to have their own IP addressing space and possibly overlapping with other tenants' address space.

Currently VLANs are used by network operators to create tenant specific networks.  Some of the issues related to VLAN are:

• VLAN IDs are limited to 4K.  If tenants require 4 networks each on average,   only 1K customers can be satisfied on a physical network.  Network operators are forced to create additional physical networks when more tenants sign up.
• Performance bottlenecks associated with the VLANs :  Even though many physical switches support 4K VLANs,  many physical switches don't provide line rate performance when the number of VLAN IDs go beyond certain limit ( some switches don't work well beyond 256 VLANs)
• VLAN based networks have operational headaches -  VLAN based network isolation requires  all L2 switches are configured when a new VLAN is created or an existing VLAN is deleted.  Though many L2 switch vendors provide central console to work with their brand L2 switches,  it is operational  difficulty when switches from multiple vendors are present.
• Loop convergence time is very high
• Extending VLANs across data center sites or extending VLANs to customer premise has operational issues with respect to interoperable protocols,  Out-of-band understanding among network operators is required to avoid VLAN ID collisions.

To avoid issues associated with  capabilities  of  L2 switches,  networks having switches from multiple vendors  and limitations associated with VLANs,  increasingly overlays are used to virtualize physical networks to create multiple logical networks.

Overlay based Network Virtualization

Any L2 network requires the preservation of L2 packet from source to destination.  Any broadcast packet should go to all network nodes attached to the L2 network.  All multicast packets should go to network nodes that are willing to receive multicast packets of groups of their choice.

Overlay based network virtualization provides above functionality by overlaying the Ethernet packets using outer IP packet - Essentially tunneling Ethernet packetsfrom one place to another.

VxLAN, NVGRE are two of the most popular overlay protocols that are being standardized.  Please see my blog post on VxLAN here.

VxLAN provides 24 bits of VNI (Virtual Network Identifier).  In theory around 16M virtual networks can be created.  Assuming that each tenant may have 4 networks on average,  in theory, 4M tenants can be supported by CSP using one physical network.  That is, there is no bottleneck with respect to identifier space. 

Openstack

Openstack is one of the popular open source cloud orchestration tools.  It is becoming formidable alternative to VMWare vCenter and VCD.   Many operators are using Openstack and  KVM hypervisor as a secondary source of cloud virtualization in their networks.  Reliability of Openstack came long way and many vendors are providing support for a fee. Due to these changes,  adoption of openstack+KVM is going up as a primary source of virtualization. Openstack mainly has four components -  'Nova' for VM management across multiple physical servers,  'Cinder' for storage management,  'Quantum' for network topology management and 'Horizon' to provide front end user experience to operators (administrators and tenants).

Quantum consists of set of plugins -  Core plugin and multiple extension plugins.  Quantum defines API for plugins and let various vendors to create backend for the plugins.  Quantum core plugin API defines the management API of virtual networks - Virtual networks can be created using VLAN,  GRE and being upgraded to support VxLAN too.

Quantum allows operators to create virtual networks.  As part of VM provisioning,  Openstack NOVA provides facility for operators to choose the virtual networks on which this VM needs to be placed on. 

When 'Nova scheduler' chooses a physical server to place the VM, it asks the quantum to provide MAC address, IP address and other information to be assigned to VM using 'create_port'  API.  Nova asks quantum as many times as number of virtual networks that VM belongs to.   Quanutm provides required information to NOVA back.  As part of this call,  Quantum comes to know about the physical server and the virtual networks that needs to be extended to the physical server.  It, then, informs the quantum agent (that sits in host Linux of each physical server) the virtual networks it needs to create.   Agent on the physical server gets more information on virtual networks from quantum and then create the needed resources.   Agent uses OVS (Open Virtual Switch ) package that is present in each physical server to do the job.  Please see some description of OVS below.   Quantum agent in each physical server creates two openflow bridges - integration bridge (br-int) and tunnel  bridge (br-tun).  Agent also connects south side of br-int to north side of br-tun  using loopback port pair.   Virtual network port creation and association to br-tun is done by quantum agent for every new virtual network or when the virtual network is deleted.  North side of br-int towards VMs is handed by libvirtd and associated drivers as part of VM management. See below.

Nova talks to 'nova-compute' package in the physical server to bring up/down VMs.  'Nova-compute' in the physical server uses 'libvirtd' package to bring up VMs,  create ports and associate them with openflow switches using OVS package.  Brief description of some of the work, libvirtd does with the help of OVS driver are:

• Creates a Linux bridge for each port that is associated with the VM.
• North side of this bridge is associated with VM Ethernet port (using tun/tap technology).
• Configures ebtables to provide isolation among the VMs.
• South side of this bridge is associated with Openflow integration bridge (br-int).  This is achieved by creating loopback port pair with one port attached to Linux bridge and another port associated with the Openflow switch, br-int.

Openvswitch (OVS)

OVS is openflow based switch implementation.  It is now part of Linux distribution.  Traditionally Linux bridges are used to provide virtual network functionality in KVM based host Linux.  In Linux 3.x kernels,  OVS has taken that responsibility and Linux bridge is used for the purposes of enabling 'ebtables'.
 

OVS provides set of utilities :  ovs-vsctl and ovs-ofctl.   "ovs-vsctl" utility is used by OVS quantum agent in physical servers to create openflow datapath entities (br-int, br-tun),   initialize Openflow tables and add both north and south bound ports to the br-int and br-tun.   "ovs-ofctl" is command line to create openflow flow entries in the openflow tables of br-int and br-tun.  It is used by OVS quantum agent to create default flow entries to enable typical L2 switching (802.1D) functionality.  Since OVS is openflow based,  external openflow controllers can manipulate the traffic forwarding by creating flows in br-int and br-tun.  Note that external controllers are required only to add 'redirect' functionality AND virtual switching functionality can be achieved even without external openflow controller. 

Just to outline various components in the physical server:

• OVS  package - Creates Openflow switches,  Openflow ports and associate them to various switches and ofcourse provides ability for external controllers to control the traffic to/from VMs to external physical networks.
• Quantum OVS Agent :  Communicates with the Quantum plugin in Openstack tool to get to know the virtual networks and configure OVS to realize those networks in the physical server.
• OVS Driver in Libvirtd :  Allows connecting VMs to virtual networks and configures 'ebtables' to provide isolation among VMs.

Current VLAN based Network Virtualization solution

Openstack and OVS together can create VLAN based networks.  L2 switching is happening with no external openflow controller.   OVS Quantum agent with the help of plugin knows the VMs, their vports and corresponding network ports. Using this information,  agent associates the VLAN ID to each vport connected to the VMs.  This information is used by OVS to know which VLAN to use when packets come from VMs.  Also,  agent creates one rule to do the LEARNING for packets coming in from the network. 

Overlay based Virtual Networks

Companies like Nicira and bigswitch networks are promoting overlay based virtual networks. OVS in each compute node (Edges of the physical network) is used as starting point of overlays.  All L2 and L3 switches connecting compute nodes are only used for transporting the tunneled packets.  They don't need to participate in the virtual networks.   Since OVS in compute nodes is encapsulating and decapsulating inner ethernet packets into/from another IP packet,  in-between switches transfer the packets using outer IP header addresses and outer MAC headers.  Essentially,  overlay tunnels start and end at compute nodes.  With this,  network operators can configure the switches in L3 mode instead of problematic L2 mode.  May be, in future, one might not see any L2 switches in the data center networks.

Typical packet flow would be something like this:

- A VM sends a packet and it lands on the OVS in the host Linux.

- OVS applies actions based on the matching flows in br-int and packet is sent to br-tun.
- OVS applies actions based on the matching flows in br-tun and packet is sent out on the port (overlay port)

- OVS sends the packet to overlay protocol layer.

- Overlay protocol layer encapsulates the packet and sends out the packet.

In reverse direction,  packet flow would look like this:

- Overlay protocol layer gets hold of the incoming packet.

- Decapsulates the packets and exposes the packet with right port to the OVS br-tun.
- After applying any actions on the packet using matching OF flows in br-tun,  packet is sent to br-int.
- OVS applies the actions on the matching flows and figures out the destination port (one-to-one mapping with VM port)

- OVS sends the inner packet to the VM.

Note that:

- Inner packet is only seen by OVS and VM.

- Physical switches only see encapsulated packet.

VxLAN based Overlay networks using Openstack

OVS and VxLAN:

There are many open source implementations of VxLAN in OVS and integration of this with openstack.   Some details about one VxLAN implementation in OVS.

• Creates as many  vports in OVS as number of VxLAN networks in the compute node.  Note that,  even though  there could be large number of VxLAN based overlay networks,  only networks to which local VMs belong are created in OVS as vports.  For example,  If there are VMs corresponding to two overlay networks,  then two vports are created. 
• VxLAN implementation depends on VTEP entries to find out the remote tunnel endpoint address for a destination MAC address of the packet received from the VMs.  IP address is used as DIP of the outer IP header.
• If there is no matching VTEP entries,  Multicast learning happens as per VxLAN.
• VTEP entries can be created manually too.   A separate command line utility is provided to create VTEP entries to vports.  
• Since Openstack has knowledge of VMs and physical servers they are hosting,  Openstack with the help of quantum agent in each compute node can create VTEP entries pro-actively.

Commercial Products

Openstack and OVS provide fantastic facilities to manage virtual networks using VLAN and overlay protocols.  Some commercial products seem to be doing following:

• Provide their own Quantum Plugin in the openstack.
• This plugin communicates with their central controller (OFCP/OVSDB and Openflow controllers).
• Central controller is used to communicate with OVS in physical servers to manage virtual networks and flows.

Essentially,  these commercial products are adding one more controller layer between Quantum in openstack and physical servers.

My views:

In my view it is not necessary.  Openstack, OVS,  OVS plugin, OVS agent, and  OVS libvirtd driver are becoming mature and there is no need for one more layer of abstraction.  It is a matter of time where these open source components would be feature rich, reliable and supported by vendors such as redhat.   With OVS being part of Linux distributions and with ubuntu providing all of above components,  operators are better of sticking with these components instead of going for proprietary software.

Since OVS is openflow based,  there could be Openflow controller to add value with respect to traffic steeering and traffic flow redirections.  It should provide value, but one should make sure that default configuration is good enough to realize virtual networks without need for openflow controller.

In summary, I believe that Openflow controllers are not required to manage virtual networks in physical servers, but are required to add value added services such as traffic steering,  traffic visualization etc..

Table Centric Processing and Openflow

Software in  Embedded network appliances consists of multiple entities - Management Plane (MP), Control Plane (CP) and Data Plane (DP).  Some examples of network appliances include routers, switches,  load balancers,  security devices and WAN Optimization devices etc.. 

MP entity typically consists of management engines such as CLI,  GUI Engines and Persistent storage Engines.

CP is mainly consists of several sub entities for different functions. For example, if a device consists of firewall,  Ipsec VPN and routing functionality,  then  there could be three sub-entities.  Each sub-entity might consists of multiple protocols and modules.  Routing function might have multiple control plane protocols such as OSPF, BGP, IS-IS etc..  CP also typically consists of exception packet processing modules.  For example, if there is no flow processing context found for a given packet in DP,  DP gives the packet to CP to process it and lets the CP to create the flow context in DP.

DP entity consists of actual packet processing logic. DP entity is typically implemented in Network Processor Units,  ASIC.  It is increasingly being implemented in Multicore processor SoCs with some cores dedicated for CP and rest of cores for DP. 

In this post, I am going to concentrate on DP (also called as datapath). 

As indicated DP implements the "datapath" processing.  Based on the type of network device, datapath consists of 'routing',  'bridging',  'nat', 'firewall', 'ipsec' and 'dpi'  modules and more.  Some background on each of the datapath functionality is listed below. Then we will see how table driven processing model helps in implementing datapaths.

Routing

Routing module typically involves following processing, upon packet reception:

• Parse and check integrity of the packet
• L2 (Ethernet, VLAN etc..)  header parsing.
• Identification of packet type 
• If it is IP,  proceed further, otherwise do non-IP specific processing.
• Integrity check involves ensuring the IP checksum is valid and packet size is at least as big as the 'total length' in the IP header.
• Extraction of fields - Source IP address,  Destination IP address,  ToS fields from the IP header.
• IP Spoofing Check:  This processing is done to ensure that the packet came from the right interface (Ethernet port).  Normally this is done by doing route lookup on source IP address.  Interface returned from the route lookup operation is compared with the port on which this packet came in. If both are same, then IP spoofing check is treated successful and further packet processing happens.  Basically, this check ensures that packet with destination IP address of current packet source IP address would go on the interface on which current packet came in. This requires "RouteLookup" operation using packet source IP address on the routing table which is populated by the CP.
• Determination of outgoing interface and next hop gateway:   This processing step involves "route lookup" operation on the "Destination IP" of the packet and optionally on TOS value of the packet.  "Route Lookup" operation indicates whether the packet needs to be consumed by local device (that is DIP is one of the local IP addresses) or whether the packet needs to be forwarded.  If the packet needs to be sent to the next hop, then the gateway IP address (or destination IP itself if the host is in the same subnet of the router) and outbound interface would be returned.   If the packet is meant to the local device itself, then packet is handed over to the CP. Rest of the processing assumes that packet is forwarded to the next hop.
• TTL Decrement :  At this processing step,  DP decrements the TTL and does incremental checksum update to the IP header checksum field.  If the TTL becomes 0, then the packet gets dropped.
• Packet Fragmentation:  If the packet size is more than PMTU value of the route or MTU of the outbound interface, then packets gets fragmented at this step.
• MAC Address Determination:  If the outbound interface is ETHRENT type  then this processing step  finds out the MAC address for the gateway IP address determined in one of the previous steps.  It refers to the "Address resolution table" populated by the CP.  It uses IP address as input and gets the MAC address.  If there is no matching entry,  packet is sent to the CP for ARP resolution.  In case the outbound interface is Point-to-Point interface,  L2 header can be found from the interface table which is populated by the CP.
• Packet update with L2 header:  DP frames the Layer 2 Header (Ethernet, VLAN, PPP etc..) and prepends in the beginning of the packet right before the IP header. 
• Packet Out:  Packet is sent out the outbound interface determined as part of "routelookup" operation.

At each step,  appropriate statistics counters are incremented. There are two types of statistics - Normal and Error statistics.  These statistics are typically  maintained globally,  on per interface basis,  on per routing entry or per ARP table entry.

Upon analysis of above steps,

IP routing datapath refers to few tables:

• IP Routing Database Table, which is LPM (Longest Prefix Match), for route lookup operation.
• MAC Address resolution table, which is exact match table,  to find MAC address for a given IP address.
• Interface Table, which index table, to find the L2 header to be used in case of  Point-to-Point interfaces.  Interface Index is typically returned by the "route lookup" operation.

IP routing in some devices also implement "Policy Based Routing" (PBR).  PBR is an ACL type of table with each entry identified by several fields, including IP header fields (SIP, DIP, TOS),  Input interface and output interface and even on the transport header fields.  The output of each ACL rule is the routing table.  Basically, "route lookup" functionality involves two steps - One finding the matching ACL rule in PBR table. This gets the routing table to use.  Then this routing table is searched with IP address and TOS to get the routing information as part of second step.  ACL rules can have field values represented in subnets, ranges and single value.   ACL is typically ordered list.

Bridging:  Bridging switches the packets without any modification to the packet.  Bridging is typically done among the Ethernet interfaces.  Bridging typically involves following steps upon receiving the packet from one of the Ethernet interfaces that belong to a bridge.

• Ethernet  Header Parsing: In this step,  Ethernet header and VLAN headers are extracted from the packet.
• Bridge Instance determination:  There could be multiple bridge instances supported by datapath.  This step determines the bridge instances based on the interface on which packet came in.  A given Ethernet or VLAN interface belongs to only one bridge.  CP populates the Interface table with entries, with each entry identified "Interface Index" and having parameters such as Bridge Instance Index (Index to bridge table),   whether this interface is allows learning,  forwarding etc.. 
• Learning Table Update:   Each Bridge maintains the learning table. It is also called FDB (Forwarding Database).  This table aims to provide Interface information given the MAC address.  This table is normally used by bridging datapath to determine the outbound interface for received packets.  But this table is updated with entries based on the incoming packet.  Its source MAC address and the interface on which packet came in used to populate the learning table. Basically,  it learns the machines on a physical network attached to the Ethernet port.   As part of this processing step,  if there is no learning table entry, then new entry gets added.  Note that learning is done if the Ethernet interface status in thhe "interface table" indicates that this interface can be used to learn the machines. In some systems,  the population of the learning table is done by the CP.  Whenever DP finds that there is no entry that matches the SMAC, then it sends the packet to the CP and CP creates the table entry.
• Determination of outbound interface:  In this step,  DP does the lookup on the FDB with DMAC address key.  If there is a matching entry, it knows the outbound interface on which the packet needs to be sent.  In case of Multicast packet, it refers to the Multicast FDB to find out the set of interfaces on which the packet need to sent out.  In case of broadcast packet (DMAC = ff:ff:ff:ff:ff:ff), then all interfaces in the bridge is selected to send the packet out.  Note that Multicast FDB is populated by the CP. CP does this by interpreting the IGMP, MLD and PIM-SM packets.
• Packet Out:  In this step, packet is sent out.  In case of unicast packet,  if the interface is known, packet is sent out.  In case the outbound interface is unknown (there is no matching FDB entry), then packet is sent out on all ports except the incoming port.   Packet is duplicated multiple times to achieve this.  Multicast packets are also sent on all ports if there is no matching  entry. Otherwise, it sends the packet to interfaces as indicated by Multicast FDB.  Broadcast packets are always sent to all ports in the bridge. Note that, in all these cases, packet is never sent on the interface on which packet came in. Also,  datapath does not send the packet if the interface is blocked for sending packet out.  This information is known from the interface table.

Bridging module maintains following tables.

• Unicast FDB, which is exact match table. Match field is 'MAC address'.  One FDB is maintained for each bridge instance.
• Mutlicast FDB, which  is also exact match table.  Match field is 'Multicast MAC Address'. One Multicast FDB is maintained for each bridge instance.
• Interface Index Table - Which is indexed by the incoming interface identifier. Typically global to the entire datapath.
• Bridge Instance Table - Which is indexed by bridge instance. Typically global to the entire datapath.

NAT/Firewall Datapath: NAT and Firewalls typically require Session Processing.  Session Management part of NAT/firewalls is typically implemented in datapath.  Sessions are typically 5-tuple sessions (Source IP, Destination IP, Protocol, Source Port and Destination Port) in case of TCP and UDP.  A Session is initiated by client and terminated by server.  NAT/Firewall devices being in between client and server machines,  it sees both sides of the connection - Client to Server and Server to Client.  Hence, in these devices,  sessions consists of two flows - C-to-S and S-to-C flows (Client-to-Server and Server-to-Client flows).   5-tuple values of C-to-S and S-to-C flow are same in case of firewall session except that values are toggled between source and destination.  That is, source IP of the C-S flow is used as destination IP of the S-C flow and destination IP of the C-S flow is used as source iP of the S-C flow. Same is true with source and destination ports.  In case of NAT,  this may not be as simple. NAT functionality translates client SIP, DIP, SP and DP to new values.  Hence the NAT session manager flows would have two different values in 5-tuples, but both belong to one session.  In summary,  a session maintained by NAT/firewall devices is pointed by two different flows.

Session Processing involves following steps in datapath:

• Packet Parsing and Integrity of headers (L2 header, IPv4/IPv6 header, TCP/UDP/ICMP/SCTP headers) :  Similar step as described in "Routing" section. 
• IP Reassembly:  Non-first IP fragments will not have transport header. Since session requires all 5-tuples for matching sessions,  if reassembly is not done, non-inital fragments will not be get matched onto the right flows. As part of the IP reassembly, many checks are made to check the anomalies and as part of this check fragments might get dropped.  Once the full transport packet is reassembled,  further processing happens.  Some of the checks that are done as part of this processing step are:
• Ensures that initial and middle fragments are at least of some configured size - This is to figure out any deliberate fragmentation by sender to make the device spend large number of CPU cycles there by creating DoS condition.
• Ensures that the total IP packet size never exceeds 64K size.
• Taking care of overlapping IP fragments.
• Ensuring that the data in the overlapped IP data is same. If not, it is considered as deliberate attempt and this processing steps drops these packets.
• Transport Header integrity checks:  At this step, transport header integrity is ensured such as
• Ensures that the packet holds transport header.
• Some datapaths may also verify transport checksum.
• Extraction of fields from the parsed headers :  Mainly SIP, DIP, P,SP and DP are extracted from network and transport headers.
• Security Zone lookup :  Every incoming interface belongs to one of the security zones.  CP programs the table of entries with each entry having Interface ID and security Zone ID it belongs to.  Security Zone ID is used in flow lookup function.
• Flow Lookup to get hold of session:  As discussed before, a session consists of two flows.  Flows are typically arranged in hash lists.  When the session is created by CP,  CP is expected to put two flows in the hash list  with each flow pointing to its session context.  Sessions are arranged in an array (Index Table).  Reference to session (array index) is put in the flow.   Basically,  there is one hash list for flows and Index table for sessions.   5-tuples and inbound security Zone ID are used to match the flow.  If there is no matching flow found,  the packet is sent to the CP.  If the flow is found,  then it goes onto next processing step.
• Anomaly Checks:  Once the flow, associated session context and complementary flow is determined,  then this processing step does checks to find out any anomalies using packet header contents and state variables that are maintained in the session.  Few things that come to my mind are: 
• Ensuring TCP sequence number of the current packets within reasonable range (Example: 128K) from the previous packet sequence.  This is done typically to ensure that MITM did not generate the TCP packet, mainly TCP packet with RST bit is set.   TCP RST packet can drop the session not only in the device, but also on the receiving client/server machine.
• Packet Modification:  This step involves packet modification.  Packets belongs to sessions that are created by firewall CP may not undergo many changes. In case of NAT, there are several modifications are possible based on session state created by CP.  Some of them are:
• Source NAT and Destination NAT modifies the SIP and DIP of the packet.
• NAPT may modify both SP and DP of TCP and UDP packets.
• To ensure that modified packet has unique IPID,  IPID also gets translated with unique IPID.
• TCP MSS value might be modified with lesser MSS value to avoid any fragmentation in future steps.
• TCP sequence numbers also may get translated. This is typically happens when CP does some packet processing before such as SYN FLOOD protection using SYN-COOKIE mechanism or due to some ALG processing.
• Packet Routing and Packet-out:  These steps are similar to IP Routing module.

 Upon analysis of above steps, one can find that there are following tables in Session Processing.

• Interface to Security Zone mappping table, which is an index table.
• Flow hash table, which is exact match hash table.
• Flow to Session ID table, which in index table.

Observations & Suggestions for new version of Openflow specifications:

All datapaths listed above use tables heavily.  Contexts created in each table are used by actions being performed.  It is not clear whether Openflow specification group is targeting to cover any type of datapath.  If so, few things to notice and improve. 

Packet Reinjection :  Each datapath might have more than one table.  Typically a table miss results into packet going to the control plane (controller in case of Openflow).  Control plane once acts on the packet and create appropriate context in the table,  it would like the packet processing in the datapath  to start from where the miss occurred.  Some times,  the processing should start from the table where the miss occurred or some times CP might do the operations on the packet and hence the packet processing needs to start in the datapath from the next table as specified in the newly created context.  Protocol between Control plane and datapath (Openflow) needs to provide this flexibility for control plane.

Atomic Update and Linkage of flows: Control plane yet times need to create multiple contexts in tables atomically.  For example, NAT/firewall control plane would need to create two contexts for two flows in atomic fashion.  Commit based facility would help in achieving this.  I think it is required to enhance openflow protocol to enable this operation.

Parsing headers & Extraction of fields:  Since there could be different datapaths that need to be exercised on a packet, one time parsing of the packet may not be sufficient. Each datapath or even each table might be preceded by parsing & extraction unit. There are different fields which are required in different paths.  There could be tunnel headers which need to be parsed for some datapaths to get to inner packet.  Each datapath might have its own requirement on which fields to be used for further processing.  That is, some datapath might need to work on IP header of tunnel and some other datapath might have requirement to use inner IP packet fields.  Hence I believe it is good to have parsing & extraction unit for each table. In cases, where the datapaths don't require different parsing & extraction units,  controller would not configure associated parsing & extraction units.  Some hardware devices (Multicore SoCs) I am familiar with support multiple parsing & extraction unit instances.  So, this is not a new thing for hardware ASIC/Multicore-SoCs. 

Pre Table lookup Actions:  Many times, there is a requirement to ensure that packet is good.  Packet integrity checks are very important during packet processing. Having some actions defined on per table basis (not on per flow basis) is good way to do these kinds of checks.  There may be a requirement in future to do even some modification to the packet before further parsing/extraction & eventual table lookup.  Pre-Table action list is one way to provide these facilities.  One can argue that, it is possible to use another table before packet is processed at current table and have context specific action on that table.  Though it is good thought,  it might result into inefficient usage of table resources in hardware.

Flexible Metadata:  Metadata is used to communicate the information among different tables and associated actions. There are two types of metadata required in datapaths - Some part of metadata is used as lookup action in further tables and some metadata is required by actions. Openflow specification defined the metadata, but it is limited to one word.  I believe that this is insufficient. Fixing the metadata size in standard is also not good. Controller can define the maximum metadata size required and program the size information in the datapath.  Datapath can allocate additional space to store metadata on per packet basis using the size information programmed by the controller.  Since there could be multiple types of datapaths and multiple tables within each type of datapath,  controller may categorize metadata for different purposes. Each metadata field and field size can be referred later on by table for lookup and associated actions.  I believe that openflow should provide that kind of flexibility - Setup the size of metadata required one time in the datapath,  language to use different fields in metadata for table lookup,  language to define action which can use different fields in the metadata or set/reset some fields in the metadata.

Actions :  Each type of datapath has its own actions - For example firewall session management datapath actions are different from the type of actions routing or bridging require.  Hence, there would be good number of actions eventually in the datapath.  And more actions would be required in future.  In addition to defining more actions in the openflow or associated specifications,  it should also provide flexibility to add new actions without having to create new hardware.  That is, openflow might need to mandate that the datapath hardware or software should be able to take newer actions in future.  To enable this,  it is required to define some kind of language to define new actions.  Since datapaths may be implemented in hardware ASIC,  the language should not have too many constructs/machine-codes.  Simple language is what is needed.  I thought simplified LLVM can be one option.

Embrane - Is this SDN Play?

Recently, I came across a company called Embrane while doing some google search on  SDN.  Then I saw a press release that Embrane made a product release announcement. I thought I would check this out and see how far it goes in SDN.  I had gone through the whitepaper published in Embrane website.  If you are interested,  you can find that paper here.

My understanding of Embrane solution:

When I first read the white paper, I was not sure about  Embrane product - Whether it is a platform/framework  to instantiate any type of network service virtual appliances from any vendor or whether the Embrane provides some network services as virtual appliances. By end of reading the whitepaper and after going through their website, it appears that Embrane's main focus is to deliver the framework for any virtual network service appliances including third party virtual appliances.

Embrane architecture mainly consists of four components.  Each component is installed as separate VMs.

• Elastic Service Manage (ELM)r:  Typically, there would be one VM of this type. Data Center operator works on this VM to provision Distributed Virtual Appliances. 
• Distributed Virtual Appliances (DVA):  Each DVA is logical set of VMs.  There are three kinds of VMs within each DVA.   Even though, there are multiple VMs within one logical DVA,  it can be treated as one appliance for all practical purposes.  As I understand,  Data Center operator need to instantiate as many DVAs as number of tenants.   If there are two types of network services is required, then there would be 2 DVAs.   So, if there are X number of tenants in a Data Center and each tenant requires Y network services (ADC,  Firewall,  Web Application Firewall, WAN Optimization etc..), then there would be a need for X * Y DVAs.    Now, coming to three kinds of VMs within each DVA.
• Network Service Virtual Appliances (NSVA):  DVA can have multiple virtual appliances.  These appliances implement actual functionality of network service such as ADC,  Firewall, WOC etc.  Obviously, there must be atleast one network service VA in a DVA.  Multiple VAs can be instantiated by ELM for scaling performance (Scale-out). 
• Data Plane Dispatcher (DPD):   There will be 1 DPD in each DVA.  DPD is the one which actually distributes the traffic across multiple NSVAs  for linear performance scaling.  
• Data Plane Manager (DPM):   One DPM VM in each DVA.  DPM is expected configure the NSVAs and DPD in the DVA on behalf of ESM.  Though it is not clear, I am assuming that this will ensure that the configuration integrity is maintained across all NSVAs.   It appears that this is the only VM that requires persistent storage and hence I am guessing that it might be storing the audit and system logs generated by DPD and NSVAs in persistent memory.

If the network service is ADC (Application Delivery Controller),  then it can be viewed that DVA provides one more level of Load balancing.  That is, DPD acts as Load Balnacer to multiple ADCs.  As we know, ADC itself acts as a load balancer to servers.  This makes sense as ADCs have become complex in recent past and computations power requirements have gone up.  Hence, one more layer of load-balancing is indeed required.  In current Data Center deployments, this is achieved using L2 switches.  L2 switches have capability to balance the load across multiple external devices based on hash result of defined fields in L2/L3 and L4 headers.

I have detailed out how L2 switches can be used to distribute the traffic across multiple devices of a cluster.  Please check that out here.

My views:

It appears that DPD functionality is similar to what I described in my earlier post.
Since DPD is a software based distributor,  I expect that it will not have limitations of  L2 switch based load distribution. As we all know that many network services work with sessions (typically 5 tuple based - SIP, DIP, P, SP, DP) to store the state across the packets. Any load distributor is expected to take care of this by sending packets corresponding to a session to only one device in the cluster.   If this is not done, there would be lot of communication across the devices (Virtual appliances) within the cluster.  This may eliminate the benefit of multiple virtual appliances in the cluster.   In my view,  DPD should be distributing the sessions (not the packets blindly) across multiple Virtual appliances.  Since it is software based solution,  it can do one more step and ensure that all sessions corresponding to application sessions are sent to the same virtual appliance.  VOIP based on SIP is one example where there can be 3 UDP sessions corresponding to one application session.  DPD kind of devices need to ensure that the traffic corresponding to all three sessions in this example are sent to one device (Virtual Appliance).  Detection of 5-tuples of data connections is only possible if DPD supports ALGs (Application Level Gateways).  Since there could be more ALG requirements in future,  the challenge is to provide these ALGs on constant fashion by the "Load Distributor" vendor and/or open up DPP architecture for third party vendors to install their own ALGs, thereby maintaining SDN spirit. 

As a described in the same earlier post,  configuration synchronization among the network service devices (NSVAs) is one important aspect of cluster based systems.  I guess DPM is the one which is taking care of it in Embrane solution.

Overall this architecture is good and replicating the physical solution into cloud solution.  It is good for environments where Data Center operators don't allow physical appliances to be deployed by their customers.

It does not appear to be Openflow based. But it can be still considered as part of SDN as it allows third party network service virtual appliances in their framework.


Challenges I see in Embrane solution:

Embrane might be having following features already.   Since I did not find any information on this, I thought it is worthwhile mentioning. I believe that following features are required in DPD kind of load distributors.

As I described above,  classifying the packets across multiple application sessions and ensuring that all packets corresponding to an application session go to the same network service virtual appliance is one big challenge for these kinds of equipment.  I know this personally and it is quite challenging to support multiple ALGs, mainly ensuring interoperability with both clients and servers. 

Some network deployments might see traffic on tunnels such as IP-in-IP, GRE, GTP-U etc..  Traffic corresponding to many sessions is sent on very few tunnels.  To ensure the distribution across multiple NVSAs,   "Load Distributor" need to have flexibility to dig deep into the tunnels and classify the packets based on inner packets.

Performance, I believe, would be the biggest challenge in virtual machine based 'Load Distributor". Classifying the packets, session load distribution and sending subsequent packets of sessions to the selected virtual appliance requires maintenance of millions of  sessions in the "Load Distributors".  What I hear is that one CPU based Virtual machines using VMWare and XEN kind of hypervisors give typically 1Gbps of performance for small size packets.  More processors can be added to the "Load Distributor" virtual machine, but achieving performance in 10s of Gigs may be very challenging.

My 2 Cents:

I believe that the "Load Distributors" need to go beyond virtual machines. Taking advantage of Openflow based switches to forward the packets would be the solution of choice in my view.  Load Distributor virtual machine can do the ALG kind of functionality, selection of Network Service Device for new sessions and create appropriate flows in Openflow switches and leave Openflow switches  to forward subsequent packets  to network service devices/virtual-appliances.  That is, openflow switch can forward the packets to "Load Distributor" if there is no matching flow.   Hence the traffic to Load Distributor is small and one virtual machine would be able to process the load.  Since Data Center operators normally have switches (eventually openflow switches),  this mechanism just work fine in Cloud environments.