DPDK based Accelerated vSwitch Performance analysis

DPDK based OVS acceleration improves the performance many folds up to 17x times over native OVS.   Please see the link here :  https://download.01.org/packet-processing/ONPS1.5/Intel_ONP_Server_Release_1.5_Performance_Test_Report_Rev1.2.pdf

It is interesting to know following :

• Performance impact with increasing number of flows.
• Performance impact when additional functions introduced.
• Performance impact when VMs are involved.

Performance report in 01.org has measured values with many variations.  To simplify the analysis, I am taking measured values with one core and hyper-threading disabled.  Also, I am taking numbers with 64 byte packets as PPS number is important for many networking workloads.

Performance impact with increasing number of flows:

Scenario :  Phy-OVS-Phy (Packets received from Ethernet ports are processed by the host Linux OVS and then sent out.  There is no VM involvement.  Following snippet is taken from Table 7-12.

One can observe from above table is that performance numbers are going  down with increasing number of flows, rather, dramatically.  Performance went down by 35.8% when the flows are increased from 4 to 4K. Performance went down  by 45% when the flows are increased from 4 to 8K.
Cloud and telco deployments typically can have large number of flows due to SFC and per-port filtering functionality.  It is quite possible that one might have 128K number of flows.  One thing to note is that the performance degradation is not very high from 4K to 8K. That is understandable that cache thrashing effect is constant after some number of simultaneous flows.  I am guessing that performance degradation with 128K flows would be around 50%.

Performance impact when additional functions introduced

Many deployments are going to have not only VLAN based switching, but also other functions such as VxLAN,  Secured VxLAN (VxLAN-o-Ipsec) and packet filtering (implemented using IPTables in case of KVM/QEMU based hypervisor).  There are no performance numbers in the performance report with VxLAN-o-Ipsec and IPTables, but it has performance numbers with VxLAN based networking.

Following snippet was taken from Table 7-26.  This tables shows PPS (packets per second) numbers when OVS DPDK is used with VxLAN.  Also, these numbers are with one core and hyper-threading disabled.

Without VxLAN,  performance numbers with 4 flows is 16,766,108 PPS.  With VxLAN, PPS number dropped to 7.347,179, a 56% degradation.

There are no numbers in the report for 4K and 8K flows with VxLAN.  That number would have helped to understand the performance degradation with both increasing number of flows & with additional functions.

Performance impact when VMs are involved

Table 7-12 of the performance report shows the PPS values with no VM involved.  Following snippet shows the performance measured when the VM is involved.  Packet flows through Phy-OVS-VM-OVS-Phy components.  Note that VM is running on a separate core and hence VM-level packet processing should not impact the performance.

Table 7-17 shows the PPS value for 64 byte packets with 4 flows.  PPS number is 4,796,202
PPS number with 4 flows without VM involvement is 16,766,108 (as shown in Table 7-12).
Performance degradation is around :  71%.

PPS value with 4K flows with VM involved (from Table 7- 21):  3,069,777
PPS value with 4K flows with no VM involved (from Table 7-12) : 10,781,154
Performance degradation is around : 71%.

Let us see the performance impact with the combination of VM involvement & increasing number of flows -

PPS number with 4 flows without VM involvement is 16,766,108 (as shown in Table 7-12).
PPS numbers with 4K flows with VM involvement is 3,069,777 (from Table 7-21)
Performance degradation is around :  81%.

Observations:

• Performance degradation is steep when the packets are handed over to the VM. One big reason being that the packet is traversed through the DPDK accelerated OVS twice unlike Phy-OVS-Phy case where the OVS sees the packet only once.  Second biggest possible culprit is   the virtio-Ethernet (vhost-user) processing in the host.
• Performance degradation is also steep when number of flows are increased. I guess cache thrashing is the culprit here. 
• Performance degradation when more functions are used.

Having said that,  main thing to note is that,  DPDK based OVS acceleration in all cases is showing almost 7 to 17x improvement over native OVS.  That is very impressive indeed.

Comments?

Death of SRIOV and Emergence of virtio

SRIOV functionality  in PCIe devices is introduced to solve a problem of sharing a physical device across multiple virtual machines in a physical server. SRIOV functionality in PCIe devices enables creation of  multiple virtual functions (VFs), typically to assign a virtual function to one virtual machine. Since each VF can be memory mapped,  virtual machines can have direct access to the device, thereby bypassing the VMM (Hypervisor).

NIC, Crypto, Compression and PCRE accelerator vendors have enhanced their PCIe devices to support Virtual Functions (VF).

It worked for some time.  Soon, with the popularity of public clouds and private clouds,  supply chain equation was changed.  It is no longer one or few vendors provide a complete solution to the operators.  Since few vendors got together to provide complete solution to the end operators, it was possible for  VM image vendors to support small number of PCIe NIC and accelerators by adding drivers to their VM images.   Soon after, operators started to procure various components that makes a system from various vendors themselves.  Operators start to get physical servers, accelerators, NICs and virtual machine images from various vendors.  Operators found this model working for them as the cost savings are huge with relatively small addition of integration by themselves.

In this new model,  virtual machine image vendors don't know what kind of accelerators and NIC cards supported by the operators.  Operators might procure newer accelerator and NIC cards in future from a new vendor.  If virtual machine is built for certain SRIOV NIC cards,  that image may not be valid in future if operators procure some other NIC card from a new vendor.  That kind of dependency is not good for all parties - Operators, NIC/Accelerator companies, virtual machine image companies.

One more major change happened in the Industry.  Operators are not just happy with simple demux of the packets based on VLAN ID or MAC ID to determine the VM destination for the incoming packets. Operators wanted to have control over the packet flows for various reasons such as :

• Service Function Chaining.
• Traffic Probe 
• Traffic Control
• Traffic Policing 
• Filtering 
• Overlays
• Secure Overlays 
• And many more...

Many NIC cards have simpler mechanisms such as VLAN, MAC level demux.  Since operators need lot of  flexibility,  they started to avoid assignment of VFs to VMs.   SDN and OVS have born out of that need. OVS running in VMM allowed operators to implement above functions without having to depend on the NIC vendors.  

These two reasons -  A. Removal of dependencies among VM image, NIC vendors and operators B.  operator need for controlling the traffic - is slowly killing SRIOV for NICs.

At least one reason - Removal of dependencies among VM image, PCI device vendors and operators - is also killing  SRIOV based accelerator cards.

Virtio:

virtio is a virtulaization standard for Ethernet, Disk and other accelerators.  Virtio in the VMM emulates devices as PCIe devices (emulated PCIe devices) and assign them to the guests. Guest when it comes up discovers these devices and enables them by loading appropriate virtio frontend drivers. Since these devices are emulated by the VMM in software, in theory, there are no reasonable limits on the number of devices that can be emulated and assigned to the guests.

In today world, virtio is used for emulated Ethernet, SCSI and other devices.  Since, this emulation is same irrespective of physical devices,  this naturally provides a flexibility where VM images are independent of physical NICs.  That became more attractive for VM image vendors.  As long as they support virtio front end drivers,  they can be rest assured that they can be run on any physical servers with any NIC or SCSI physical devices. 

If I take NIC as an example,  VMM receives the packets from the physical NIC,  runs through the various functions such as OVS, Linux IPTables etc.. and then sends the packet to the right guest using virtio emulated Ethernet device. In this case, only VMM need to have drivers for the physical NIC. All VM images only need to worry about virtio-Ethernet.

Next step in Evolution 

Though there is a freedom for VM images from physical devices,  but soon,  operators start to find the problems with the performance.  Since packets are traversing through VMM in both directions (ingress and egress),  major performance drops are observed.

That gave the birth to smart-NICs where smart-NICs do almost all VMM packet processing.  To enable flexibility of packet processing enhancements, most of smart-NIC started to be built using programmable entities  (FPGA or Network processors etc..). In addition,  smart-NIC vendors have also started  implement virtio-Ethernet interface to connect to VMs without involving VMM. With this, smart-NIC vendors are solving two issues - Performance issue where the device is directly connected to the guests and VM image independence from hardware. Since virtio-Ethernet is exposed by smart-NIC,  VMs need not know the difference between VMM emulating virtio-Ethernet and smart-NIC emulating virtio-Ethernet.

With great success in Ethernet,  virtio based interface is being extended in other hardware devices such as iSCSI,  Crypto/compression/regex accelerators. 

SRIOV death just began and in few years I won't be surprised if nobody talks about SRIOV.

Comments?