Data Center Equipment & Offload card requirements

I  have been attending some web conferences focusing on Data Center equipment.  From last few years,  there are more and more discussions on offloading some processing out of current data center equipment. Before getting details on offload functions,  it is good to revisit some of data center equipment.

Data centers, whether they are private, public or cloud,  have following equipment.

• L2/L3 Switches and Routers
• Server Load balancers or Application Delivery controllers.
• WAN Optimization devices
• Network Security Devices.
• Monitoring Servers for network visibility, Surveillance and others.
• Application Servers (Web, EMAIL etc..)
• Database Servers
• Storage Devices.

At this time, all servers are x86 servers.   I gather that, many of the network equipment such as SLB, ADCs, WOC, UTMs are also implemented on x86 servers with Crypto, Compressions and Pattern matching done on add-on PCIe cards.  Some vendors of these equipment are seriously considering moving to embedded multicore SoCs.   There are two camps.  One camp of vendors want to continue with x86 for their applications, but use embedded Multicore SoCs for offloading some functions beyond algorithmic offloads.  Other camp of vendors would like to replace multiple x86 processors with multiple Embedded Multicore SoCs.   

This post talks about networking functions that equipment vendors looking to see in offload cards. There are two types of offload cards that are becoming popular - PCIe cards that can go into x86 based network equipment and  proprietary cards that go into L2/L3 switch. Both kinds of cards are expected to take away some processing out of networking equipment. If it is PCIe card, more offload functions are possible due to proximity with the target application.  In this post,  I am going to list down the offload functions that are possible and expected by the networking equipment vendors.

PCIe offload cards act NIC card to the x86 equipment while doing offloading of some networking functions.  Hence some call them  intelligent NIC cards. Offload card receives all the packets from the network, it processes offload functions and then sends the result packets to the server via PCIe interface. These packets are presented to the server TCP/IP stack via PCIe Ethernet driver.  Similarly outbound packets would come to offload card from the server.  Offload card does offload functions necessary on egress traffic and sends the result packets out on the network.

List of offload functions in different categories are given below.

Algorithmic Acceleration:  This is already being done and requires no further explanation.  Algorithmic acceleration now typically involves

• Public Crypto and Symmetric Crypto acceleration
• Symmetric Crypto acceleration with Protocol offload such as SSL Record layer : Though this is provided by offline cards,  increasingly network applications requiring SSL are using AES-NI SIMD functions provided Intel SSE4.  Many PCIe based crypto accelerators are mainly used for public crypto acceleration.
• Compression/Decompression  acceleration.
• Regular expression Search acceleration.

Basic Networking offloads functions:

• Offloading of ARP for IPv4 and ICMP based neighbor discover in case of IPv6:  As networks in the data center becoming flatter, there is more and more ARP broadcast traffic in the network.  TCP/IP stack of network equipment receives all broadcast packets.  Due to increased broadcast traffic,  CPU cycles are being spent in processing interrupts, and ARP processing.  Lot of times, ARP requests may not even be satisfied as there is no neighborhood entry.  Offload cards are expected to remove this overhead by responding to neighborhood requests by itself without involving the server TCP/IP stack.  How does it work?
• Server TCP/IP stack is expected to create/delete neighborhood entries with local IP addresses and associated MAC addresses  in the offload cards whenever new entries are created locally and whenever they are deleted.  Offload card maintains the neighborhood entries database on per physical and logic port basis (in case of VLAN based interface).
• When offload card receives the neighborhood request, it checks  its database first. If there is no matching cache entry, ARP packet is dropped.  If there is any matching entry, it processes the request and reply is sent to the network without involving server.  
• Offloading of packet integrity checks and dropping of invalid inbound packets:  Any TCP/IP stack including servers TCP/IP stack spend some CPU cycles checking whether the packet received is good.  These cycles can be saved if offload cards do the checks.  Some of the checks that offload card can do are:
• Check the SMAC of the packet :  If the source MAC is one of the local MAC addresses, then packet can be dropped by offload engine.
• Check the IP length with the packet size:  If the IP length is more than the packet size,  it is a bad packet and packet can be dropped.  If the IP length in header is less than the packet size, then packet needs to be truncated to length in IP header.
• Check the checksum of IP packet:  If the checksum  is bad, then offload card can drop the packet.
• Check the TTL:  If TTL is 0, then the packet can be dropped by offload engine.
• Check the SIP and DIP of IP header:  If both the addresses are same, then packet can be dropped by offload card.
• Check the SIP of IP header with local IP addresses:  If the source IP is one of local server IP addresses, then the packet can be dropped by offload card. Similarly, if the source IP addressis one of the local broadcast addresses or multicast address, packet can be dropped by the offload card.
• Transport header length checks:  If the length of the packet is not as per the length in the transport header, then the packet can be dropped by the offload card.
• Transport header checksum checks:  If the checksum of the packet does not match with checksum in the header, packet can be dropped by the offload card.
• When the checksums are verified,  to avoid similar checks in the server TCP/IP stack,  offload card can communicate the successful checksum verification indication along with each packet to the server TCP/IP stack.  Many server TCP/IP stacks such as Linux has capability to indicate this information in their buffers.  For example, Linux TCP/IP stack has capability to avoid checksum verification if some special fields are set in SKB.
• Offloading of IP reassembly on inbound packets:   Offload Engines by doing IP reassembly and sending one single packet to the server TCP/IP stack can save good number of CPU cycles. Any server terminating any kind of tunnel (such as IPsec, GTP, GRE, IPinIP etc..) would tremendously would benefit from this offload.  Offload cards are expected to do reassembly of IP packets that involve two packets at the minimum as many tunnel protocols make two fragments from the IP packet. IP reassembly function is not only required to save some CPU cycles in server, but also required to do other offloads such as 'Traffic policing',  'stateless and stateful filtering',  'inline DPI offload'  as they require access to transport selectors.  These offloads will not work well on packets if they don't have transport selectors.
• Offloading of IP fragmentation on outbound packets:  This offload again saves CPU cycles by moving this function from server TCP/IP stack to offload engine on outbound packets from the server.  Server TCP/IP stack can give PMTU value along with the packet.  Offload card can fragment the packet as per PMTU and send them out. By offloading this functionality to offload card,  offload card can do better job of traffic shaping if it involves transport selectors.  IP Fragmentation can be done after the traffic shaping there by allowing traffic shaper to look at full IP packet.
• Offloading of hash calculation  on inbound packets:  Many networking applications running in x86 server require hash result to index into  hash buckets of their session blocks.  Bob Jenkins,  CRC-32 and CRC-64 are some of common hash algorithms used on packet fields.  Offload cards are expected to take configuration from x86 server networking applications.  Configuration include 'hash algorithm', 'packet fields' and their order. Offload cards, then expected to, extract the fields, do the hash on the fields and send the hash result along with the packet to the server TCP/IP stack. 
• Packet parsing & Field extraction on inbound packets :  Almost all networking application require to extract some common fields from the packet.  Cycles required to extract the fields can be saved by offloading this part to the offload cards.  Server networking applications can program the offload card on the fields it is interested in.  Offload cards based on configuration extracts relevant fields and provide them along with the received packet.

Inbound Packet filtering

Non-security network equipment devices would normally service certain protocols.  Unrelated traffic that goes to the devices would only decrease the performance of the device. Offload cards can filter out the packets.  Device software can program the rules to allow only the traffic required to go to the devices. Offload cards can reject the traffic that does not match the rules programmed.

Traffic Policing of inbound packets

Network equipment devices run complex operations on the packets or data. For example, ADC/WOC/Security devices do SSL, TCP, HTTP and other application protocol proxying to act on the data.  Proxy operations are expensive in nature.  Even the best of equipment can't process more than few gbps. If more traffic is sent towards the device, packets would be dropped at some point and  it only worsens the performance of device.  Devices are typically rated for some performance for different types of traffic workloads.  Offloads cards are expected to provide traffic policing on different types of traffic and discard the packets that exceed the policing parameters.  Device software is expected to program the offload cards with different types of protocols and traffic policing parameters.

TCP stateless offload :

Based on some conversations with DC equipment vendors as well as offload card vendors,  it appears that majority of them don't like to have TCP termination in offload cards and corresponding TCP/IP stack bypass in the x86 device.  It appears that device vendors are happy with device Linux or NetBSD operating system's TCP/IP stack.  Only peripheral changes to the TCP/IP stack and new Ethernet driver is only thing allowed by these vendors due to offload cards.  Maturity of Linux TCP/IP stack,  constant upgrade of TCP/IP stack with latest standards (RFCs),  avoiding application porting or testing of system are some of the reasons cited.  

Due to this, offload cards are expected to provide only TCP stateless offload. TCP stateless offload functions increase the performance of proxy applications.

• Offload of hash function :  This is described above under 'Basic networking offload'.  TCP layer of device TCP/IP stack uses hash function to find the TCP control block (TCB).  If offload card does this, then device TCP/IP stack can skip this and save some CPU cycles.
• Large Receive Offload :  As described in the post "Linux TCP Large Receive Offload optimization to increase performance" LRO feature reduces the number of packets that are seen by the device TCP/IP stack. Lesser the number of packets, better the performance of TCP/IP stack. Offload cards can do this by implementing LRO function by aggregating multiple consecutive TCP packets of a flow of a connection into a single large packet.
• Transmit Segmentation offload :  Similar to LRO, this feature reduces the number of packets that traverse the device TCP/IP stack, but this time on the transmit packets.  Device TCP stack normally segments the packets based on MSS which is PMTU-TCPheader length - IP headerlength.   This would increase the number of packets on transmit direction and also some cycles are spent in segmenting the big data.  TSO functionality in offload cards can reduce the number of packets and also save CPU cycles from segmenting the packet in the device.  Device TCP stack can provide the big packet along with segment size and PMTU to the offload card. Offload card along with TSO and IP fragmentation functionality can segment and if required fragment the packets before sending them out on to the network.
• ACK offload : As we know,  TCP/IP stacks are expected to send at least one ACK for every two TCP segments it receives.  Hence, there would be large number of ACKs.  Similar to TSO,  reducing the number of ACKs traversing the device TCP/IP stack can increase the performance of device.  Device TCP/IP stack generates the template ACK with the number of ACKs to be generated and corresponding sequence/ACK number information to the offload card.  Offload card, then, can generate multiple ACKs from the template ACK.

Traffic Shaping of outbound packets

Many network equipment device types do some sort of traffic shaping on the outbound packets.  Traffic shaping is almost the last step on the outbound packets.  QoS or traffic shaping is quite the same in all devices and there is really not much a value addition by the device software.  Hence this function can be done by the offload cards without impacting any device functionality.  Traffic shaping policies can be programmed in the offload cards by the device software.  Offload cards can act on the outbound packets based on the configuration.  It is important to that offload cards support multiple scheduling algorithms and hierarchical shaping.  This function in offload cards can save good number of CPU cycles and hence more CPU power can be made available to the device applications.

IPsec termination offload

Some network equipment devices such as network security devices implement IPsec VPN.  For inbound packets,  IPsec is done in the early stages of packet processing. For outbound packets, it is done one step before traffic shaping.  Hence this is also good candidate for offloading along with the traffic shaping.  By offloading IPsec inbound SA processing,  device software only sees the clear packets.  Similarly,  by offloading Ipsec outbound SA processing, device software does not need to do the expensive IPsec outbound processing.   Device software is expected to program the tunnels (pairs of Inbound and outbound SAs) upon IKE negotiation and delete them when life time expires or when there explicit delete indication from the remote gateway.

Offload cards are expected to provide IPsec termination offload function.  Offload cards not only do outbound and inbound SA processing, they also expected to do inbound policy verification on both decapsulated inbound packets and also on the original clear packets.  On the outbound side,  offload function can expect device software to provide outbound SA reference along with the packet. 

Inline DPI offload on inbound packets

Network equipment devices having IDS, IPS and Application detection functionality require some kind of pattern match on the packets.  These applications analyze the TCP, UDP, IP payloads for some detections and for others it may require processed data of L7 protocols.  Offload cards have access to TCP, UDP and IP payloads and may not have L7 intelligence.  At least for TCP, UDP and IP payload based detection, offload cards can be used.  Device software programs the patterns in the offload card and its regular expression engine.  Offload card after doing all its operations on inbound packets such as 'IP Reassembly', 'LRO', 'Packet filtering', it does pattern match on the payload and sends the results along with the packet.  Each result is expected to have pattern ID, offset in the payload.  Device software can avoid pattern matching on basic protocol payloads and there by utilizing saved CPU cycles on other processing.

Multicore Distribution

Network equipment devices normally use multicore x86 processors.  Without any support of offload cards,  distribution of packets to multiple cores happen either via software based interrupt distribution or by affining interrupts to different cores.   Some NICs have capability of distributing the load across multiple large number of descriptor rings with each descriptor ring associated with one interrupt. When these kinds of NICs are replaced with the offload card, it is expected that offload cards also provide similar capability of distributing the inbound packet load across multiple descriptor rings with associated interrupt.   In addition to this,  offload cards are expected to place traffic of a given connection (whether it is client-to-server or server-to-client) on to the same descriptor ring.  That is hash value calculated on 5 tuples should be same whether it is C-S and S-C traffic of a connection.  As we know,  SIP, DIP are in opposite order in IP header between C-S and S-C traffic a connection. Same is true with TCP and UDP ports.  Some hash algorithms generate different values if the order of input to the hash algorthm  is different. To ensure that same hash is generated,   offload cards are expected to arrange the input to hash in  fashion that input is same for both C-S and S-C traffic.  One method of doing this is to pass the input based on the value of the field.  That is, always lesser value of SIP and DIP can be passed first to the  hash algorithm before passing higher value among SIP and DIP.  Same needs to be for SP and DP too.  This will ensure that same hash value is generated.

Device Ethernet Driver and offload card should have understanding of number of receive descriptor rings and accordingly the offload cards uses the hash result to map to one of the descriptor rings.

Virtualization Distribution

Some network equipment devices support virtualization with each VM serving different DC customer traffic. VLANs are typically used to distinguish different customer traffic.  Using IOMMU functionality of x86 and PCIe SR_IOV functionality, offload cards are expected to send the inbound traffic directly to the VMs memory without involving the hypervisor layer.  IOMMU also allows direct mapping of different descriptor rings to the user space processes.  For more information on IOMMU works, please refer to the AMD paper on IOMMU specification.

Multiple cores might have been assigned to VMs.  In those cases, this distribution should be combined with the 'Multicore Distribution'.

Miscellaneous

Since offload cards are intelligent NICs and they are part of Data center environment,  data center bridging functionality is expected to be provided by NIC cards, even though these are not offload functions.  Some of the functions expected are:

802.1qaz:  Enhanced Transmission Selection

802.1qbb: Priority based flow control

802.3bd:  MAC control frame for priority based flow control.
802.1 Qau: Congestion notification for end points to take action.

802.1qbg and 802.1qbh :  Virtual bridge (VEPA, VN-Tag)

I hope it helps. If you find any functions that can be offloaded, please drop a comment.

What are Traffic Monitoring Enabler Switches?

There is increasing trend of  Traffic Monitoring Enabler Switches (TMES) in Enterprise, Data Center and Service provider environments.

Need for TMES:

Traffic monitoring devices are increasingly becoming requirement for networks in Enterprise, Data Center and Service provider environments.  There are multiple types of monitoring devices are being deployed in networks.

• Traffic Monitoring for intrusion detection:  Security is very important aspect of Enterprise networks.  Intrusion detection is one of the components of comprehensive network security.  IDS devices listen for the traffic passively and do the intrusion analysis on the traffic.  Intrusion attempts and intrusion events are sent to the administrators for out-of-band action.  IDS devices also can be configured to send TCP resets in case of intrusion detection to stop any more traffic going on the TCP connection.  IDS devices also can be configured to block certain traffic for certain amount of time by informing local firewall devices. 
• Surveillance:  Due to government regulations, all important data needs to be recorded.  Surveillance monitoring devices again listen for the traffic passively and record them in persistent storage for later interpretation.  Surveillance devices also provide capability to recreate the sessions such as Email conversations, file transfer conversations,   Voice and video conversations from the recorded traffic.  Some surveillance devices also provide run time recreation of conversations too.  
• Network Visibility :  These monitoring devices capture the traffic passively and provide complete network visibility of the traffic. They provide capabilities such as 'Identification of applications such as P2P, IM,  Social networking and many more ',  'Bandwidth usage of different applications,  networks'  and provide analysis for network administrators with valuable information to maintain networks and bandwidth to make  Enterprise critical applications work always.
• Traffic Trace:   Traffic trace devices help network administrators to find the bottlenecks in different network segments.    These devices tap the traffic at multiple locations in the network and provide the trace capability for finding out the issues in network such as misconfiguration of different devices in network,  choke points etc..

Network administrators face following challenges to deploy multiple monitoring devices.

• Few SPAN ports in existing L2 switch infrastructure:  Many L2 switch vendors provide one or at the most two SPAN ports.  L2 switches replicate the packets to SPAN ports.  Since there are only two SPAN ports at the most,  only two types of monitoring devices can be connected.  This is one big limitation network administrators face.  
• Multiple network tap points :  In complex network infrastructure, there are multiple points where monitoring is required.  Placing  multiple monitoring devices at each point is too expensive. Network administrators would like to use same monitoring devices to capture traffic at multiple locations. 
• Capacity limitations of monitoring devices:  With increasing bandwidth in the networks, it is possible that one monitoring device may not be able to cope with the traffic.  Administrators would like to multiple monitoring devices of same type to capture the traffic with some external component doing load balancing the sessions across multiple monitoring devices.
• High Capacity Monitoring devices :  There could be instances where monitoring device can take more load. In these cases, one monitoring device can take load from several tap points.  Administrators look for facility to aggregate the traffic from multiple points to one or few monitoring devices of same type.
• Non-Switch capture points :  Network administrator may like monitoring of traffic in a point where there are no switches -  Router to Server,  Wireless LAN Access Point to Access Concentrator etc..   Since there is no switch, there are no SPAN ports.  Network administrators look for some mechanism such as Inline TAP functionality to capture the traffic for monitoring.

What is TMES?

TMES is a switch device with monitoring enabling intelligence to allow connectivity of multiple monitoring devices of different types without any major changes to the existing network infrastructure.

This device taps the traffic from SPAN ports of existing switches in the network and direct the traffic to attached monitoring devices.

TMES allows:

• Centralization of monitoring devices.
• Filtering of the traffic.
• Balancing of traffic to multiple monitoring devices of a given type.
• Replication of traffic to different types of monitoring devices.
• Aggregation of traffic from multiple points to same set of monitoring devices.
• Truncation of data 
• Data manipulation & masking  of the sensitive content of the traffic being sent to monitoring devices.
• Inline TAP functionality to allow capture points where there are no SPAN ports.
• Time Stamp functionality
• Stripping off  Layer 2 and Tunnel headers that are unrecognized by monitoring devices.
• Conditioning of the burst traffic going to the monitoring devices.

Centralization of Monitoring Devices: 

Without TMES,  monitoring devices need to be placed at different locations in the network.  With TMES,  TAP points are connected to TMES and monitoring devices are connected to only TMES ports.

Filtering of Traffic:

This feature of TMES allows filtering of unwanted traffic to a given monitoring device.  Monitoring devices are normally listen for traffic in promiscuous mode. That is, monitoring device gets all the traffic that is going on the wire.  But all the traffic is not interesting to the monitoring device.   Typically monitoring device itself does the filtering.  By offloading filtering out of it,  it saves valuable cycles in receiving the traffic (interrupts) and filtering the traffic.  TMES takes this load out of monitoring device and thereby increase the capacity of monitoring devices.

Filtering of traffic should not only be restricted to unicast. It should be made available even for Multicast and broadcast packets.

Balancing the traffic to Multiple Monitoring devices of a given type of monitoring:

If the amount of traffic that needs to be recorded is very high, then multiple monitoring devices will be deployed.   TMES allows multiple monitoring devices to take the load.  TMES load balances the sessions (not packets) to multiple monitoring devices based on performance of monitoring device.  By balancing based on sessions,  TMES ensures that all the traffic for a given connection go to one monitor device.

Replication of Traffic

When there are different types of monitoring devices, each device is expected to get the traffic.  As discussed above,  traditional L2 switches have at the most two SPAN ports.  TMES is expected to replicate the traffic as many number of times as number of different monitoring device types and send the replicated traffic to the monitoring devices.


Combining the replication feature with load balancing:  Assume that a deployment requires the traffic to be sent to two types of monitoring devices -  IDS and Surveillance. This deployment requires  that the 6Gbps bandwidth traffic to be analyzed and recorded.  If IDS and Surveillance devices can analyze and record only 2Gbps bandwidth, then the deployment requires 3 IDS devices and 3 Surveillance devices.  In this case, TMES is expected to replicate the original traffic twice - One for IDS devices and another for Surveillance devices.  Then TMES is expected to balance the one set of replicated packets to go to one of 3 IDS devices and second set to go to one of three Surveillance devices.  

Aggregation of traffic from multiple points to same set of monitoring devices

As discussed in 'Centralization of Monitoring devices',  traffic from different locations of the network can be tapped.   TMES is expected to provide multiple ports to receive the traffic from multiple locations in the network, filter the traffic, replicate and balance the traffic across monitoring devices.  It is possible that traffic of a given connection might be going through multiple points and hence there could be duplication of traffic coming to the TMES.  It is also possible that duplicated traffic might be going to the same monitoring device.  Monitoring device might get confused with duplicated traffic.  To avoid this scenario,  TMES is expected to mark the packets based on the incoming port on TMFS (that is the capture point) such as adding different VLAN ID based on the capture point or adding an IP option etc..    This would allow monitoring device to distinguish the same connection traffic across different capture points.




Truncation or Slicing of packets


Some monitoring device types such as traffic measuring devices don't require complete data of the packets to come in. By slicing the packet to smaller packet would increase the performance of those monitoring devices. TMESs are expected to provide this functionality before sending the packets to the monitoring devices. Truncate value is with respect to payload of TCP, UDP etc..  Some monitoring devices are only interested in headers upto layer 4.  In this case, truncation value can be 0.  Some monitoring devices may expect to see few bytes of payload. TMESes are expected to provide this flexibility of configuring truncate value.

Truncation of packet content should not reflect in the IP payload size. It should be kept intact to ensure that monitoring devices can figure out the original data length even though it receives truncated packets.



Data  masking 

Based on type of monitoring devices,  administrator may like to mask some sensitive information such as credit cards,  user names and passwords from being recorded.   TMES is expected to provide this functionality  of pattern match and mask the content there by removing privacy concerns.

TMES also might support Data replacement (DR).   DR feature might increase the size of the data. Though it is not a big issue for UDP type of sessions, it requires good amount of handling for TCP connections.  As we all know TCP sequence number represent the bytes, not packets.  So, any changes in the data size requires sequence number update.  It not only requires sequence number update in the affected packet, but also further packets going on the session.  All new packets would undergo the sequence number update. Similarly, ACK sequence number of reverse packets also should be updated while sending the packets to the monitoring devices.

When DR feature is combined with the 'Replication' feature,  this delta sequence number update can be different for different replicated packets.  Delta sequence number update feature is required to ensure that monitoring devices find the packet consistency with respect to sequence numbers and the data.

Some TMES vendors call this as part of DPI feature.


Inline TAP functionality:



Many places in the network might not have L2 switch to get hold of traffic from SPAN ports. If the traffic needs to be monitored from those points, then one choice is to place L2 switch and pass the traffic to the monitoring devices through the SPAN ports.  If the capture points are high, then there is a need for placing multiple L2 switches.  Inline TAP functionality is expected to be there in the TMES.  Two ports of TMES are required to TAP the traffic from these capture points.  These two ports will act as L2 switch while replicating traffic for monitoring devices.  Baiscally, TMES is expected to act as L2 swtich for these capture points. Since there are many capture points, TMES essentially become Multi-switch device with each logical switch having two ports.

Time Stamping of Packets

Analysis of traffic that was recorded acorss multiple monitoring devices would be a requirement in general.  It means that the recording devices should have same clock reference so that analysis engine knows the order in which the packets were received.  Yet times, it is not practical to assume that the monitoring devices will have expensive clock synchronization mechanisms.  Since TMES is becoming a central location to get hold of traffic and redirecting them to monitoring devices, TMES is expected to add time stamp in each packet that is being sent to the monitoring devices. 

IP protocol provides an option called 'Internet TimeStamp'.  This option expects TMES to fill its IP address and timestamp in milliseconds with midnight UT.  

Stripping of L2 and Tunnel headers


Many monitoring devices don't understand complicated L2 headers such as MPLS, PPPoE and tunnel headers such as PPTP (GRE),  GTP (in case of wireless core networks),  L2TP-Data,  IP-in-IP,  IPv6 in IPv4 (Toredo,  6-to-4, 6-in-4) and many more.  Monitoring devices are primarily interested in inner packets. TMESs are expected to provide stripping functionality and provide basic IP packets to the monitoring devices.  Since monitoring devices expect to see some known L2 header,  TMESes typically are expected to strip off tunnel headers and complicated L2 headers and keep Ethernet header intact.  If Ethernet header is not present,  TMESes are expected to add dummy Ethernet header to satisfy the monitoring device reception.


Traffic Conditioning

Monitroing devices are normally rated for certain amount of Mbps. Yet times, there could be bursts in the traffic, even though overall average traffic rate is within the device rating.  To avoid any packet drop due  to brusts,  TMES is expected to condition the traffic going to monitoring device.

Players :

I came across few vendors who are providing solutions meeting most of abvoe requirements.

Gigamon :  http://www.gigamon.com/
Anue Systems:  http://www.anuesystems.com/
NetOptics:  http://www.netoptics.com


I believe this market is yet to mature and there is lot of good upside potential.

There is good need for monitoring devices and hence the need for TMES will only go up in coming years.

User space Packet processing applications - Execution Engine differences with processors

Please read this post to understand Execution Engine.

Many processors with descriptor based IO devices have their own interrupts.  For each device, there is corresponding UIO device.  Hence software poll based EE provides 'file descriptor' based interface to register, deregister and get hold of indication through callbacks.  EE applications are expected to read the packets from the the hardware by themselves and do rest of the processing.

As discussed in UIO related posts,  we have discussed ways to share the interrupts across devices. As long as UIO related application kernel driver knows the type of event for which interrupt is generated, appropriate UIO FD is woken up and  things will work fine.

Non-descriptor based IO is becoming quite common in recent Multicore processors.  Hardware events (packets from the Ethernet controllers,  acceleration results from the acceleration engines) are given to the software through set of HW interfaces.  Selection of  HW interface by the hardware is based on some load balancing algorithms or based on some software inputs.  But the point is that, the events which are being given to the software through one HW interface are from multiple hardware IO sources.  Each HW interface is normally associated with one interrupt.  One might say that this can be treated as interrupt being shared across multiple devices.  But, some of  the Multicore processors don't have facility to know the reason for HW interrupt.  Nor they have facility to know the event type of first pending event in HW interface.  Unless the event is dequeued from the HW interface, it is impossible to know the type of event.  Also, due to interrupt coalescing requirements, a given interrupt instance might represent multiple events of different IO source devices.  Due to this behavior,  there may be only one  UIO device for multiple IO devices.  Hence responsibility of demultiplexing these events to right EE application falls on the EE itself.  EE needs to read the event and find out the right application and call the appropriate callback function registered to it. Let us call this functionality in EE as 'EE Event DeMux'.

In Descriptor based systems,  EE applications are expected to read the HW events (packets & acceleration results) by each EE application.  Callback function invocation only provides indication for EE application to read the events from associated hardware descriptors.  In case of 'EE Event DeMux',  the event is already read by the EE itself.  Hence, event is expected to be passed to the callback function.

'EE Event DeMux' submodule registers itself with the rest of EE module to get UIO indication in case of software poll method.  In case of  hardware poll,  'EE Event DeMux' in invoked by the  hardware poll function.

Multicore processors normally provides HW interface for multiple IO devices for the devices which are part of the Multicore processors.  External devices such as PCI and other HW bus based IO devices are still implemented using descriptor based mechanism.  Software poll based EE should not assume that all devices are satisfied using 'EE Event DeMux'.   As far as core Software poll system is concerned,  'EE Event DeMux' is another EE application.  Hardware Poll based method, if they need to use descriptor based HW interfaces, then the hardware poll should also poll descriptor based HW interfaces.

When 'EE Event DeMux' is used by EE applications (such as Ethernet Driver,  Accelerator drivers,), it is necessary that 'EE Event DeMux' considers following requirements.

• It should have its own 'Quota' as number of maximum events it is going to read from the HW interface as part of the callback function invocation by the EE core. Once it reads the 'Quota' number of events or if there are no more events, then it should return back to the 'Core EE' module.  
• Since this the module which demuxes to some EE applications, it should provide its own register/De-register functions.
• When 'Core EE' module invokes this module callback function due to interrupt generation or due to hardware poll,  as described above, it is expected to read at the most 'quota' number of events. While giving the control back to the  'Core EE', it is expected to call EE applications that there are no more events in this iteration. Some EE applications might register to get this indication.  For example, Ethernet driver application might register for this to do the 'Generic Receive Offload' function.  GRO functionality requires to know when to give up while doing TCP coalescing functionality.  In case descriptor based drivers, this issue does not arise as each Ethernet driver as part of callback invocation by the EE itself reads the events and knows when to give up.

Thanks for reading my earlier post.  I hope this helps.

User space Packet processing applications - Execution Engine

If you plan to port your data plane network processing application from Linux kernel space to user space,  first thing you would think is how you can port your software to user space with minimal changes to your software.  Execution Engine is the first thing one would think of.

Many kernel based networking applications don't create their own threads.  They work with the threads which are already present in the kernel.   For example,  packet processing applications such as Firewall, IDS/IPS, Ipsec VPN work in the context of Kernel TCP/IP stack.  This is mainly done for performance reasons.  Additional threads for these applications result in multiple context switches.  Also, it results into pipeline processing as packets handover from one execution context to another execution context.  Pipelining requires queues involving enqueue, dequeue operations which take some core cycles.  Also, it results into flow control issues when one thread processing is more than other threads. 

Essentially,  Linux kernel itself provides execution contexts and networking packet processing applications work within these contexts.  Linux TCP/IP stack itself works in softirq context.  SoftIRQ processing in normal kernel runs from both IRQ context as well as softirqd context.  I would say 90% of the time SoftIRQ processing happens in the IRQ context.  In PREEMPT_RT patched kernel, network IRQs are mapped to the IRQ threads.  In any case,  the context at which network packet processing applications run is unknown to the applications.  Since Linux kernel execution contexts are per core basis,  there are  less shared data structure and hence less locking requirement.  Kernel and underlying hardware also provides mechanism to balance the traffic across different execution contexts with flow granularity. In cases where hardware does not provide any load balancing functionality,  IRQs are dedicated to different execution contexts.  If there are 4 Ethernet devices and 2 cores (hence two execution contexts),  four receive interrupts of Ethernet controllers are assigned equally to two execution contexts.. If the traffic from all four Ethernet devices is same or similar, then both the cores are used effectively.  

Execution Engine in user space packet processing applications, if made similar to the Kernel execution contexts, then application porting becomes simpler.  Execution Engine (EE) can be considered part of infrastructure to enable DP processing in user space.  EE design should consider following.

• There could be multiple data plane processing applications in user space.  Each DP daemon may be assigned to run in fixed set of cores - core mask may be provided at startup time.
• If DP daemon is not associated with any core mask, then it should assume that the daemon may be run by all cores. That is, it should considered as if core mask contains all core bits set.
• Set of cores are dedicated to the daemon. That is, those cores don't do anything else other than doing DP processing of the DP daemon.  This facility is typically used to ensure in Multicore processors providing hardware poll.  Recent generation of Multicore processors have facility to provide incoming events and acceleration results through single portal (or station or work group).  Since the core is dedicated, there is no software polling is required.  That is hardware polling can be used if underlying hardware supports it and if the core(s) are dedicated to the process.

It appears that number threads in the process equaling the number of cores assigned to the process provides the best performance. Also, this also provides great similarity with kernel execution contexts. With the above background,  I believe EE needs to have following capabilities:

• Provide capability to assign the core mask. 
• Provide capability to indicate whether the cores are dedicated or assigned.
• If no core mask is provided, it should have capability to read the number of cores in the system and should assume that all the cores are given in core-mask.
• Provide capability to use software poll or hardware poll.  Hardware poll should be validated and accepted only if underlying hardware supports it and only if the cores are dedicated to it. Hardware polling has performance advantages as it does not require interrupt generation and interrupt processing. But the disadvantage is that the core is not used for anything else.  One should weigh the options based on the application performance requirements. 
• API it exposes for its applications should be same whether the execution engine uses software poll (such as epoll()) or hardware poll.

Typically capabilities are provided through command line parameters or via some configuration file.  EE is expected to create as many threads as number of cores in the core mask.  Each thread should provide following functionality:

• Software timers functionality - EE should provide following functionality.
• Creation and deletion of timer block
• Starting, stopping, restarting timers in each timer block.
• Each application can create one ore more timer blocks and use large number of timers in each timer block.
• As in Kernel,  it is required that EE provides cascaded timer wheels for each timer block.
• Facility for applications to register/De register for events and passing the events.
• API function (EEGetPollType()) to return the type of poll - Software or Hardware : This function would be used by EE applications to use file descriptors such as UIO and other application oriented kernel drivers for software poll or use hardware facilities for hardware poll 
• Register Event Receiver :  EE applications can use this function to register the FD, READ/Write/Error, associated callback function pointer and callback argument.
• Deregister Event receiver:  EE applications can call this to de register the event receiver which was registered using 'Register' function.  
• Variations above API will need to be provided by EE if it is configured with hardware poll. Since each hardware device has its own way of representing this,  there may as many number of API sets. Some part of each EE application have hardware specific initialization code and calls the right set of API functions.
• Note that one thread handles multiple devices (multiple file descriptors in case of software poll). Every time epoll() comes out,  callback functions of ready FDs would need be called. These functions which are provided by the EE applications are expected to get hold of packets in case of Ethernet controllers, acceleration results in case of  acceleration devices or other kinds of events from different kinds of devices. From the UIO discussion,  if the applications use UIO based interrupts to wakeup the thread, then it is expected that all events are read from the device to reduce the number of wakeups (UIO coalescing capability).  Some EE application might be reading lot of events. For each event it reads, it is going to call its own application function.  These applications can be very heavy too. Due to this,  if there are multiple FDs are ready,  one EE application may take very long time before it returns back to the EE.  This results into unfair assignment of EE thread to FDs which are also ready.  This unfairness might even result into packet drops or increase the jitter if high priority traffic is pending to be read in other devices.  To ensure the fairness,   it is expected that EE applications process only 'quota' number of events at any time before returning back to the EE.  'Quota' is tunable parameter and can be different for different types of devices.  EE is expected to callback the same application after it runs through all other ready file descriptors.  Until all ready EE applications indicate that they don't have anything to process, EE should not be calling ePoll().  To allow EE to know whether to call the application callbacks again,  there should be some protocol.  Each EE application can indicate to the EE while returning from the callback function on whether it processed all events.  EE based on this indication will decide to call the EE application again or not before it goes to the epoll() again.  Note that epoll() is expensive call and hence it is better if all events are processed in fairness fashion before epoll() is called again.   In case of hardware poll based configuration,  this kind of facility is not required as polling is not expensive. Also Multicore SoCs implementing the single portal for all events have fairness capabilities built in.   Since the callback function definition is same for both software and hardware poll based systems,  these parameters exist, but they are not used by hardware poll based systems.

EE before creating threads should initialize itself and then create threads. Once created it should load the shared libraries of its applications one by one.  For each EE application library, it is expected to call 'init' function by getting hold of address of 'init()' symbol.  Init() function is expected to initialize its own module.  Each EE packet processing threads is expected to call another function of EE application. Let us call this symbol name is 'EEAppContextInit()'.   EEAppContextInit function expected to real initialization such as opening UIO and other character device drivers and registering with the software poll() system.

EE also would need to call 'EEAppFinish()' function when the EE is killed.  EEAppFinish does whatever graceful shutdown required for its module.

Each thread, if it is software based poll, does epoll on all the FDs registered so far.  Polling happens in the while() loop.  epoll() can take the timeout argument. Timeout argument must be next lowest timer expiry timeout of all software timer blocks.  When epoll() returns, it should call the software timer library for any timer expiry processing. In case of hardware poll,  specific hardware specific poll function would need to be used.

In addition to above functions,  EE typically needs to emulate other capabilities provided by Linux Kernel for its applications such as - Memory Pool library,  Packet descriptor buffer library,  Mutual exclusion facilities using Futexes and user space RCU  etc.. 

With these above capabilities, EE can jump start the application development. This kind of functionality only requires changes at very few places in the applications.

Hope it helps.

UIO - Acceleration Device mapping in user space

Please see this post on how to use UIO frame work to implement device drivers in user space. As noted in that post,  UIO framework predominantly is used to install the interrupt handler and to wake up the user space process implementing the device driver. Please read the earlier post, before going further.

There are two types of devices that get mapped to user space for zero-copy drivers - Ethernet devices and Accelerator devices such as Crypto Engine,  Pattern Matching Engine etc..   Normally, a given Ethernet device is completely owned by one user process.  But accelerator devices are normally needed across multiple processes and also is needed by kernel applications.  Hence acceleration device usage is more challenging. 

To enable usage of acceleration devices by multiple user processes, acceleration device normally support multiple individual descriptor rings.  I know of some acceleration devices supporting four descriptor rings, with each descriptor ring working independent of each other, that is, one descriptor ring is sufficient for issuing the command and read the result .  In this scenario,  a given user process at least should own one descriptor ring for zero copy driver.  If the acceleration device contains four descriptor rings, then four user processes can use the acceleration device without involving the kernel. Since a typical system contains more processes than the descriptor rings,  it is necessary that at least one descriptor ring is reserved for kernel usage and other application processes.  In the example where one acceleration device supports four descriptor rings,  in one scenario,  one can choose three critical user processes that require zero copy driver. Each of these critical processes use one descriptor ring each. All other user processes and kernel share one descriptor ring.  

Each process requiring zero copy driver should memory map the descriptor ring space.  Since many chip vendors provide Linux kernel drivers for acceleration engines,  my suggestion is to make changes to this acceleration engine driver to provide some additional API functions to detach and attach the descriptor rings on demand basis.  When user process requires a descriptor ring, the associated application kernel module can call the acceleration driver 'detach' function for that descriptor ring.  When the process dies, the associated kernel module should attach back the descriptor ring to the kernel driver. This way, each user process need not work on the initialization of security engine.  It only need to worry about the filling up the descriptors with commands and reading responses.  It also provides the benefit that the descriptor rings can be dynamically allocated and freed based on the applications running at that point of time.

If there are as many interrupts as number of descriptor rings, then each process's zero copy driver can have its own interrupt line.  Yet times, even though there are multiple descriptor rings,  number of interrupts are less than the descriptor rings. In this case, interrupt need to be shared across multiple descriptor rings. Fortunately Linux kernel and UIO frame work provides mechanism for multiple application kernel modules to register different interrupt handlers for the same interrupt line. irq_flags field as part of uio_info structure that is registered with the UIO framework should have IRQ_SHARED bit set.  Linux Kernel and UIO frame work call the interrupt handler one by one in sequence. Interrupt handler that has data pending to be read from corresponding descriptor ring should return IRQ_HANDLED.    It means that the device should have capability to check the pending data without reading it out.  Note that reading the acceleration result should be done by the user space.   When the handler returns IRQ_HANDLED, UIO framework wakes up the user process.  Since one IRQ line is shared by multiple processes,  as described in earlier post, masking and unmasking the interrupts can't be done by the interrupt handler and user process. Since interrupts can't be disabled, one can't use natural coalescing capability as described in the earlier post.  But fortunately, many acceleration devices provide hardware interrupt coalescing capability. Hardware can be programmed to generate interrupt for X number of events or within Y amount of time. If the hardware device you have chosen does not have the coalescing capability and require IRQ to be shared across multiple user processes, then you are out of luck. Either don't use UIO facilities or live with too many interrupts coming in.

All other user process  without dedicated descriptor rings should work with accelerator kernel driver that is provided by the OS/Chip vendors.  That is, they need to send the command buffer to the kernel driver and read the result from the kernel driver.  Kernel drivers are normally intelligent enough to service multiple consumers and hence many user processes can use the acceleration engine.


Comments?

Multicore Networking applications - Mitigating the Performance bottlenecks

I had given this talk in 2010 Multicore Expo in San Jose.  It was in presentation document in concise form. I voiced the most of the details during my talk.  Many people requested me to provide details in written form.  I tried to give details here in this post.  I hope this post would give enough details on 'New techniques to improve software performance with increasing number of cores'.

Before going further, I would like to differentiate two kinds of applications - Packet processing applications and Stream processing applications.

Packet processing applications in my definition are the ones which  take  packet by packet, work on the packet and send out the same packet or packet with some minor modifications.  In packet processing applications,  there is one-to-one correspondence between input and output packets except for very small number of exceptions. One example where there is no one-to-one correspondence is when there is IP reassembly or fragmentation. Other example is when the packet is dropped by the application. Example applications in this category are:  IP forwarding, L2 Bridging,  Firewall/NAT,  Ipsec and even some portions of IDS/IPS.

Stream processing applications are the ones which may take packets or stream of data,  work on data and send out the data or  send out different packets. Most of  the TCP socket based proxy applications come under this category. Examples:  HTTP Proxy,  SMTP Proxy,  FTP Proxy etc..

This post tries to aid the programmers debugging the software to find out the performance bottlenecks in Multicore networking applications.

Always Ensure to do  flow/Session Parallelization 

Ensure that only one core is processing the session at any given time.  If multiple packets from the same session are being processed by more than one core at the same time,  then there would be requirement to ensure that Mutual exclusion on the session variables.  That would be very expensive.  Multicore SoCs actually aid you to do flow parallelization in packet processing applications.  Many Multicore SoCs support parsing the fields from the packets,  calculate hash on the software defined fields and distribute the packets across the multiple queues based on the hash value.  And then they provide provision for software threads to dequeue the packets from the queues.  These SoCs also provide provision to stop dequeue of packets from threads until the control of the queue is given up explicitly.  This ensures that a given flow is processed by only software thread at any time.

Many Multicore SoCs also have facility to bind the queues to the software threads and each software thread to the core.  If the number of flows are small, there is a possibility of cache being warmed with contexts due to previous packets. This reduces the data movement from DDR.   Also, many Multicore SoCs provide facility to stash the context as part of dequeue operation which reduces the cache thrashing issue even if  binding of the queues to the cores are not done. 

Flow parallelization not only eliminates the need for Mutexes, it also ensures that there is no packet mis-ordering in the flows.

Many stateful packet processing applications require not only flow parallelization, but also session parallelization.  Session typically consists of two flows - Client to Server traffic and Server to Client traffic.  It is possible that two packets from both the flows may be coming to the device and two separate software threads might be processing these packets. Stateful applications share many state variables across these two flows. Due to this, you may require mutual exclusion operation if both the packets are allowed to be processed at the same time.  Session Parallelization as described here would eliminate the need for mutual exclusion.  Unlike flow parallelization, session parallelization is not available in many Multicore SoCs for cases where the tuple values are different in both the flows and hence needs to be done in software.  Packet tuples are different when NAT is applied. Note that many Multicore SoCs enqueue the packet to the same queue if there is no NAT.  They are intelligent enough to generate the same hash value even though the tuples position get changed, that is, source IP in one flow would be destination IP in reverse flow and same is true for destination IP, Source Port and Destination Port.

Stream processing modules such as proxies would need to ensure that both client side and server side sockets are processed by the same software thread to ensure that there is no  Mutual exclusion operations requirement to  protect the sanctity of state variables.  Stream processing modules typically create many software threads - worker threads.  Master thread terminates the client side connections and handover the connection descriptor to one of the less loaded worker threads.  Worker thread is expected to create new connection to the server and do rest of the application processing.  Worker threads are typically implement FSM for processing multiple sessions. More often,  the number of worker threads would be same as number of cores dedicated for that application.  In cases where the threads need to block for some operations such as waiting for accelerator results, then more threads, in multiples of number of cores, would be created to take advantage of full power of accelerators.

Eliminate the Mutual Exclusion Operation while Searching for Session/Flow Context

This technique is also expected to ensure that there are no mutual exclusion operations in the packet path.  Any networking application do some search operations on the data structures to figure out the operations and other action to be done on the packet/data.  Upon the incoming packet/data,  search is done to get hold of session/flow context and then further packet processing happens based on the state variables in the session.  For example,  IP routing does search on the routing table to figure out the destination port, PMTU and other information for operations such as fragmentation, TTL decrement and packet transmit.  Similarly firewall/IPsec packet processing applications maintain the sessions in a easy to search data structures such as RB trees, hash lists etc..   Since the sessions are created or removed dynamically from these structures, it is necessary to protect the data structure while doing operations such as add/delete/search.  Mutual exclusion operations using spinlock,  futex, up/down are one way to do this.   RCU (Read-Copy-Update) is another method that can be used which eliminates the Mutex operation during search.   RCU operation is described in earlier post.  Please check that here and here.  RCU lock/unlock operations in many operating systems is very simple operation. Note that Mutex operations are still required for add/delete even in RCU based usage.

Eliminate Reference Counting 

One of the other bottlenecks in the Multicore programming is the need to keep the session safe from deletion while it is being used by other software threads. Traditionally this is achieved by doing 'reference counting'. Reference counting is used in two cases - During packet processing operation or  When neighbor module store the reference.

In the first case, reference count of the session context is incremented as part of the session lookup operation.  During packet processing, the session is referred many times to get hold of state variable values and to set the new values in the state variables of the session.  It is expected that if the session is deleted, it should not be freed until the current thread is done with its operation. Otherwise, it would corrupt some other memory if the session memory is freed and allocated to somebody else during packet processing.  To ensure that the session ownership is not given away, the reference count is checked as part of 'delete operation'.  If is is not zero, then the session is marked for deletion, but not freed until the reference count becomes zero.  If the value is zero, it indicates there is no reference to this session and the session gets freed. 
 
Since RCU operation postpones the delete operation until current processing cycles of all other threads,  reference counting becomes redundant.  Elimination of reference count not only helps in improving the performance, but also reduces the maintenance complexity. Note that reference counting operation requires atomic usage of count variable. Atomic operations are not inexpensive.

Second use case of reference count is when the neighbor modules store the reference (pointer) to the sessions in their session contexts. By eliminating the storage of pointer,  reference count usage can be eliminated.  This post helps you understand how this can be done.

Linux user space programs also can take advantage of RCUs. See this post for more details.

Use the Cache Effectively

Once the matching session is found upon incoming data/packet,  processing functionality uses many variables in the session. If these variables are together in a cache line,  any cache fill due to access of one variable result all other variable in the cache line available in the cache.  That is, Access to other variables will not result in going to DDR.  But all variables may not fill in one cache line. In those cases, it is necessary to group the related variables together to reduce going to DDR.

To effectively use instruction cache, always arrange your code with likely/unlikely compiler directives. Compilers will try to arrange the likely() code together. 

Reduce Cache Thrashing due to Statistics variables

Almost all networking applications update statistics variables.  Some variables are global and some of them are session context specific variables.  There are two types of statistics variables -  increment variables and add variables. Increment variables are typically used to maintain the count of packets.  Add variables are used to maintain the byte count.  Updating these variables require getting hold of current values and then add or increment operation.   If these variables are updated by multiple threads (with each thread running on a specific core), then every time an variable is updated,  cache information of this variable is no longer valid in other cores.  When one of other cores needs to do same operations,  it needs to get the current value first from the DDR and apply the operation.  In worst case scenario, where packets are going to round robin fashion to different software threads (hence cores), then the cache thrashing due to statistics variables would be very high and this would reduce the performance dramatically.

Always use 'per core/thread statistics counters'  whenever possible.  Please see this post for more details. 

Some Multicore SoCs provide special feature which also eliminates the need for 'per core' statistics maintenance.  These SoCs provide facility to allocate memory block for statistics.  These SoCs provide facility to fire the operation and forget about it.  Firing the operation involves the operation type (increment, decrement, add X or sub Y etc..) and memory address (32 bit or 64 bit).  SoCs internally do this operation without cache thrashing.  I suggest strongly to use this feature, if it is available in your SoC.

Use LRO/GRO facilities

Many networking applications' performance depends on the number of packets being processed than the number of bytes processed.  Examples: IP Forwarding,  Firewall/NAT and Ipsec with hardware acceleration.  So, reducing the number of packets processed becomes key in improving the performance.

LRO/GRO facilities provided by operating system in Ethernet drivers or by Multicore SoCs reduce the number of TCP packets, if multiple packets from the same TCP flow are pending to be processed.  Since TCP is byte oriented stream protocol, it does not matter whether or not the processing happens on packets.  Please see this post for more information on LRO feature in Linux operating system.  If it is supported by your operating system or Multcore SoC, always make use of it.

Process Multiple Packets together


Each packet processing module does set of operations on the packets/data - such as Search,  Process and  Pkt out.  If the packet is going through multiple modules, there are many C functions get called.  Each invocation of C function has its own overhead such as pushing the variables in the stack, initializing some local variables etc..    By bunching multiple packets of same flow together can reduce search/pkt out overhead and overhead associated with the C functions.


Some Multicore SoCs provide facility to coalesce packets together on per queue basis with coalescing parameters -  Packet threshold and time threshold.  Queue does not let the target thread to dequeue until one of the conditions reached - either number of packets in the queue exceed the packet threshold parameter or if no packet was dequeued for time 'time threshold'.   If this facility is available, ensure that your software dequeues multiple packets together and processes them together. 

Yet times, there is no one-to-one correspondence between queues and sessions.  In that case, one might ask that search overhead can't be reduced as there is no guarantee that the packets in the same queue belong to the same session.  Though it is correct, it might still have some improvements due to cache warming if there are more than one packet belonging to same session in the bunch.

As a software developer,  it would be required to strive for one-to-one correspondence between queues and sessions. This can be done easily among the modules running in software.  Some Multicore SoCs provide queues for not only to access hardware blocks, but also for inter-module communication.  Software can take advantage of this to create one-to-one mapping between queues and destination module's sessions.

It is true that when the packets are being read from the Ethernet controllers, there is no way to ensure that a queue only holds packets of one session as the queue selection happens based on the hash value of packet fields.  Two different sessions may fall into same queue.  In those cases,  as mentioned above you might not see improvement from 'serach' functionality, but you would still see improvements due to less number of invocations of C functions.

Many Multicore SoCs also have functionality to take multiple packets together for acceleration and for sending the packets out. This also will reduce the number of invocation to acceleration functions and for sending packets out.  If this facility is available in your Multicore SoCs,  take advantage of it. 

Eliminate usage of software queues

Some Multicore applications need to send the packets/data/control-data to other modules.  If multiple threads send the data to the queue, then there is a need for mutual exclusion to protect these data structure queues.

Many Multicore SoCs provide queues for software usage.  These queues would eliminate the need for software queues and hence eliminate the mutual exclusion problem, there by improving performance.   Some Multicore SoCs also provide facility to group multiple queues together into a queue group which allows sending and receiving applications to enqueue priority items and dequeue based on priority.  These queues can be used even among different processes or virtual machines as long as shared memory is used for items that get enqueued and dequeued.  Some Multicore SoCs even went a step further to provide 'copy' feature which avoids shared memory and there by providing good isolation. This feature makes a copy of these items from source process to internal managed memory by Multicore SoCs and copy to the destination process memory as part of dequeue operation.

Always use this feature if it is available in your Multicore SoC.

Eliminate the usage of Software Free pools 

Networking applications use free pools of memory blocks for memory management.  These free pools are used to allocate/free session contexts, buffers etc..   Many software threads would require these facilities at different times. Software typically maintains the memory pools on per core basis to avoid mutual exclusion operations on per allocation basis.  Since there is a possibility of asymmetric usage of pools by different threads, yet times there is a possibility of memory allocation failures even though there are free memory blocks in other threads' pools.   To avoid this, software does complex operations during these scenarios of moving memory blocks from one pool to another through global queues.   Many Multicore SoCs provide 'free pool' functionality in hardware.  Allocation and free can be done by any thread at any time without mutual exclusion operations. Use this facility whenever it is available.  It saves some core cycles.  More than that is provides efficient usage of memory blocks.

Use Multicore SoC acceleration features to improve performance

There are many acceleration features that are available in Multicore SoCs.  Try to take advantage of them.  I classify acceleration functions in Multicore SoCs into three buckets -  Ingress In-flow acceleration,  In-flight acceleration and Egress in-flow acceleration.

Ingress In-flow acceleration:  Acceleration functions that are done by Multicore SoCs in the hardware on the packets before they are handed over to software are called Ingress In-flow accelerations.  Some of the features, I am aware, in Multicore SoCs are:

• Parsing of Packet fields :  Some Multicore SoCs parse the headers and make those fields available to the software along with the packet.  Software needing the fields can eliminate the parsing of fields.   These SoCs also provide facility for software to choose the fields to be made available along with the packet.  They also provide facilities for software to create parser to extract fields from proprietary headers or from non pre-defined headers.  Try to take advantage of this feature.
• Distribution of packets across threads:  This is basic feature required in Multicore environments.  Packets needs to be distributed to different software threads.  Many Multicore SoCs also ensure that packets belonging to one flow go to one software thread at any time to ensure that packets will not get mis-ordered within a flow.  As described above,  multiple queues would be used by hardware to place the packets.  Selection of queue is based on hash value calculated on the set of software programmable fields.  As a software developer, take advantage of this feature rather than implementing the distribution in software.
• Packet Integrity checks & Processing offloads:  Many Multicore SoCs do quite a bit of integrity checks on the packet as  listed below.  Ensure that your software don't do them again to save some core cycles.
• IP Checksum verification.
• TCP, UDP checksum verification.
• Ensuring that the headers are there in full.
• Ensure that size of packet is not less than the size indicated in the headers.
• Invalid field values.
• IPsec inbound processing.
• Reassembly of fragments
• LRO/GRO as described above.
• Packet coalescing as described above.
• Many more.
• Policing :  This feature can police the traffic and reduce the amount of traffic that is seen by the software.  If your software requires policing of some particular traffic to stop cores from getting overwhelmed, this feature can be used rather than doing it in the lowest layers of software.
• Congestion Management :  This feature ensures that the number of buffers used up by the hardware won't go up exponentially. Without this feature, cores may not find buffers to send out the packets if all buffers are used up by the receiving hardware. This situation typically happens when the core is doing lot of processing and hence slow in dequeuing while lot more packets are coming in.  Many Multicore SoCs also have facility to generate pause frames in case of congestion. 

Egress In-flow acceleration:   Acceleration functions that are done in the hardware once the packets are handed over to it by software to send the packets out are called Egress in-flow acceleration functions.  Some of the Egress in-flow acceleration functions are given below.  If these are available, take advantage of them in your software as these can reduce significant number of cycles in the core.

• Shaping and Scheduling :  High priority packets are sent out within the shaped bandwidth.  Many Multicore SoCs provide facilities to program the effective bandwidth. These SoCs shape the traffic with this bandwidth. Packets which are queued to it by software would be scheduled based on the priority of the packets.  Some SoCs even provide multiple scheduling algorithms and provide facility for software to choose the algorithm on per physical or logical port.  Some SoCs even provide hierarchical scheduling and shaping.  Take advantage of this in your software if you require shaping and scheduling of the traffic.
• Checksum Generation for IP and TCP/UDP transport packets :  Checksum generation, especially for locally generated TCP and UDP packets is very expensive.   Use the facilities provided by hardware.  
• Ipsec Outbound processing :  Some Multicore SoCs provide this functionality in hardware.  If you require Ipsec processing,  use this facility to save large number of cycles on per packet basis.
• TCP Segmentation and IP Fragmentation :  Some Multicore SoCs provide this functionality.  TCP segmentation performs well for local generated packets. Use this functionality to get best out of your Multicore.

In-flight Acceleration :   Acceleration functions provided by hardware that can be used during packet processing are called In-flight acceleration functions.  Crypto,  Crypto with protocol offload,  Pattern Matching,  XML acceleration are some of the acceleration functions that come in this category.  Here the packet/data for acceleration is handed over to the hardware acceleration functions by software. Software reads the results at later time when the results are ready.  Take advantage of these feature in your software wherever they are available . Some Multicore SoCs differentiate themselves by doing lot more in the acceleration functions.  For example,  some Multicore SoCs do protocol offload along with crypto such as Ipsec ESP,  SSL record layer protocol , SRTP and MACSec offloads which do beyond crypto offload.

I see many times people asking me a question on how to use the acceleration functions.  I had detailed this long time back here. Please see the details there and there.

Software Directed Ingress In-flow accelerations:

As described before, Ingress in-flow acceleration is applied before the packets are given to the software. Packets that are received on integrated Etherent controllers go through this acceleration.  But many times this acceleration is required from software too.  Take the example of Ipsec, SSL or any tunneling protocol.  Once the software processes these packets, that is once it gets hold of inner packets,  software would like ingress in-flow acceleration to be applied on the inner packets for distribution across cores and other acceleration functions.  To facilitate these kinds of scenarios, some Multicore SoCs provide concept of 'offline port' which allows software to reserve the offline ports and send the traffic for ingress in-flow acceleration.  Some software features that can take advantage of this feature are:

• Tunneled traffic as described above to let the inner packets to go through the ingress in-flow acceleration,
• IP reassembled traffic - Once the fragments are reassembled, it would have all 5-tuples which can be used to distribute the traffic through offline port.
• L2 encapsulated packets - Such as IP packet from PPP, FR etc..
• Ethernet controllers on PCI and Traffic from Wireless interfaces :  Here the traffic might need to be read by the software and Ingress in-flow acceleration might not have been implemented for these features. Software after getting hold of packets can be directed to in-flow acceleration functions through offline ports.

Use Multicore core features wherever they are available

Multicore SoCs from different vendors have different core architecture. Some Multicore SoCs are based on power pc, some based on MIPS core and Intel Multicore is based on x86 processors. Multicore SoC vendors provide different features to improve performance of Multicore applications.  Whenever they are available, software should make use of them to get the best performance out of cores.  Some of the features that I am aware of are listed below.

Single Instruction & Multiple Data instructions (SIMD)

Multicore SoCs from Freescale and Intel have this block in their cores.   This feature in the cores allows software do a given operation on the multiple data elements.  This kind of parallelism is called 'Data level parallelism'.  'Add' operation in typical cores is performance either on 32 bit or at the most 64 bit operands.  Current generation of SIMD do this operation on 128 bit operands. They also provide flexibility to do multiple 16 bit, 32 bit add operations on different parts of data simultaneously.  SIMD greatly helps in operations which involve arithmetic, bit, copy, compare operations on large amount of data.  Any operation that is done in a loop can be accelerated using SIMD.   In networking world,  SIMD is helpful in following cases:

• Memory compare, copy,  clear operations.
• String compare, copy, tokenization and other string operations.
• WFQ scheduling of QoS, where multiple queues need to be checked to figure out which queues need to be scheduled based on sequence number property of queues.  If the sequence numbers are arranged in array form, then SIMD can be used very effectively.
• Crypto operations.
• Big Number arithmetic which is useful in RSA, DSA and DH operations.
• XML Parsing and schema validations.
• Search algorithms -  Accelerating compare operation to find matching entry from collision elements in a hash list.
• Check-sum verification and generation:  In some cases Ingress and Egress In-flow accelerations can't be used to verify and generate the checksums.  One example is,  TCP and UDP packets that come in IPsec tunnel.   Since the packets are in encrypted form,  ingress and egress accelerators will not be able to verify and generate checksums in inner packets.  Even packets that get encapsulated in tunnels will not be able to take advantage of Ingress & Egress in-flow accelerations.  Checksum verifications and generations need to be done in software by cores.  SIMD would help in those cases tremendously.
• CRC verification and generation:  These algorithms are not very expensive to have In-flight acceleration and not inexpensive for core to do.  SIMD in these cases help as it does not involve any architecture changes to the software and still get lot better performance over the cores which don't have SIMD.

Normally SIMD based cores give at least 50% more performance improvement for typical workloads.  So, as a software developer, figure out the ones that can be improved using SIMD and modify the code to improve performance of your application.

Speculative Hardware Data Prefetching & Software Directed Prefetching

This feature fetches the next cache line worth of data from the current memory access in the hopes that software would use next memory line.  Many core technologies provide control on enabling and disabling this at run time.  Software can take advantage of this while doing memory copy, set and compare operations.  Any data is arranged in linear fashion in the memory (such as arrays) can get good boost of performance with this feature. Note that, if this feature is not used selectively and carefully, it might even give degradation in performance. Be careful in using this feature.

Many cores also provide special instruction to warm the cache given a memory address. Software developers know the kind of processing (next module) and many times next module session context is also known. In those cases, software can be developed such a way that next module session is prefetched while packet processing happens in current module.  When the next module gets the control of the packet, it already has session context in the cache which avoids getting it DDR in serial fashion.  My experience is that using software directed prefetching gives very good results.  This also ensures that the performance does not go down even with large number of sessions.

Some Multicore SoCs provide support for Cache warming on the incoming packets.  As part of making packets ready for the software, these SoCs warm the cache with some part of packet content,  annotation data containing parsed fields and software issued context data.   When the software dequeues the packet, most of the information required to process the packet of the module that is getting hold of packet is in place in the cache, thereby, avoiding on-demand DDR access.  Software can program its context on per queue basis.  Note that, this feature is useful for the first module that receives the packet.  Actually that is good enough as this module can prefetch the next module context while the packet is being processed in the current module.  As long as each modules does this, there is no performance degradation even with high capacity. 

As described before,  hardware queues may not have one-to-one correspondence with the receiving module session contexts.  A queue might be having packets for multiple session contexts. Many times, software maintains the sessions in the hash table with large number of hash buckets.  All collision sessions are arranged in linked list or RB tree.  Software can ensure that there are as many queues as number of hash buckets and program the first collision element in the queue.  If the matching context is not same as the one that was programmed, then one might not get the full benefit of cache warming by the hardware. But if there are 4 collision elements and the traffic across these four are same, cache warming would come in handy 25% of the time. Some software developers might even store the collision elements in an array and program the array to the queue.

Software directed prefetch works very well as long as there is one-to-one correspondence between current module session context and next module session context.  In this case,  current module session context can cache the reference to the next module session context and use this to do prefetch operation.  This scheme also work fine if next module context is super set of multiple current module contexts.  But it does not work well if the next module context is finer granular.  Example:  Ipsec SA transfer packets from  multiple firewall/NAT sessions.  In this case, 'Software Directed Ingress In-flow acceleration' method can be used to direct the hardware to send the packet to next module.  This method not only provides cache warming, but also distributes the processing to multiple cores.


Hardware Page Table walk:

Some cores provide nested hardware page table walk to find out the physical address given the virtual address.  This is really useful for user space applications in Linux kind of operating systems.  Hardware page table walk feature is expected to be taken care by operating system vendors.  But unfortunately many OS vendors are not taking advantage of this feature.  As a software developer, if your Multicore SoC provide this feature, don't forget to ask your OS vendors to take advantage of this.  This will ensure that your performance does not go down when you move your application from Bare-metal environment (where the TLB are fixed and there is no page walk required) to Linux user space.

I hope it helps.

Fastpath Ipsec implementations - Developer integration tips on Inbound policy check

Basic purpose of Ipsec fast path implementations is to reduce the IPsec processing load on the main processing cores.  Since most of the Ipsec processing is same across different kinds of packets, offloading of this processing to hardware makes sense.

There are companies today who provide fast path implementations - either as software component or as an add-on card such as PCIe card that goes onto PCI slot of main processing unit such as x86 based mother board.

Software based fast path implementations are becoming quite popular in Multicore processing environment.  Fast path is run on some cores and rest of the cores are used for other applications.

Ipsec fast path implementations typically work as follows:

• Fast path typically owns the Ethernet and other L2 ports. That is, all packets come to the fast path plane first.
• If there is enough state information to process the packet,  fast path implementations act on the packets without involving normal path running in main cores. Packet might even get transmitted out after working on the packet.  If the packet requires some other application processing that is not present in the fast path,  then the packet is handed over to normal path processing unit.  In case of Ipsec fast path,  decrypted packets are  given to the normal path in inbound direction. In outbound direction,  it does Ipsec processing before packet is sent out.  

Basic purpose of fast path is to save CPU cycles so that it can do some other processing.
All fast path implementation from different vendors are not created equal.

In this post, I specifically would like to concentrate on 'Inbound Policy Check'.  Some fast path implementation skip this check. Reasons given by vendors of the fast path implementation typically is that this is done for performance reasons.  Some people believe that it can be done without any implications on security too.   Unfortunately, that is not true.


What is inbound policy check?

Inbound policy check ensures that the decapsulated IPsec packets used the SA that was formed for this traffic.   And also it ensures that the inbound policy rules allow this traffic to come through.


What are the issues if the inbound policy check is not done?

I can think of two issues -  DoS attack &  Allowing the traffic that is supposed to be denied (Access control violation).

DoS attack:

Let us assume that  a corporation gateway  has two tunnels to two different partners - Partner1 and Partner2. Without inbound policy check, it is possible for partner1 to interfere with the sessions/traffic between corporate gateway and partner2. That is, partner1 can create denial of service attack on partner2 traffic.  Even though I have taken the example of partners, this kind of attack is possible among IPsec remote users.

Let us assume this scenario:

10.1.10.0/24-----------SGW---------Internet-----------PSGW1-----------10.1.11.0/24
                                                             |
                                                             |-----------------PSGW2------------10.1.12.0/24


SGW:  Security Gateway of a corporation - It is protecting network 10.1.10.0/24
PSGW1:  Partner 1 Security Gateway - Its LAN is 10.1.11.0/24
PSGW2:  Parnter 2 Security Gateway. Its LAN is 10.1.12.0/24

There are two security tunnels from SGW - One to PSGW1 and another to PSGW2. Let us call them Tunnel1 and Tunnel2 respectively.

Tunnel1 is negotiated to secure traffic between 10.1.10/24 to 10.1.11.0/24.  Tunnel2 is negotiated to secure traffic between 10.1.10/24 and 10.1.12.0/24.   Let us also assume that the Tunnel1 SPI at the SGW is  SPI1 and the Tunnel2 SPI at the SGW is SPI2.  

It is expected that any packets coming from PSGW1 are expected to have SPI1 in its ESP header and inner IP packet SIP address  be one of addresses in 10.1.11.0/24 and DIP address  be one of 10.1.10.0/24.  Similarly it is expected that any packet coming from PSGW2 is expected to have SPI2 in its ESP header and inner IP packet SIP address be one of addresses in 10.1.12.0/24 and DIP address  be one of 10.1.10.0/24. 

Now to the attack scenario:

If PSGW1 network sends the inner packets whose IP addresses is other than 10.1.11.0/24 and 10.1.10.0/24 , then the SGW is expected to drop those packets.  SGW can only drop the traffic only if SGW does the inbound policy check.  If PSGW1 is allowed to send any inner packets, then it is possible that PSGW1 and its network can misuse this by sending packets with inner packets with IPs of PSGW2 LAN and SGW LAN.  Since it sends the traffic on the right SA using its own SPI,  SGW IPsec packet processing will happen smoothly.  If no other check is done, this traffic can go to SGW LAN.   Based on type of traffic, different attacks are possible.  Some of the attacks that are possible are:

• If attacker at PSGW1 guesses the TCP ports of some long lived sessions between PSGW2 network and SGW network, it can send the RST packets or ICMP Error messages to terminate the connections.
• Attacker at PSGW1 can send ICMP Echo message to SGW1 LAN network multicast IP address with SIP as PSGW2 LAN machine.  Replies from all machines in the SGW1 LAN go to PSGW2 victim machine and overwhelm the machine.

If SGW checks the inbound policy after the IPsec decapsulation is done using inner IP packet,  then it would have found that the SA used for the matching inbound policy is not same as the SA used to decapsulate the packet.  Whenver there is any mismatch, it is expected to drop the packet.  Due to this, no malicious traffic would have gone to the SGW LAN in above scenario.  Also, by logging these events,  administartor can find out the misbehaving peer security gateway and take appropriate out-of-band action.

Access Control Violation:

This is one more problem that can be faced if inbound policy check is not done.
Many Ipsec normal path implementations provide facility for administartors to add multiple rules with different actions to the secuirty policy database (SPD).   Rules normally have 5-tuple selectors in ranges/subnets/exact IP addresses for source and destination and ranges/exact values for UDP/TCP ports.  Actions can be one of 'Bypass', 'Discard' and 'Apply'.   Rules are arranged in a ordered list.  During packet processing,  rule search is done.  Rule search is stopped upon match.  Action specified on the matching rule is taken.  If the action is 'Bypass', then the packet is forwarded without any Ipsec processing.  'Discard' action indicates the packet is to be dropped.  Apply action indication that Ipsec processing is to be applied.  Normally administartors configure the rules with respect to outbound traffic.  Inbound policy rules are created automatically by the system from outbound policy rules by reversing the selectors - That is SIP becomes DIP and vice versa. Similary SP becomes DP and vice versa.

Now let us look at the possible access policy violation with following example:

Let us take these two policy rules in outbound list:

Rule 1:  SIP:  10.1.10.0/24  DIP 10.1.11.0/24  Protocol UDP   Action :  Discard
Rule 2:  SIP:  10.1.10.0/24  DIP 10.1.11.0/24  All Protocols   Action : Apply.

Inbound policy rule list would look like this:

Rule 1:  SIP:  10.1.11.0/24  DIP:  10.1.10.0/24  Protocol:  UDP   Action : Discard
Rule 2:  SIP:  10.1.11.0/24  DIP: 10.1.10.0/24  Protoco:  All   Action:  Apply.

Administartor creates the rules in above fashion to indicate that to discard any UDP traffic between the networks, but allow everything else by securing the traffic.

Assume that above policy rules are created in SGW1.

10.1.10.0/24----------SGW1----------Internet-----------SGW2-------10.1.11.0/24

When a TCP packet is sent from the SGW1 LAN and SGW2 LAN, then second rule gets matched and SA is created to allow traffic 10.1.10/24 to/from 10.1.11.0/24 for all protocols.  If SGW2 either misconfigured or intentionally sends UDP traffic in the SGW1-SGW2 tunnel,  then SGW1 is expected to drop the packet even if it successfully decrypts and decapsualtes the packet.  This can only happen if the SGW1 does the inbound policy check on inner IP packet.

If SGW1 does not do any inbound policy check, UDP traffic would have been passed to the 10.1.10.0/24 network thereby violating the access rules configured by the administartor.

I hope I could give good reasoning on why inbound policy check is required.  Some fast path implementation don't do this.   So, as a development integration engineer, please ensure that not only your implementation, but also fast path implementation does all the checks that are required.

Comments?