Hibernate Versus JDBC

Long back I worked on a project in Java, Centralized Management System, to  manage multiple networking devices.  At that time  I remember having a  big debate on whether to use Hibernate for Java Object persistence or use JDBC directly.  We have finally decided to use Hibernate at that time.

I happened to come across this link which talks about 'when to use Hibernate' and also provides advantages and disadvantages of using hibernate.  I felt that it captures many points. Hence I thought of sharing this.

Please find it here : http://www.mindfiresolutions.com/mindfire/Java_Hibernate_JDBC.pdf

One point  to consider by developers when using  Hibernate:

• Different parts of application require different fields from a database table or from a set of related tables to be persistent.  Don't try to put all possible fields from database in one single java object class. Some database fields are more often required for your application than some other fields.  In those cases, you are better off using multiple persistent classes - for frequently used fields and for rarely used fields.  If you go with one single java persistent class, then you may be using more heap memory than necessary.   You know your application. Decide on the number of java persistent classes based on 'type of data',  'how often they are required' for applications etc.. For example, some data which is needed for auditing, which is once-in-a-while activity, may require separate java persistent class so that this object can be instantiated and removed on demand basis.  

Features I like in Hibernate (JPA):

• It separates out the database organization with the business logic.  Due to Object-Relational mapping,  business logic always deals with the java class.  Mapping of java class members with the fields in database tables is defined in the XML file.  Any change in the database organization requires change only in the XML file.  
• Change in the database server from vendor to another during field upgrade or during development has no or very less impact.
• Automatic Version  & Time stamping feature of Hibernate ensures that there is no unintentional update of database by one thread using older data.

Above link has lot more information on the benefits. 

Cloud Multi-Tenancy Support - Authentication, Authorization and Auditing requirements

Authentication and Authorization requirements are common in any web based applications.  Authentication is the process by which the application takes the user credentials, typically using 'log-in' forms and checks the existence of user by checking against user databases (LDAP,  RDBMS, RADIUS etc..).    Authorization is the process in which application allows the access to different parts of application based on type of user accessing it.   In typical application, authorization consists of - Assignment of roles to users as part of creating user in the user database (LDAP, RADIUS etc..).  Mapping the part of application (resource)  to the roles and providing set of permissions to the roles to access mapped resources.  Resources and related permissions for different roles is also called 'Access Control'. 

Many web application development frameworks have minimized the amount of development one needs to do for Authentication and Authorization.   For example, spring-security component of spring framework (http://static.springsource.org) takes care of Authentication & Authorization requirements of many web applications. That is,  for many web based applications,  web developers only need to concentrate on the business logic and can leave the Authentication and Authorization part to the spring-security.  

Cloud applications', mainly SaaS applications',  Authentication & Authorization requirements can't be met by spring-security (Version 3.0.5) as is.  But spring-security provides multiple hooks for extending it.

It is good to know the typical requirements of Cloud applications with respect to Authentication and Authorization first.  This is what this post is concentrating on.  I will try to discuss the how to bridge the gaps in spring-security to enable cloud application in future posts.

Authentication Requirements

Like any web applications,  Cloud applications also require some portions of the application resources to be accessible only to some set of users.  Authentication  is the process where it takes the user credentials using 'login' forms and validates the user by communicating with pre-created user database.   Cloud applications differ from the traditional web applications where cloud application provides services for multiple tenants. That is cloud application resources are instantiated for each tenant.  Rather than having separate server and separate application instance for each tenant,  cloud application services multiple tenants using one server and one application.  Typically, this is achieved using 'Tenant ID" in application and in database tables.  When single tenant based application gets converted to multi-tenant application,  one would see the introduction of 'Tenant ID" in each database table schema.   Note that the tenant is not equivalent to user.  Tenant is typically a company to which the SaaS application provides the services for.  Each tenant would have its own set of users - typically employees to access  its resources in the cloud application. 

Some of the characteristics of Cloud applications with respect to Authentication:

• Tenant identification :  In many cloud applications,  email address is taken as the login ID.  Email address consists of user name and domain name.  Domain name is, typically , used to identify the tenant instance.  If email address is  xyz@example.com, then xyz is user name and example.com is the domain name.
• Each tenant has their own authentication database (user database) :  Cloud application vendors are increasingly allowing  their customers (tenants) to provide  authentication server information.  It would eliminate the need for duplicating the user accounts in the cloud applications by companies (tenants). Cloud applications are expected to get the appropriate authentication server information from the domain name of the login and validate the user by communicating with the authentication server.  Since tenants might use different types of authentication servers and hence it becomes requirement for cloud application to support multiple authentication protocols. 

Any authentication framework for cloud applications should satisfy following requirements (this list is beyond what is available in spring-security in its 3.0.5 version)

•  Ability to identify the tenant from the login credentials
• From domain name of email address.   Note that  multiple domain names might be part of one tenant ID.  That is, one should not assume that there is one-to-one correspondence between domain name and tenant ID.   Multiple domain names for a given tenant ID can arise  for multiple reasons -  Mergers of companies where domain names of merged companies exist for few months after merger,  Multiple divisions of company may be represented in the domain name portion of the tenant.  
• Ability to identify the tenant ID using application specific mapping table - Domain name versus tenant ID.
• As a separate POST variable indicating the tenant ID.
• Ability to get the Authentication Server information from the RDBMS where the authentication server information is stored based on the tenant ID.  Authentication Server information based on the tenant ID is part of the application specific RDBMS table.
• A tenant might have multiple authentication servers. One can apply different strategies to select the authentication server.  Some of the strategies that are useful are:
•  Round-Robin:  Yet times, the tenant has multiple authentication servers either for load distribution and for high availability.  In those cases, this strategy might be chosen.  If this strategy chosen,  servers are chosen on round-robin fashion.
• Until authentication is successful :   Multiple authentication servers might be provided in order of priority.   If this strategy is chosen,  authentication servers are tried in the roder until authentication is successful or until all authentication servers are tried.
• Sub-domain based authentication server:  If this strategy is chosen,  then the set of authentication servers is chosen based on the domain name of email address.  That is,  domain name is not only used to determine the tenant ID, but also is used to get the list of authentication servers.  Then the above strategy can be applied on this list of authentication servers.
• Ability to support multiple authentication protocols:  As discussed above,  Cloud application vendor will not be able to mandate their customers (tenants) to go with one kind of authentication servers.  Though LDAP and Active Directory (which also can be accessed using LDAP) are most commonly used, it is necessary that framework allows cloud applications work with  multiple authentication protocols.   It is expected that the information regarding communication protocol such as x.509 certificate and Certificate chain in case of SSL protocols would be expected from the application specific tables in RDBMS.  This information can come along with authentication servers' information. 
• LDAPv3 with/without SSL:  I believe SSL is must, as one would not like to send the user credentials in clear on the wire.
• CAS (Centralized Authentication Service)
• SAMLv2  (IDP is fast becoming common choice of authentication in recent past).
• OpenID
• And more ...  (Though RADIUS is one popular protocol, I don't see it being used that often, may be it is due that RADIUS is based on  UDP and related security concerns).
• Firewall Traversal while communicating with authentication servers:  Some companies (tenants) though like cloud applications to authenticate their users with company authentication database, they don't like to open the firewall hole for cloud applications to make a connection to their servers.  Some company administrators are paranoid to create an inbound hole in their firewall. But they are fine with creating outbound hole. In those cases, it is required that framework provide a proxy that can be installed in the company premises behind firewall.  Proxy is expected to work as proxy between cloud applications and internal authentication servers.  Proxy is expected to make a persistent connection to the cloud application server all the time.  Cloud applications, rather than communicating with authentication server directly,  it sends the authentication messages to the proxy on the established connection and proxy in turn makes the connection to the authentication server and proxies the messages to the authentication server. Similarly, it receives the messages from the authentication servers and forwards them the cloud application server using pre-created connection. 

Authorization Requirements

Traditional single-tenant based web applications can work with authorization functionality provided by spring-security.   Method/Class based authorization and page level authorization is good enough for many single-tenant based web applications.  spring-security provides mechanism to associate method/page/class with granted authorities.  As we discussed earlier, each user is associated with the roles in the authentication database.  As part of authentication process, in addition to validation of user with the authentication database, the authentication process also gets the authorities (Roles) of the user and keeps it in the security context.  At later time during authorization phase, this information is used by spring-security.

As I understand spring-security in later versions introduced the concept of ACL and domain objects.  This seems to be promising.  Using this new framework,  application business logic can create ACL which consists of 'Domain Object',  'Roles' and 'Permissions'.  Domain object is reference to the application business level object.  'Roles' is list of roles that are allowed to access the domain object and 'Permissions' is set of bits indicating permissions given to the roles to access the domain object.  Though this seems to be promising for cloud applications,  it still does not meet the requirements of cloud applications.

Characteristics of Cloud applications with respect to Authorization:

• There are multiple types of users in Cloud applications
• Users of Cloud application vendor.   These users are typically administer the cloud application as a whole.  There are different users with different roles to operate different parts of the application.  
• Users of tenants:   Cloud applications, as discussed above, support multiple tenants.  Each tenant has multiple employees.  Based on the type and authority of each employee, access to different resources of the application would be permitted.    Each tenant might define different roles.  That is, role names across different tenants might not be same in the system.
•  Each user is configured with the roles in the authentication database.
•  Support for Collaboration :  Some (not all) cloud application require a resource of one tenant to be accessible to other tenants.  Hence some cloud application provide ACL for a given domain object with not only owner tenant ID, but also with respect to other tenants.  That is, ACL is expected to consist of multiple rules for a given domain object with each rule having tenant ID and permissions.

Any authorization framework for cloud applications should satisfy following requirements

•  Ability for application to specify (add/delete/modify) the ACL for needed domain objects:
• As discussed, each ACL record in the ACL table corresponds to one domain object.  To allow multi-tenancy,  Domain object needs to be represented by tenant ID,  application name of the domain  and domain object identification.  Each ACL itself consists of multiple rules - Access Control Rules.  They can be represented in a separate table with each row corresponds to one rule.  Rule consisting of  "Tenant ID"/'Group of tenant IDs",  "Roles of that tenant ID" that can access the domain object, "Permissions" on the domain object.  Due to Collaboration based Cloud applications,  tenant ID in the rule can be other tenants than the tenant of the domain object in the ACL.   To simplify the configuration of collaboration related AC rules,  tenant IDs may be grouped and groups can be reused across multiple AC rules of different ACLs.
• Ability for business logic to find out whether the authenticated principal and his/her authorities can access the domain object that is being accessed. 
• Performance considerations:  Since it is possible to have ACL for many domain objects,ACLs can become very big in the system.   It may be necessary to have reference to the ACL from the domain object. This could avoid costly query of the ACL table.  
• Ability to inherit the ACL from the parent domain object: 
• Cloud applications have different resources and configuration of ACL and associated rules can become very complex.  Inheritance of ACLs can help reduce administrator overhead.   Domain objects that are being configured with ACLs can inherit the ACLs of its parent or grand parent or any other resource. It does not seem to be good to allow inheritance of the ACLs from domain objects of other tenants.  To allow inheritance, ACL for the domain object can be configured with list of  'Inherit from" and the identification of the parent domain object - Application name of the domain and its identification of the domain object.  In addition to inheriting it from the other ACLs, it can also its own Access control rules. 
• Due to inheritance,  there could be multiple ACLs to search for a given domain object - Its own ACL,  multiple ACLs as it can inherit from multiple other domain objects.  Also Domain objects, from which ACLs are inherited from, may themselves be inheriting ACLs from others.  Due to multiple ACLs, order of ACL to find the permissions is important. It is always good to check its own ACL first and then first level inheritance and second level inheritance and so on.  

Auditing Requirements

Any configuration changes to some important resources of an application must be logged for later auditing.  Since cloud applications have multiple tenants,  tenant ID must be logged along with configuration audit logs. 
Configuration audit logging must have information related the exact changes that were applied to the resource.  Addition of any application record must have all the information (field names and values) in the audit log.  In case of deletion of any application record,. audit log  must have similar information as 'Add'.   Modify operation should result in audit log having old information and new information. 

It is also important that the audit log also contains Date & Time at which the update happened,  User name of the user that changed the configuration,  role that was used to give the permission. 

Internet Resources on Multi-tenancy and spring-security.:

I have found two valuable resources related to Multi-tenancy and spring-security.  It gives a good picture on how spring-security can be enhanced to support multi-tenancy.  Though they solve some of the requirements mentioned above, they don't  satisfy all of above requirements. But it gives  very good understanding of how spring-security can be customized.   This knowledge can be used to customize spring-security for above requirements.

Securing a multitenant SaaS application 
Extend Spring Security to Protect Multi-tenant SaaS Applications

Experience with installing VMware Server 2.0 on windows 7

I was helping a friend to install Ubuntu 10.10 Linux on Windows 7 using VMware server 2.0.

I  had few hiccups, but finally could do it.

• Installation of VMware Server was a breeze.  No issue found in downloading and installing.  Just follow the instructions given in VMware site. 
• After VMware Server was installed, you would see few entries  at "Start"  of windows 7,  All Programs->VMWare->VMware Server->VMware Server Home Page.   Click on it.   It opens your browser.
• It asks for user name and password.  This is where I was stuck.  I was giving all my windows 7 user names and passwords, it could not succeed.  Finally, I found from searching VMware site that VMware server always expects the user name "Administrator".  By default windows 7 does not enable administrator account with "Administrator" name.  To do this, you need to execute following command at the command line prompt. 
• Command window must be opened with "Run as Administrator".  You can do this by going to "All Programs->Accessories".  Here you would see "Command Prompt".  Take your cursor there and press right click on that line.  You would see "Run as Administrator".  Click on that. It opens the command prompt window.
• There you should type "net user administrator /active:yes".  This would be enable hidden "Administrator" user.
• Next step is to assign the password to "Administrator" user.  Go to 'Control Panel->User Accounts and Family Safety->Add or remove user accounts".  Here you would see "Administrator" user. Click on it and add a password to it.
• Now you should be able to login in to the VMware server using "Administrator" and password you have set for the user.
• Then I was trying to install Ubuntu.  First step is to create a virtual machine. You could do this using "Virtual machine" menu.  Click on "Create Virtual Machine" item.  It takes you to set of screens to assign the memory,  hard drive space,  network etc..  These are all simple and you could follow the documentation.  As far as the Ubuntu image is concerned, I thought I could have iso image in my windows hard drive somewhere and could point to that. But I was not successful in giving the path for ISO image anywhere.  Finally I decided to burn the ISO image on CD and selected 'DVD' option.
• So far so good.
• Once the virtual machine is created, I started it.  I wanted to see the console of virtual machine. Selected "console" menu.   It asked to install a plugin.  Clicked on it, it installed the plugin. Then it asked me to click on anywhere on the window to open the console.  Unfortunately, I keep getting error that "request timed out".  Finally, I found that Firefox can't be used to open the console.  So, make sure that you always use "Internet Explorer".  Once I started using Internet Explorer, I could see new console window getting opened.
• After that,  ubuntu installation started on that console window and now I have ubuntu working on Windows 7 using VMware 2.0

Hope it helps.

Srini

IGMP Filtering - Developer tips

IGMP protocol allows hosts to report their interest in Multicast address membership with adjacent routers.  These adjacent routers in turn propagate the consolidate membership with upstream routers using PIM-SM or using IGMP proxy functionality.

IGMP protocol sits right on top of IP layer, at the same level as ICMP, UDP and TCP.  IGMPv1 and IGMPv2 protocols are older protocols to IGMPv3.  IGMPv1 and IGMPv2 join the multicast membership by providing Multicast address in IGMP report message.  IGMPv3 takes one step further and even ask for membership with respect to source.  That is, there could be  multiple Multicast sources  sending same streams using one Multicast address. Hosts have choice of receiving the Multicast traffic only from some specific sources - It can specifically include the source addresses in the report message or can exclude some specific sources.

Typical firewalls today don't have capability to restrict the hosts on a specific interface from joining specific Multicast stream.  Firewalls today can allow or deny the IGMP packets on a specific interface though. But this is not sufficient.  It should be possible to restrict a given multicast stream on a specific interface.  This is possible only if firewall on the router deny only IGMP reports having restricted Multicast addresses.

IGMP filtering functionality in routers expected to provide following functions.  Some vendors call this 'IGMP filtering for Multicast authentication'.  I never understood why they use term 'Authentication' though.

• Provide facility for admin user on per interface basis 'Allow Only List' or 'Disallow list'.  Each list contains multiple records. 
• Multicast Address,  Source address.
• Source address can be 'ALL'.  

Some implementation considerations:

It is always good to implement this module separate from IGMP Router or IGMP Proxy modules.  In case of IGMPv1 and IGMPv2,  each membership report message only contains one Multicast address.  If this multicast address is to be restricted,  then the complete message can be dropped.  In case of IGMPv3,  one membership report message can contain multiple Group records with each group record having multicast address, source addresses and qualifier to all the source addresses listed in the group record - include or exclude.  IGMP filter module needs to do quite a bit of work to identify the group records and corresponding sources and remove only the affected ones from the message.  Then rest of the message should be allowed to pass through.  If the complete Multicast address is restricted as per configuration, then the complete Group record from the IGMP report message can be removed.  It is not as simple as though.  From the configuration, some times only specific source addresses are to be removed from the group record in the message. That is, IGMP filter is expected to selectively remove not only group records, but also source addresses selectively in the group records.

Clustering of devices with traffic distribution by L2 Switch - One limitation & Mitigation

In my last post on "Data Center/Enterprises Clustering of Devices"  I discussed on how L2 switches are enabling device equipment vendors to provide cluster solution to take up the increasing load on the networks.  Many L2 switches are capable of analyzing multiple different types of  layer 2 headers to get to the inner IP packet and use inner IP packet source and destination IP address fields to distribute the traffic across multiple devices in the cluster using hash distribution.  L2 switches typically understand Ethernet and MPLS related headers such as Ethernet DIX,  LLC/SNAP,  802.1Q VLAN headers and MPLS Label headers.  That is, L2 switch can get hold of IP packets if packets are sent over above mentioned L2 headers. 

In some deployments, IP packets may be encapsulated in multiple L2 headers or L3 tunnels.  Some examples are : PW (Pseudo wire) header,  Ethernet over Ethernet using PW,  GRE/UDP,  GTP/UDP, IPinIP,  Mobile IP and may more.  L2 switches in the market today are not capable of understanding these headers to to get to the inner IP header.   In these cases, distribution based on inner IP header fileds will not be possible.  In these deployments, L2 switches may need to resort to distribution based on L2 header fields such as source and destination MAC or tunnel IP header fields.  Unfortunately,  distribution based on these fields may not be good at all.  If you take an example of this cluster being places between two routers,  MAC addressees of every packet traversing the cluster will be same.  Hence any distribution based on the MAC addresses would go to only one device in the cluster.  Similar would be the case, if distribution depends on the tunnel IP header fields. 

Switches have one capability though.  They can generate the hash based on calculated CRC on some part of the packet or CRC of the Ethernet packet.   CRC of the Ethernet packet can be assumed to provide good distribution as CRC on Ethernet packet is based on the complete Frame payload, that is, including the inner IP packet payload. Switches have capability to take few bits of CRC to distribute the packets across multiple cluster devices.  But the issue with this is that packets belonging to same connection would go to different cluster devices.  Base on my earlier post, cluster devices assume that packets belonging to a connection would always land on the same device. This assumption is no longer true if the cluster solution is being deployed in above mentioned environments.  These types of environments are not common at all in Data Center and Enterprises environments.  So, this problem may not be there in many instances. But service provider environments, multiple L2 and tunnel header situations are not uncommon.

How do cluster solutions work in these environments?

In these environments, if CRC based distribution is used,  switches are really doing packet sprinkling across multiple devices.  In these cluster devices should have additional intelligence.  

• Cluster devices should be able to get past the headers to get to the inner IP header.
• Cluster devices among themselves should have understanding of session distribution. One simple method is to do what switches were doing.  That is, they can generate the hash on the IP header fields (source and destination IP) and figure out the device which got the packet is the one which needs to server based on the hash value.  If it is,  it should continue processing the packet. If it is not, then it should give the packet to the device that owns the hash value.  

There could be good amount of traffic among cluster devices.  They can use the switch as their back plane to send and receive the traffic among them.  To avoid other device doing same thing, that is getting hold of the inner IP header and hashing on the inner IP header fields again,  the sending device can send this information along with the packet and receiving device can avoid doing same operations again.

As indicated above,  only few deployments where L2 switch does not do inner IP header based distribution.  If same cluster solution being provided for all kinds of deployments, then it is good for network equipment vendors of cluster solution to provide configuration on the cluster whether the packets which are being distributed by L2 switch is packet sprinkler or intelligent IP flow based sprinkling.  If it is packet based sprinkling, then additional logic in devices can kick in to figure out the real destination device.

Data Center/Enterprises - Clustering of Network devices

Throughput requirements of Data center/Enterprise network equipment are going up with increased traffic in data centers and Enterprises.  In addition,  computational requirements of network equipment are also going up.  Some examples of why more computation power is required.

• Intrusion Detection/Prevention now requires almost 3 - 4 times the  computation power on per Mbps of traffic than what was required few years back.  I guess it is mainly due to sophisticated nature of attacks and evasion techniques adopted by attackers.  Javascript analysis itself takes 10 times computational power  than the typical DPI based pattern matching.  Javascript analysis requires proxy based functionality to get hold of the javascript and script analysis for attack detection.  These two tasks require lot more CPU cycles than typical pattern matching.
• Traditional Server Load Balancers (SLB) used to select the internal server based on the IP, UDP/TCP header values.  Next generation server load balancers (SLB) called ADCs do deep packet inspection, such as HTTP, SIP URL,  HTTP Request headers,  to select the internal server to send the load.  DPI requires more CPU cycles.
• Application Firewalls such as Web Application Firewalls and SIP firewalls not only  do the deep packet inspection, but also deep data inspection and that  requires more horse power from CPUs.
• DDOS prevention requires real time analysis of not only packet-by-packet analysis, but also sessions and application protocol level analysis across sessions to identify the attacks.  Many DDOS attacks on per session basis look exactly same as the normal traffic.  Hence the analysis across sessions is required for detecting the anomaly.  This capability requires not only lot of memory, but also good amount of computational power.

Multicore processors are helping some extent in solving performance issues.  Clustering of multiple Multicore SoCs are becoming necessary to solve above performance issues in Data Center and Large Enterprise markets.  Typically,  multiple blades, each using Multicore SoCs, running the same application are clustered to take up the load. L2 switches are increasingly used to front end the cluster.  L2 switches now can be configured to balance the load across multiple devices of cluster.  One might see the cluster and L2 switch in one enclosure giving a feeling that it is one big box providing tens of gigabits of performance.

What features of L2 switch are enabling clustering?

• Distribution of sessions across multiple devices in cluster:  Majority of L2 switches have capability to distribute the  traffic coming from incoming ports (Data Ports) across multiple ports (Device ports).  By connecting devices in the cluster to these ports, then each device gets the traffic that was redirected to that port. But many of the network devices expect that all packets of any given session go to the same device.  For example, all packets belonging to one HTTP connection should go to one device.  If packets of the sessions are distributed across multiple devices,  they will not be able to do their operation of analysis, proxy etc..    A given connection traffic involves both Client to Server and Server to Client traffic.  Though L2 switches don't have session intelligence, due to the hash based distribution mechanism they adopt,  same hash value gets generated for session traffic whether it is C-S or S-C traffic of a connection.   Some cautions:
• L2 switches don't do IP reassembly.  Due to this, hash generated for first fragment of a packet can be different from the non-initial fragments if the hash generation block is configured with L4 fields (TCP source and destination ports).  So, it is advisable to configure the hash block with IP addresses and IP protocol.   This may give rise to unequal distribution. But with large number of sessions in DC, this may not be a big limitation.  
• Some application sessions require multiple connections. Example:  SIP (Session Initiation Protocol).   SIP voice call typically involves three connections - SIP control connection,  RTP for voice/video data and RTCP for control frames.  Many devices expect that all three connections land on the same device.  If all three connections have same source, destination and protocol fields, then all packets of SIP application session would be sent to the same device by the switch.  But, RTP and RTCP IP addresses may be different from the IP addresses of SIP control connection.  If your device needs to support this,  then it is responsibility of cluster devices.  Cluster devices need to have intelligence of ownership of these kinds of  application sessions. If a device receives packets belonging to application that is owned by some other device, it needs to redirect the traffic to that  device that owns the SIP session.
• As indicated implicitly above,  L2 switch port are divided into - Network ports (Data ports) that connect to the DC/Enterprise networks and Device ports where the cluster of devices are connected.  With large density of ports in current generation of switches, some ports even can be dedicated to inter-device communication, there by avoiding any other back plane such as infiniband or some other L2 switch fabric.  L2 switches and devices providing ETS (802.1qaz)  and 10G ports'  support can use the same port for both inter-cluster communication as well as for network traffic. 

New Generation Configuration Framework

Even though there are multiple devices in cluster,  it is required that admin user configures the cluster only once.  Admin users should not be expected to configure each device in the cluster.   Fortunately new generation of configuration framework are designed to handle cluster configuration. 

New generation configuration frameworks support the mechanism to ensure that configuration is same across the devices in the cluster.  Increasingly,  configuration architecture supports central management system which takes care synchronization of configuration across devices on per operation basis.

Network devices maintain several statistics. With multiple devices in the cluster, each device maintains its own set of statistics. Admin user typically expects to see the consolidated list of statistic counter values across all devices in the cluster. Again, new configuration frameworks reads the statistics from each device, consolidates them and show the consolidated output. 

On image upgrade : When new image version is available,   new configuration frameworks allow admin users to upgrade the image only once for the cluster. All devices in the cluster would get the image from the central configuration framework.

With these advancements in L2 switches and configuration frameworks,  clustering is again back in the networks.

Data Center Equipment & Offload card requirements

I  have been attending some web conferences focusing on Data Center equipment.  From last few years,  there are more and more discussions on offloading some processing out of current data center equipment. Before getting details on offload functions,  it is good to revisit some of data center equipment.

Data centers, whether they are private, public or cloud,  have following equipment.

• L2/L3 Switches and Routers
• Server Load balancers or Application Delivery controllers.
• WAN Optimization devices
• Network Security Devices.
• Monitoring Servers for network visibility, Surveillance and others.
• Application Servers (Web, EMAIL etc..)
• Database Servers
• Storage Devices.

At this time, all servers are x86 servers.   I gather that, many of the network equipment such as SLB, ADCs, WOC, UTMs are also implemented on x86 servers with Crypto, Compressions and Pattern matching done on add-on PCIe cards.  Some vendors of these equipment are seriously considering moving to embedded multicore SoCs.   There are two camps.  One camp of vendors want to continue with x86 for their applications, but use embedded Multicore SoCs for offloading some functions beyond algorithmic offloads.  Other camp of vendors would like to replace multiple x86 processors with multiple Embedded Multicore SoCs.   

This post talks about networking functions that equipment vendors looking to see in offload cards. There are two types of offload cards that are becoming popular - PCIe cards that can go into x86 based network equipment and  proprietary cards that go into L2/L3 switch. Both kinds of cards are expected to take away some processing out of networking equipment. If it is PCIe card, more offload functions are possible due to proximity with the target application.  In this post,  I am going to list down the offload functions that are possible and expected by the networking equipment vendors.

PCIe offload cards act NIC card to the x86 equipment while doing offloading of some networking functions.  Hence some call them  intelligent NIC cards. Offload card receives all the packets from the network, it processes offload functions and then sends the result packets to the server via PCIe interface. These packets are presented to the server TCP/IP stack via PCIe Ethernet driver.  Similarly outbound packets would come to offload card from the server.  Offload card does offload functions necessary on egress traffic and sends the result packets out on the network.

List of offload functions in different categories are given below.

Algorithmic Acceleration:  This is already being done and requires no further explanation.  Algorithmic acceleration now typically involves

• Public Crypto and Symmetric Crypto acceleration
• Symmetric Crypto acceleration with Protocol offload such as SSL Record layer : Though this is provided by offline cards,  increasingly network applications requiring SSL are using AES-NI SIMD functions provided Intel SSE4.  Many PCIe based crypto accelerators are mainly used for public crypto acceleration.
• Compression/Decompression  acceleration.
• Regular expression Search acceleration.

Basic Networking offloads functions:

• Offloading of ARP for IPv4 and ICMP based neighbor discover in case of IPv6:  As networks in the data center becoming flatter, there is more and more ARP broadcast traffic in the network.  TCP/IP stack of network equipment receives all broadcast packets.  Due to increased broadcast traffic,  CPU cycles are being spent in processing interrupts, and ARP processing.  Lot of times, ARP requests may not even be satisfied as there is no neighborhood entry.  Offload cards are expected to remove this overhead by responding to neighborhood requests by itself without involving the server TCP/IP stack.  How does it work?
• Server TCP/IP stack is expected to create/delete neighborhood entries with local IP addresses and associated MAC addresses  in the offload cards whenever new entries are created locally and whenever they are deleted.  Offload card maintains the neighborhood entries database on per physical and logic port basis (in case of VLAN based interface).
• When offload card receives the neighborhood request, it checks  its database first. If there is no matching cache entry, ARP packet is dropped.  If there is any matching entry, it processes the request and reply is sent to the network without involving server.  
• Offloading of packet integrity checks and dropping of invalid inbound packets:  Any TCP/IP stack including servers TCP/IP stack spend some CPU cycles checking whether the packet received is good.  These cycles can be saved if offload cards do the checks.  Some of the checks that offload card can do are:
• Check the SMAC of the packet :  If the source MAC is one of the local MAC addresses, then packet can be dropped by offload engine.
• Check the IP length with the packet size:  If the IP length is more than the packet size,  it is a bad packet and packet can be dropped.  If the IP length in header is less than the packet size, then packet needs to be truncated to length in IP header.
• Check the checksum of IP packet:  If the checksum  is bad, then offload card can drop the packet.
• Check the TTL:  If TTL is 0, then the packet can be dropped by offload engine.
• Check the SIP and DIP of IP header:  If both the addresses are same, then packet can be dropped by offload card.
• Check the SIP of IP header with local IP addresses:  If the source IP is one of local server IP addresses, then the packet can be dropped by offload card. Similarly, if the source IP addressis one of the local broadcast addresses or multicast address, packet can be dropped by the offload card.
• Transport header length checks:  If the length of the packet is not as per the length in the transport header, then the packet can be dropped by the offload card.
• Transport header checksum checks:  If the checksum of the packet does not match with checksum in the header, packet can be dropped by the offload card.
• When the checksums are verified,  to avoid similar checks in the server TCP/IP stack,  offload card can communicate the successful checksum verification indication along with each packet to the server TCP/IP stack.  Many server TCP/IP stacks such as Linux has capability to indicate this information in their buffers.  For example, Linux TCP/IP stack has capability to avoid checksum verification if some special fields are set in SKB.
• Offloading of IP reassembly on inbound packets:   Offload Engines by doing IP reassembly and sending one single packet to the server TCP/IP stack can save good number of CPU cycles. Any server terminating any kind of tunnel (such as IPsec, GTP, GRE, IPinIP etc..) would tremendously would benefit from this offload.  Offload cards are expected to do reassembly of IP packets that involve two packets at the minimum as many tunnel protocols make two fragments from the IP packet. IP reassembly function is not only required to save some CPU cycles in server, but also required to do other offloads such as 'Traffic policing',  'stateless and stateful filtering',  'inline DPI offload'  as they require access to transport selectors.  These offloads will not work well on packets if they don't have transport selectors.
• Offloading of IP fragmentation on outbound packets:  This offload again saves CPU cycles by moving this function from server TCP/IP stack to offload engine on outbound packets from the server.  Server TCP/IP stack can give PMTU value along with the packet.  Offload card can fragment the packet as per PMTU and send them out. By offloading this functionality to offload card,  offload card can do better job of traffic shaping if it involves transport selectors.  IP Fragmentation can be done after the traffic shaping there by allowing traffic shaper to look at full IP packet.
• Offloading of hash calculation  on inbound packets:  Many networking applications running in x86 server require hash result to index into  hash buckets of their session blocks.  Bob Jenkins,  CRC-32 and CRC-64 are some of common hash algorithms used on packet fields.  Offload cards are expected to take configuration from x86 server networking applications.  Configuration include 'hash algorithm', 'packet fields' and their order. Offload cards, then expected to, extract the fields, do the hash on the fields and send the hash result along with the packet to the server TCP/IP stack. 
• Packet parsing & Field extraction on inbound packets :  Almost all networking application require to extract some common fields from the packet.  Cycles required to extract the fields can be saved by offloading this part to the offload cards.  Server networking applications can program the offload card on the fields it is interested in.  Offload cards based on configuration extracts relevant fields and provide them along with the received packet.

Inbound Packet filtering

Non-security network equipment devices would normally service certain protocols.  Unrelated traffic that goes to the devices would only decrease the performance of the device. Offload cards can filter out the packets.  Device software can program the rules to allow only the traffic required to go to the devices. Offload cards can reject the traffic that does not match the rules programmed.

Traffic Policing of inbound packets

Network equipment devices run complex operations on the packets or data. For example, ADC/WOC/Security devices do SSL, TCP, HTTP and other application protocol proxying to act on the data.  Proxy operations are expensive in nature.  Even the best of equipment can't process more than few gbps. If more traffic is sent towards the device, packets would be dropped at some point and  it only worsens the performance of device.  Devices are typically rated for some performance for different types of traffic workloads.  Offloads cards are expected to provide traffic policing on different types of traffic and discard the packets that exceed the policing parameters.  Device software is expected to program the offload cards with different types of protocols and traffic policing parameters.

TCP stateless offload :

Based on some conversations with DC equipment vendors as well as offload card vendors,  it appears that majority of them don't like to have TCP termination in offload cards and corresponding TCP/IP stack bypass in the x86 device.  It appears that device vendors are happy with device Linux or NetBSD operating system's TCP/IP stack.  Only peripheral changes to the TCP/IP stack and new Ethernet driver is only thing allowed by these vendors due to offload cards.  Maturity of Linux TCP/IP stack,  constant upgrade of TCP/IP stack with latest standards (RFCs),  avoiding application porting or testing of system are some of the reasons cited.  

Due to this, offload cards are expected to provide only TCP stateless offload. TCP stateless offload functions increase the performance of proxy applications.

• Offload of hash function :  This is described above under 'Basic networking offload'.  TCP layer of device TCP/IP stack uses hash function to find the TCP control block (TCB).  If offload card does this, then device TCP/IP stack can skip this and save some CPU cycles.
• Large Receive Offload :  As described in the post "Linux TCP Large Receive Offload optimization to increase performance" LRO feature reduces the number of packets that are seen by the device TCP/IP stack. Lesser the number of packets, better the performance of TCP/IP stack. Offload cards can do this by implementing LRO function by aggregating multiple consecutive TCP packets of a flow of a connection into a single large packet.
• Transmit Segmentation offload :  Similar to LRO, this feature reduces the number of packets that traverse the device TCP/IP stack, but this time on the transmit packets.  Device TCP stack normally segments the packets based on MSS which is PMTU-TCPheader length - IP headerlength.   This would increase the number of packets on transmit direction and also some cycles are spent in segmenting the big data.  TSO functionality in offload cards can reduce the number of packets and also save CPU cycles from segmenting the packet in the device.  Device TCP stack can provide the big packet along with segment size and PMTU to the offload card. Offload card along with TSO and IP fragmentation functionality can segment and if required fragment the packets before sending them out on to the network.
• ACK offload : As we know,  TCP/IP stacks are expected to send at least one ACK for every two TCP segments it receives.  Hence, there would be large number of ACKs.  Similar to TSO,  reducing the number of ACKs traversing the device TCP/IP stack can increase the performance of device.  Device TCP/IP stack generates the template ACK with the number of ACKs to be generated and corresponding sequence/ACK number information to the offload card.  Offload card, then, can generate multiple ACKs from the template ACK.

Traffic Shaping of outbound packets

Many network equipment device types do some sort of traffic shaping on the outbound packets.  Traffic shaping is almost the last step on the outbound packets.  QoS or traffic shaping is quite the same in all devices and there is really not much a value addition by the device software.  Hence this function can be done by the offload cards without impacting any device functionality.  Traffic shaping policies can be programmed in the offload cards by the device software.  Offload cards can act on the outbound packets based on the configuration.  It is important to that offload cards support multiple scheduling algorithms and hierarchical shaping.  This function in offload cards can save good number of CPU cycles and hence more CPU power can be made available to the device applications.

IPsec termination offload

Some network equipment devices such as network security devices implement IPsec VPN.  For inbound packets,  IPsec is done in the early stages of packet processing. For outbound packets, it is done one step before traffic shaping.  Hence this is also good candidate for offloading along with the traffic shaping.  By offloading IPsec inbound SA processing,  device software only sees the clear packets.  Similarly,  by offloading Ipsec outbound SA processing, device software does not need to do the expensive IPsec outbound processing.   Device software is expected to program the tunnels (pairs of Inbound and outbound SAs) upon IKE negotiation and delete them when life time expires or when there explicit delete indication from the remote gateway.

Offload cards are expected to provide IPsec termination offload function.  Offload cards not only do outbound and inbound SA processing, they also expected to do inbound policy verification on both decapsulated inbound packets and also on the original clear packets.  On the outbound side,  offload function can expect device software to provide outbound SA reference along with the packet. 

Inline DPI offload on inbound packets

Network equipment devices having IDS, IPS and Application detection functionality require some kind of pattern match on the packets.  These applications analyze the TCP, UDP, IP payloads for some detections and for others it may require processed data of L7 protocols.  Offload cards have access to TCP, UDP and IP payloads and may not have L7 intelligence.  At least for TCP, UDP and IP payload based detection, offload cards can be used.  Device software programs the patterns in the offload card and its regular expression engine.  Offload card after doing all its operations on inbound packets such as 'IP Reassembly', 'LRO', 'Packet filtering', it does pattern match on the payload and sends the results along with the packet.  Each result is expected to have pattern ID, offset in the payload.  Device software can avoid pattern matching on basic protocol payloads and there by utilizing saved CPU cycles on other processing.

Multicore Distribution

Network equipment devices normally use multicore x86 processors.  Without any support of offload cards,  distribution of packets to multiple cores happen either via software based interrupt distribution or by affining interrupts to different cores.   Some NICs have capability of distributing the load across multiple large number of descriptor rings with each descriptor ring associated with one interrupt. When these kinds of NICs are replaced with the offload card, it is expected that offload cards also provide similar capability of distributing the inbound packet load across multiple descriptor rings with associated interrupt.   In addition to this,  offload cards are expected to place traffic of a given connection (whether it is client-to-server or server-to-client) on to the same descriptor ring.  That is hash value calculated on 5 tuples should be same whether it is C-S and S-C traffic of a connection.  As we know,  SIP, DIP are in opposite order in IP header between C-S and S-C traffic a connection. Same is true with TCP and UDP ports.  Some hash algorithms generate different values if the order of input to the hash algorthm  is different. To ensure that same hash is generated,   offload cards are expected to arrange the input to hash in  fashion that input is same for both C-S and S-C traffic.  One method of doing this is to pass the input based on the value of the field.  That is, always lesser value of SIP and DIP can be passed first to the  hash algorithm before passing higher value among SIP and DIP.  Same needs to be for SP and DP too.  This will ensure that same hash value is generated.

Device Ethernet Driver and offload card should have understanding of number of receive descriptor rings and accordingly the offload cards uses the hash result to map to one of the descriptor rings.

Virtualization Distribution

Some network equipment devices support virtualization with each VM serving different DC customer traffic. VLANs are typically used to distinguish different customer traffic.  Using IOMMU functionality of x86 and PCIe SR_IOV functionality, offload cards are expected to send the inbound traffic directly to the VMs memory without involving the hypervisor layer.  IOMMU also allows direct mapping of different descriptor rings to the user space processes.  For more information on IOMMU works, please refer to the AMD paper on IOMMU specification.

Multiple cores might have been assigned to VMs.  In those cases, this distribution should be combined with the 'Multicore Distribution'.

Miscellaneous

Since offload cards are intelligent NICs and they are part of Data center environment,  data center bridging functionality is expected to be provided by NIC cards, even though these are not offload functions.  Some of the functions expected are:

802.1qaz:  Enhanced Transmission Selection

802.1qbb: Priority based flow control

802.3bd:  MAC control frame for priority based flow control.
802.1 Qau: Congestion notification for end points to take action.

802.1qbg and 802.1qbh :  Virtual bridge (VEPA, VN-Tag)

I hope it helps. If you find any functions that can be offloaded, please drop a comment.

What are Traffic Monitoring Enabler Switches?

There is increasing trend of  Traffic Monitoring Enabler Switches (TMES) in Enterprise, Data Center and Service provider environments.

Need for TMES:

Traffic monitoring devices are increasingly becoming requirement for networks in Enterprise, Data Center and Service provider environments.  There are multiple types of monitoring devices are being deployed in networks.

• Traffic Monitoring for intrusion detection:  Security is very important aspect of Enterprise networks.  Intrusion detection is one of the components of comprehensive network security.  IDS devices listen for the traffic passively and do the intrusion analysis on the traffic.  Intrusion attempts and intrusion events are sent to the administrators for out-of-band action.  IDS devices also can be configured to send TCP resets in case of intrusion detection to stop any more traffic going on the TCP connection.  IDS devices also can be configured to block certain traffic for certain amount of time by informing local firewall devices. 
• Surveillance:  Due to government regulations, all important data needs to be recorded.  Surveillance monitoring devices again listen for the traffic passively and record them in persistent storage for later interpretation.  Surveillance devices also provide capability to recreate the sessions such as Email conversations, file transfer conversations,   Voice and video conversations from the recorded traffic.  Some surveillance devices also provide run time recreation of conversations too.  
• Network Visibility :  These monitoring devices capture the traffic passively and provide complete network visibility of the traffic. They provide capabilities such as 'Identification of applications such as P2P, IM,  Social networking and many more ',  'Bandwidth usage of different applications,  networks'  and provide analysis for network administrators with valuable information to maintain networks and bandwidth to make  Enterprise critical applications work always.
• Traffic Trace:   Traffic trace devices help network administrators to find the bottlenecks in different network segments.    These devices tap the traffic at multiple locations in the network and provide the trace capability for finding out the issues in network such as misconfiguration of different devices in network,  choke points etc..

Network administrators face following challenges to deploy multiple monitoring devices.

• Few SPAN ports in existing L2 switch infrastructure:  Many L2 switch vendors provide one or at the most two SPAN ports.  L2 switches replicate the packets to SPAN ports.  Since there are only two SPAN ports at the most,  only two types of monitoring devices can be connected.  This is one big limitation network administrators face.  
• Multiple network tap points :  In complex network infrastructure, there are multiple points where monitoring is required.  Placing  multiple monitoring devices at each point is too expensive. Network administrators would like to use same monitoring devices to capture traffic at multiple locations. 
• Capacity limitations of monitoring devices:  With increasing bandwidth in the networks, it is possible that one monitoring device may not be able to cope with the traffic.  Administrators would like to multiple monitoring devices of same type to capture the traffic with some external component doing load balancing the sessions across multiple monitoring devices.
• High Capacity Monitoring devices :  There could be instances where monitoring device can take more load. In these cases, one monitoring device can take load from several tap points.  Administrators look for facility to aggregate the traffic from multiple points to one or few monitoring devices of same type.
• Non-Switch capture points :  Network administrator may like monitoring of traffic in a point where there are no switches -  Router to Server,  Wireless LAN Access Point to Access Concentrator etc..   Since there is no switch, there are no SPAN ports.  Network administrators look for some mechanism such as Inline TAP functionality to capture the traffic for monitoring.

What is TMES?

TMES is a switch device with monitoring enabling intelligence to allow connectivity of multiple monitoring devices of different types without any major changes to the existing network infrastructure.

This device taps the traffic from SPAN ports of existing switches in the network and direct the traffic to attached monitoring devices.

TMES allows:

• Centralization of monitoring devices.
• Filtering of the traffic.
• Balancing of traffic to multiple monitoring devices of a given type.
• Replication of traffic to different types of monitoring devices.
• Aggregation of traffic from multiple points to same set of monitoring devices.
• Truncation of data 
• Data manipulation & masking  of the sensitive content of the traffic being sent to monitoring devices.
• Inline TAP functionality to allow capture points where there are no SPAN ports.
• Time Stamp functionality
• Stripping off  Layer 2 and Tunnel headers that are unrecognized by monitoring devices.
• Conditioning of the burst traffic going to the monitoring devices.

Centralization of Monitoring Devices: 

Without TMES,  monitoring devices need to be placed at different locations in the network.  With TMES,  TAP points are connected to TMES and monitoring devices are connected to only TMES ports.

Filtering of Traffic:

This feature of TMES allows filtering of unwanted traffic to a given monitoring device.  Monitoring devices are normally listen for traffic in promiscuous mode. That is, monitoring device gets all the traffic that is going on the wire.  But all the traffic is not interesting to the monitoring device.   Typically monitoring device itself does the filtering.  By offloading filtering out of it,  it saves valuable cycles in receiving the traffic (interrupts) and filtering the traffic.  TMES takes this load out of monitoring device and thereby increase the capacity of monitoring devices.

Filtering of traffic should not only be restricted to unicast. It should be made available even for Multicast and broadcast packets.

Balancing the traffic to Multiple Monitoring devices of a given type of monitoring:

If the amount of traffic that needs to be recorded is very high, then multiple monitoring devices will be deployed.   TMES allows multiple monitoring devices to take the load.  TMES load balances the sessions (not packets) to multiple monitoring devices based on performance of monitoring device.  By balancing based on sessions,  TMES ensures that all the traffic for a given connection go to one monitor device.

Replication of Traffic

When there are different types of monitoring devices, each device is expected to get the traffic.  As discussed above,  traditional L2 switches have at the most two SPAN ports.  TMES is expected to replicate the traffic as many number of times as number of different monitoring device types and send the replicated traffic to the monitoring devices.


Combining the replication feature with load balancing:  Assume that a deployment requires the traffic to be sent to two types of monitoring devices -  IDS and Surveillance. This deployment requires  that the 6Gbps bandwidth traffic to be analyzed and recorded.  If IDS and Surveillance devices can analyze and record only 2Gbps bandwidth, then the deployment requires 3 IDS devices and 3 Surveillance devices.  In this case, TMES is expected to replicate the original traffic twice - One for IDS devices and another for Surveillance devices.  Then TMES is expected to balance the one set of replicated packets to go to one of 3 IDS devices and second set to go to one of three Surveillance devices.  

Aggregation of traffic from multiple points to same set of monitoring devices

As discussed in 'Centralization of Monitoring devices',  traffic from different locations of the network can be tapped.   TMES is expected to provide multiple ports to receive the traffic from multiple locations in the network, filter the traffic, replicate and balance the traffic across monitoring devices.  It is possible that traffic of a given connection might be going through multiple points and hence there could be duplication of traffic coming to the TMES.  It is also possible that duplicated traffic might be going to the same monitoring device.  Monitoring device might get confused with duplicated traffic.  To avoid this scenario,  TMES is expected to mark the packets based on the incoming port on TMFS (that is the capture point) such as adding different VLAN ID based on the capture point or adding an IP option etc..    This would allow monitoring device to distinguish the same connection traffic across different capture points.




Truncation or Slicing of packets


Some monitoring device types such as traffic measuring devices don't require complete data of the packets to come in. By slicing the packet to smaller packet would increase the performance of those monitoring devices. TMESs are expected to provide this functionality before sending the packets to the monitoring devices. Truncate value is with respect to payload of TCP, UDP etc..  Some monitoring devices are only interested in headers upto layer 4.  In this case, truncation value can be 0.  Some monitoring devices may expect to see few bytes of payload. TMESes are expected to provide this flexibility of configuring truncate value.

Truncation of packet content should not reflect in the IP payload size. It should be kept intact to ensure that monitoring devices can figure out the original data length even though it receives truncated packets.



Data  masking 

Based on type of monitoring devices,  administrator may like to mask some sensitive information such as credit cards,  user names and passwords from being recorded.   TMES is expected to provide this functionality  of pattern match and mask the content there by removing privacy concerns.

TMES also might support Data replacement (DR).   DR feature might increase the size of the data. Though it is not a big issue for UDP type of sessions, it requires good amount of handling for TCP connections.  As we all know TCP sequence number represent the bytes, not packets.  So, any changes in the data size requires sequence number update.  It not only requires sequence number update in the affected packet, but also further packets going on the session.  All new packets would undergo the sequence number update. Similarly, ACK sequence number of reverse packets also should be updated while sending the packets to the monitoring devices.

When DR feature is combined with the 'Replication' feature,  this delta sequence number update can be different for different replicated packets.  Delta sequence number update feature is required to ensure that monitoring devices find the packet consistency with respect to sequence numbers and the data.

Some TMES vendors call this as part of DPI feature.


Inline TAP functionality:



Many places in the network might not have L2 switch to get hold of traffic from SPAN ports. If the traffic needs to be monitored from those points, then one choice is to place L2 switch and pass the traffic to the monitoring devices through the SPAN ports.  If the capture points are high, then there is a need for placing multiple L2 switches.  Inline TAP functionality is expected to be there in the TMES.  Two ports of TMES are required to TAP the traffic from these capture points.  These two ports will act as L2 switch while replicating traffic for monitoring devices.  Baiscally, TMES is expected to act as L2 swtich for these capture points. Since there are many capture points, TMES essentially become Multi-switch device with each logical switch having two ports.

Time Stamping of Packets

Analysis of traffic that was recorded acorss multiple monitoring devices would be a requirement in general.  It means that the recording devices should have same clock reference so that analysis engine knows the order in which the packets were received.  Yet times, it is not practical to assume that the monitoring devices will have expensive clock synchronization mechanisms.  Since TMES is becoming a central location to get hold of traffic and redirecting them to monitoring devices, TMES is expected to add time stamp in each packet that is being sent to the monitoring devices. 

IP protocol provides an option called 'Internet TimeStamp'.  This option expects TMES to fill its IP address and timestamp in milliseconds with midnight UT.  

Stripping of L2 and Tunnel headers


Many monitoring devices don't understand complicated L2 headers such as MPLS, PPPoE and tunnel headers such as PPTP (GRE),  GTP (in case of wireless core networks),  L2TP-Data,  IP-in-IP,  IPv6 in IPv4 (Toredo,  6-to-4, 6-in-4) and many more.  Monitoring devices are primarily interested in inner packets. TMESs are expected to provide stripping functionality and provide basic IP packets to the monitoring devices.  Since monitoring devices expect to see some known L2 header,  TMESes typically are expected to strip off tunnel headers and complicated L2 headers and keep Ethernet header intact.  If Ethernet header is not present,  TMESes are expected to add dummy Ethernet header to satisfy the monitoring device reception.


Traffic Conditioning

Monitroing devices are normally rated for certain amount of Mbps. Yet times, there could be bursts in the traffic, even though overall average traffic rate is within the device rating.  To avoid any packet drop due  to brusts,  TMES is expected to condition the traffic going to monitoring device.

Players :

I came across few vendors who are providing solutions meeting most of abvoe requirements.

Gigamon :  http://www.gigamon.com/
Anue Systems:  http://www.anuesystems.com/
NetOptics:  http://www.netoptics.com


I believe this market is yet to mature and there is lot of good upside potential.

There is good need for monitoring devices and hence the need for TMES will only go up in coming years.

User space Packet processing applications - Execution Engine differences with processors

Please read this post to understand Execution Engine.

Many processors with descriptor based IO devices have their own interrupts.  For each device, there is corresponding UIO device.  Hence software poll based EE provides 'file descriptor' based interface to register, deregister and get hold of indication through callbacks.  EE applications are expected to read the packets from the the hardware by themselves and do rest of the processing.

As discussed in UIO related posts,  we have discussed ways to share the interrupts across devices. As long as UIO related application kernel driver knows the type of event for which interrupt is generated, appropriate UIO FD is woken up and  things will work fine.

Non-descriptor based IO is becoming quite common in recent Multicore processors.  Hardware events (packets from the Ethernet controllers,  acceleration results from the acceleration engines) are given to the software through set of HW interfaces.  Selection of  HW interface by the hardware is based on some load balancing algorithms or based on some software inputs.  But the point is that, the events which are being given to the software through one HW interface are from multiple hardware IO sources.  Each HW interface is normally associated with one interrupt.  One might say that this can be treated as interrupt being shared across multiple devices.  But, some of  the Multicore processors don't have facility to know the reason for HW interrupt.  Nor they have facility to know the event type of first pending event in HW interface.  Unless the event is dequeued from the HW interface, it is impossible to know the type of event.  Also, due to interrupt coalescing requirements, a given interrupt instance might represent multiple events of different IO source devices.  Due to this behavior,  there may be only one  UIO device for multiple IO devices.  Hence responsibility of demultiplexing these events to right EE application falls on the EE itself.  EE needs to read the event and find out the right application and call the appropriate callback function registered to it. Let us call this functionality in EE as 'EE Event DeMux'.

In Descriptor based systems,  EE applications are expected to read the HW events (packets & acceleration results) by each EE application.  Callback function invocation only provides indication for EE application to read the events from associated hardware descriptors.  In case of 'EE Event DeMux',  the event is already read by the EE itself.  Hence, event is expected to be passed to the callback function.

'EE Event DeMux' submodule registers itself with the rest of EE module to get UIO indication in case of software poll method.  In case of  hardware poll,  'EE Event DeMux' in invoked by the  hardware poll function.

Multicore processors normally provides HW interface for multiple IO devices for the devices which are part of the Multicore processors.  External devices such as PCI and other HW bus based IO devices are still implemented using descriptor based mechanism.  Software poll based EE should not assume that all devices are satisfied using 'EE Event DeMux'.   As far as core Software poll system is concerned,  'EE Event DeMux' is another EE application.  Hardware Poll based method, if they need to use descriptor based HW interfaces, then the hardware poll should also poll descriptor based HW interfaces.

When 'EE Event DeMux' is used by EE applications (such as Ethernet Driver,  Accelerator drivers,), it is necessary that 'EE Event DeMux' considers following requirements.

• It should have its own 'Quota' as number of maximum events it is going to read from the HW interface as part of the callback function invocation by the EE core. Once it reads the 'Quota' number of events or if there are no more events, then it should return back to the 'Core EE' module.  
• Since this the module which demuxes to some EE applications, it should provide its own register/De-register functions.
• When 'Core EE' module invokes this module callback function due to interrupt generation or due to hardware poll,  as described above, it is expected to read at the most 'quota' number of events. While giving the control back to the  'Core EE', it is expected to call EE applications that there are no more events in this iteration. Some EE applications might register to get this indication.  For example, Ethernet driver application might register for this to do the 'Generic Receive Offload' function.  GRO functionality requires to know when to give up while doing TCP coalescing functionality.  In case descriptor based drivers, this issue does not arise as each Ethernet driver as part of callback invocation by the EE itself reads the events and knows when to give up.

Thanks for reading my earlier post.  I hope this helps.

User space Packet processing applications - Execution Engine

If you plan to port your data plane network processing application from Linux kernel space to user space,  first thing you would think is how you can port your software to user space with minimal changes to your software.  Execution Engine is the first thing one would think of.

Many kernel based networking applications don't create their own threads.  They work with the threads which are already present in the kernel.   For example,  packet processing applications such as Firewall, IDS/IPS, Ipsec VPN work in the context of Kernel TCP/IP stack.  This is mainly done for performance reasons.  Additional threads for these applications result in multiple context switches.  Also, it results into pipeline processing as packets handover from one execution context to another execution context.  Pipelining requires queues involving enqueue, dequeue operations which take some core cycles.  Also, it results into flow control issues when one thread processing is more than other threads. 

Essentially,  Linux kernel itself provides execution contexts and networking packet processing applications work within these contexts.  Linux TCP/IP stack itself works in softirq context.  SoftIRQ processing in normal kernel runs from both IRQ context as well as softirqd context.  I would say 90% of the time SoftIRQ processing happens in the IRQ context.  In PREEMPT_RT patched kernel, network IRQs are mapped to the IRQ threads.  In any case,  the context at which network packet processing applications run is unknown to the applications.  Since Linux kernel execution contexts are per core basis,  there are  less shared data structure and hence less locking requirement.  Kernel and underlying hardware also provides mechanism to balance the traffic across different execution contexts with flow granularity. In cases where hardware does not provide any load balancing functionality,  IRQs are dedicated to different execution contexts.  If there are 4 Ethernet devices and 2 cores (hence two execution contexts),  four receive interrupts of Ethernet controllers are assigned equally to two execution contexts.. If the traffic from all four Ethernet devices is same or similar, then both the cores are used effectively.  

Execution Engine in user space packet processing applications, if made similar to the Kernel execution contexts, then application porting becomes simpler.  Execution Engine (EE) can be considered part of infrastructure to enable DP processing in user space.  EE design should consider following.

• There could be multiple data plane processing applications in user space.  Each DP daemon may be assigned to run in fixed set of cores - core mask may be provided at startup time.
• If DP daemon is not associated with any core mask, then it should assume that the daemon may be run by all cores. That is, it should considered as if core mask contains all core bits set.
• Set of cores are dedicated to the daemon. That is, those cores don't do anything else other than doing DP processing of the DP daemon.  This facility is typically used to ensure in Multicore processors providing hardware poll.  Recent generation of Multicore processors have facility to provide incoming events and acceleration results through single portal (or station or work group).  Since the core is dedicated, there is no software polling is required.  That is hardware polling can be used if underlying hardware supports it and if the core(s) are dedicated to the process.

It appears that number threads in the process equaling the number of cores assigned to the process provides the best performance. Also, this also provides great similarity with kernel execution contexts. With the above background,  I believe EE needs to have following capabilities:

• Provide capability to assign the core mask. 
• Provide capability to indicate whether the cores are dedicated or assigned.
• If no core mask is provided, it should have capability to read the number of cores in the system and should assume that all the cores are given in core-mask.
• Provide capability to use software poll or hardware poll.  Hardware poll should be validated and accepted only if underlying hardware supports it and only if the cores are dedicated to it. Hardware polling has performance advantages as it does not require interrupt generation and interrupt processing. But the disadvantage is that the core is not used for anything else.  One should weigh the options based on the application performance requirements. 
• API it exposes for its applications should be same whether the execution engine uses software poll (such as epoll()) or hardware poll.

Typically capabilities are provided through command line parameters or via some configuration file.  EE is expected to create as many threads as number of cores in the core mask.  Each thread should provide following functionality:

• Software timers functionality - EE should provide following functionality.
• Creation and deletion of timer block
• Starting, stopping, restarting timers in each timer block.
• Each application can create one ore more timer blocks and use large number of timers in each timer block.
• As in Kernel,  it is required that EE provides cascaded timer wheels for each timer block.
• Facility for applications to register/De register for events and passing the events.
• API function (EEGetPollType()) to return the type of poll - Software or Hardware : This function would be used by EE applications to use file descriptors such as UIO and other application oriented kernel drivers for software poll or use hardware facilities for hardware poll 
• Register Event Receiver :  EE applications can use this function to register the FD, READ/Write/Error, associated callback function pointer and callback argument.
• Deregister Event receiver:  EE applications can call this to de register the event receiver which was registered using 'Register' function.  
• Variations above API will need to be provided by EE if it is configured with hardware poll. Since each hardware device has its own way of representing this,  there may as many number of API sets. Some part of each EE application have hardware specific initialization code and calls the right set of API functions.
• Note that one thread handles multiple devices (multiple file descriptors in case of software poll). Every time epoll() comes out,  callback functions of ready FDs would need be called. These functions which are provided by the EE applications are expected to get hold of packets in case of Ethernet controllers, acceleration results in case of  acceleration devices or other kinds of events from different kinds of devices. From the UIO discussion,  if the applications use UIO based interrupts to wakeup the thread, then it is expected that all events are read from the device to reduce the number of wakeups (UIO coalescing capability).  Some EE application might be reading lot of events. For each event it reads, it is going to call its own application function.  These applications can be very heavy too. Due to this,  if there are multiple FDs are ready,  one EE application may take very long time before it returns back to the EE.  This results into unfair assignment of EE thread to FDs which are also ready.  This unfairness might even result into packet drops or increase the jitter if high priority traffic is pending to be read in other devices.  To ensure the fairness,   it is expected that EE applications process only 'quota' number of events at any time before returning back to the EE.  'Quota' is tunable parameter and can be different for different types of devices.  EE is expected to callback the same application after it runs through all other ready file descriptors.  Until all ready EE applications indicate that they don't have anything to process, EE should not be calling ePoll().  To allow EE to know whether to call the application callbacks again,  there should be some protocol.  Each EE application can indicate to the EE while returning from the callback function on whether it processed all events.  EE based on this indication will decide to call the EE application again or not before it goes to the epoll() again.  Note that epoll() is expensive call and hence it is better if all events are processed in fairness fashion before epoll() is called again.   In case of hardware poll based configuration,  this kind of facility is not required as polling is not expensive. Also Multicore SoCs implementing the single portal for all events have fairness capabilities built in.   Since the callback function definition is same for both software and hardware poll based systems,  these parameters exist, but they are not used by hardware poll based systems.

EE before creating threads should initialize itself and then create threads. Once created it should load the shared libraries of its applications one by one.  For each EE application library, it is expected to call 'init' function by getting hold of address of 'init()' symbol.  Init() function is expected to initialize its own module.  Each EE packet processing threads is expected to call another function of EE application. Let us call this symbol name is 'EEAppContextInit()'.   EEAppContextInit function expected to real initialization such as opening UIO and other character device drivers and registering with the software poll() system.

EE also would need to call 'EEAppFinish()' function when the EE is killed.  EEAppFinish does whatever graceful shutdown required for its module.

Each thread, if it is software based poll, does epoll on all the FDs registered so far.  Polling happens in the while() loop.  epoll() can take the timeout argument. Timeout argument must be next lowest timer expiry timeout of all software timer blocks.  When epoll() returns, it should call the software timer library for any timer expiry processing. In case of hardware poll,  specific hardware specific poll function would need to be used.

In addition to above functions,  EE typically needs to emulate other capabilities provided by Linux Kernel for its applications such as - Memory Pool library,  Packet descriptor buffer library,  Mutual exclusion facilities using Futexes and user space RCU  etc.. 

With these above capabilities, EE can jump start the application development. This kind of functionality only requires changes at very few places in the applications.

Hope it helps.

UIO - Acceleration Device mapping in user space

Please see this post on how to use UIO frame work to implement device drivers in user space. As noted in that post,  UIO framework predominantly is used to install the interrupt handler and to wake up the user space process implementing the device driver. Please read the earlier post, before going further.

There are two types of devices that get mapped to user space for zero-copy drivers - Ethernet devices and Accelerator devices such as Crypto Engine,  Pattern Matching Engine etc..   Normally, a given Ethernet device is completely owned by one user process.  But accelerator devices are normally needed across multiple processes and also is needed by kernel applications.  Hence acceleration device usage is more challenging. 

To enable usage of acceleration devices by multiple user processes, acceleration device normally support multiple individual descriptor rings.  I know of some acceleration devices supporting four descriptor rings, with each descriptor ring working independent of each other, that is, one descriptor ring is sufficient for issuing the command and read the result .  In this scenario,  a given user process at least should own one descriptor ring for zero copy driver.  If the acceleration device contains four descriptor rings, then four user processes can use the acceleration device without involving the kernel. Since a typical system contains more processes than the descriptor rings,  it is necessary that at least one descriptor ring is reserved for kernel usage and other application processes.  In the example where one acceleration device supports four descriptor rings,  in one scenario,  one can choose three critical user processes that require zero copy driver. Each of these critical processes use one descriptor ring each. All other user processes and kernel share one descriptor ring.  

Each process requiring zero copy driver should memory map the descriptor ring space.  Since many chip vendors provide Linux kernel drivers for acceleration engines,  my suggestion is to make changes to this acceleration engine driver to provide some additional API functions to detach and attach the descriptor rings on demand basis.  When user process requires a descriptor ring, the associated application kernel module can call the acceleration driver 'detach' function for that descriptor ring.  When the process dies, the associated kernel module should attach back the descriptor ring to the kernel driver. This way, each user process need not work on the initialization of security engine.  It only need to worry about the filling up the descriptors with commands and reading responses.  It also provides the benefit that the descriptor rings can be dynamically allocated and freed based on the applications running at that point of time.

If there are as many interrupts as number of descriptor rings, then each process's zero copy driver can have its own interrupt line.  Yet times, even though there are multiple descriptor rings,  number of interrupts are less than the descriptor rings. In this case, interrupt need to be shared across multiple descriptor rings. Fortunately Linux kernel and UIO frame work provides mechanism for multiple application kernel modules to register different interrupt handlers for the same interrupt line. irq_flags field as part of uio_info structure that is registered with the UIO framework should have IRQ_SHARED bit set.  Linux Kernel and UIO frame work call the interrupt handler one by one in sequence. Interrupt handler that has data pending to be read from corresponding descriptor ring should return IRQ_HANDLED.    It means that the device should have capability to check the pending data without reading it out.  Note that reading the acceleration result should be done by the user space.   When the handler returns IRQ_HANDLED, UIO framework wakes up the user process.  Since one IRQ line is shared by multiple processes,  as described in earlier post, masking and unmasking the interrupts can't be done by the interrupt handler and user process. Since interrupts can't be disabled, one can't use natural coalescing capability as described in the earlier post.  But fortunately, many acceleration devices provide hardware interrupt coalescing capability. Hardware can be programmed to generate interrupt for X number of events or within Y amount of time. If the hardware device you have chosen does not have the coalescing capability and require IRQ to be shared across multiple user processes, then you are out of luck. Either don't use UIO facilities or live with too many interrupts coming in.

All other user process  without dedicated descriptor rings should work with accelerator kernel driver that is provided by the OS/Chip vendors.  That is, they need to send the command buffer to the kernel driver and read the result from the kernel driver.  Kernel drivers are normally intelligent enough to service multiple consumers and hence many user processes can use the acceleration engine.


Comments?

Multicore Networking applications - Mitigating the Performance bottlenecks

I had given this talk in 2010 Multicore Expo in San Jose.  It was in presentation document in concise form. I voiced the most of the details during my talk.  Many people requested me to provide details in written form.  I tried to give details here in this post.  I hope this post would give enough details on 'New techniques to improve software performance with increasing number of cores'.

Before going further, I would like to differentiate two kinds of applications - Packet processing applications and Stream processing applications.

Packet processing applications in my definition are the ones which  take  packet by packet, work on the packet and send out the same packet or packet with some minor modifications.  In packet processing applications,  there is one-to-one correspondence between input and output packets except for very small number of exceptions. One example where there is no one-to-one correspondence is when there is IP reassembly or fragmentation. Other example is when the packet is dropped by the application. Example applications in this category are:  IP forwarding, L2 Bridging,  Firewall/NAT,  Ipsec and even some portions of IDS/IPS.

Stream processing applications are the ones which may take packets or stream of data,  work on data and send out the data or  send out different packets. Most of  the TCP socket based proxy applications come under this category. Examples:  HTTP Proxy,  SMTP Proxy,  FTP Proxy etc..

This post tries to aid the programmers debugging the software to find out the performance bottlenecks in Multicore networking applications.

Always Ensure to do  flow/Session Parallelization 

Ensure that only one core is processing the session at any given time.  If multiple packets from the same session are being processed by more than one core at the same time,  then there would be requirement to ensure that Mutual exclusion on the session variables.  That would be very expensive.  Multicore SoCs actually aid you to do flow parallelization in packet processing applications.  Many Multicore SoCs support parsing the fields from the packets,  calculate hash on the software defined fields and distribute the packets across the multiple queues based on the hash value.  And then they provide provision for software threads to dequeue the packets from the queues.  These SoCs also provide provision to stop dequeue of packets from threads until the control of the queue is given up explicitly.  This ensures that a given flow is processed by only software thread at any time.

Many Multicore SoCs also have facility to bind the queues to the software threads and each software thread to the core.  If the number of flows are small, there is a possibility of cache being warmed with contexts due to previous packets. This reduces the data movement from DDR.   Also, many Multicore SoCs provide facility to stash the context as part of dequeue operation which reduces the cache thrashing issue even if  binding of the queues to the cores are not done. 

Flow parallelization not only eliminates the need for Mutexes, it also ensures that there is no packet mis-ordering in the flows.

Many stateful packet processing applications require not only flow parallelization, but also session parallelization.  Session typically consists of two flows - Client to Server traffic and Server to Client traffic.  It is possible that two packets from both the flows may be coming to the device and two separate software threads might be processing these packets. Stateful applications share many state variables across these two flows. Due to this, you may require mutual exclusion operation if both the packets are allowed to be processed at the same time.  Session Parallelization as described here would eliminate the need for mutual exclusion.  Unlike flow parallelization, session parallelization is not available in many Multicore SoCs for cases where the tuple values are different in both the flows and hence needs to be done in software.  Packet tuples are different when NAT is applied. Note that many Multicore SoCs enqueue the packet to the same queue if there is no NAT.  They are intelligent enough to generate the same hash value even though the tuples position get changed, that is, source IP in one flow would be destination IP in reverse flow and same is true for destination IP, Source Port and Destination Port.

Stream processing modules such as proxies would need to ensure that both client side and server side sockets are processed by the same software thread to ensure that there is no  Mutual exclusion operations requirement to  protect the sanctity of state variables.  Stream processing modules typically create many software threads - worker threads.  Master thread terminates the client side connections and handover the connection descriptor to one of the less loaded worker threads.  Worker thread is expected to create new connection to the server and do rest of the application processing.  Worker threads are typically implement FSM for processing multiple sessions. More often,  the number of worker threads would be same as number of cores dedicated for that application.  In cases where the threads need to block for some operations such as waiting for accelerator results, then more threads, in multiples of number of cores, would be created to take advantage of full power of accelerators.

Eliminate the Mutual Exclusion Operation while Searching for Session/Flow Context

This technique is also expected to ensure that there are no mutual exclusion operations in the packet path.  Any networking application do some search operations on the data structures to figure out the operations and other action to be done on the packet/data.  Upon the incoming packet/data,  search is done to get hold of session/flow context and then further packet processing happens based on the state variables in the session.  For example,  IP routing does search on the routing table to figure out the destination port, PMTU and other information for operations such as fragmentation, TTL decrement and packet transmit.  Similarly firewall/IPsec packet processing applications maintain the sessions in a easy to search data structures such as RB trees, hash lists etc..   Since the sessions are created or removed dynamically from these structures, it is necessary to protect the data structure while doing operations such as add/delete/search.  Mutual exclusion operations using spinlock,  futex, up/down are one way to do this.   RCU (Read-Copy-Update) is another method that can be used which eliminates the Mutex operation during search.   RCU operation is described in earlier post.  Please check that here and here.  RCU lock/unlock operations in many operating systems is very simple operation. Note that Mutex operations are still required for add/delete even in RCU based usage.

Eliminate Reference Counting 

One of the other bottlenecks in the Multicore programming is the need to keep the session safe from deletion while it is being used by other software threads. Traditionally this is achieved by doing 'reference counting'. Reference counting is used in two cases - During packet processing operation or  When neighbor module store the reference.

In the first case, reference count of the session context is incremented as part of the session lookup operation.  During packet processing, the session is referred many times to get hold of state variable values and to set the new values in the state variables of the session.  It is expected that if the session is deleted, it should not be freed until the current thread is done with its operation. Otherwise, it would corrupt some other memory if the session memory is freed and allocated to somebody else during packet processing.  To ensure that the session ownership is not given away, the reference count is checked as part of 'delete operation'.  If is is not zero, then the session is marked for deletion, but not freed until the reference count becomes zero.  If the value is zero, it indicates there is no reference to this session and the session gets freed. 
 
Since RCU operation postpones the delete operation until current processing cycles of all other threads,  reference counting becomes redundant.  Elimination of reference count not only helps in improving the performance, but also reduces the maintenance complexity. Note that reference counting operation requires atomic usage of count variable. Atomic operations are not inexpensive.

Second use case of reference count is when the neighbor modules store the reference (pointer) to the sessions in their session contexts. By eliminating the storage of pointer,  reference count usage can be eliminated.  This post helps you understand how this can be done.

Linux user space programs also can take advantage of RCUs. See this post for more details.

Use the Cache Effectively

Once the matching session is found upon incoming data/packet,  processing functionality uses many variables in the session. If these variables are together in a cache line,  any cache fill due to access of one variable result all other variable in the cache line available in the cache.  That is, Access to other variables will not result in going to DDR.  But all variables may not fill in one cache line. In those cases, it is necessary to group the related variables together to reduce going to DDR.

To effectively use instruction cache, always arrange your code with likely/unlikely compiler directives. Compilers will try to arrange the likely() code together. 

Reduce Cache Thrashing due to Statistics variables

Almost all networking applications update statistics variables.  Some variables are global and some of them are session context specific variables.  There are two types of statistics variables -  increment variables and add variables. Increment variables are typically used to maintain the count of packets.  Add variables are used to maintain the byte count.  Updating these variables require getting hold of current values and then add or increment operation.   If these variables are updated by multiple threads (with each thread running on a specific core), then every time an variable is updated,  cache information of this variable is no longer valid in other cores.  When one of other cores needs to do same operations,  it needs to get the current value first from the DDR and apply the operation.  In worst case scenario, where packets are going to round robin fashion to different software threads (hence cores), then the cache thrashing due to statistics variables would be very high and this would reduce the performance dramatically.

Always use 'per core/thread statistics counters'  whenever possible.  Please see this post for more details. 

Some Multicore SoCs provide special feature which also eliminates the need for 'per core' statistics maintenance.  These SoCs provide facility to allocate memory block for statistics.  These SoCs provide facility to fire the operation and forget about it.  Firing the operation involves the operation type (increment, decrement, add X or sub Y etc..) and memory address (32 bit or 64 bit).  SoCs internally do this operation without cache thrashing.  I suggest strongly to use this feature, if it is available in your SoC.

Use LRO/GRO facilities

Many networking applications' performance depends on the number of packets being processed than the number of bytes processed.  Examples: IP Forwarding,  Firewall/NAT and Ipsec with hardware acceleration.  So, reducing the number of packets processed becomes key in improving the performance.

LRO/GRO facilities provided by operating system in Ethernet drivers or by Multicore SoCs reduce the number of TCP packets, if multiple packets from the same TCP flow are pending to be processed.  Since TCP is byte oriented stream protocol, it does not matter whether or not the processing happens on packets.  Please see this post for more information on LRO feature in Linux operating system.  If it is supported by your operating system or Multcore SoC, always make use of it.

Process Multiple Packets together


Each packet processing module does set of operations on the packets/data - such as Search,  Process and  Pkt out.  If the packet is going through multiple modules, there are many C functions get called.  Each invocation of C function has its own overhead such as pushing the variables in the stack, initializing some local variables etc..    By bunching multiple packets of same flow together can reduce search/pkt out overhead and overhead associated with the C functions.


Some Multicore SoCs provide facility to coalesce packets together on per queue basis with coalescing parameters -  Packet threshold and time threshold.  Queue does not let the target thread to dequeue until one of the conditions reached - either number of packets in the queue exceed the packet threshold parameter or if no packet was dequeued for time 'time threshold'.   If this facility is available, ensure that your software dequeues multiple packets together and processes them together. 

Yet times, there is no one-to-one correspondence between queues and sessions.  In that case, one might ask that search overhead can't be reduced as there is no guarantee that the packets in the same queue belong to the same session.  Though it is correct, it might still have some improvements due to cache warming if there are more than one packet belonging to same session in the bunch.

As a software developer,  it would be required to strive for one-to-one correspondence between queues and sessions. This can be done easily among the modules running in software.  Some Multicore SoCs provide queues for not only to access hardware blocks, but also for inter-module communication.  Software can take advantage of this to create one-to-one mapping between queues and destination module's sessions.

It is true that when the packets are being read from the Ethernet controllers, there is no way to ensure that a queue only holds packets of one session as the queue selection happens based on the hash value of packet fields.  Two different sessions may fall into same queue.  In those cases,  as mentioned above you might not see improvement from 'serach' functionality, but you would still see improvements due to less number of invocations of C functions.

Many Multicore SoCs also have functionality to take multiple packets together for acceleration and for sending the packets out. This also will reduce the number of invocation to acceleration functions and for sending packets out.  If this facility is available in your Multicore SoCs,  take advantage of it. 

Eliminate usage of software queues

Some Multicore applications need to send the packets/data/control-data to other modules.  If multiple threads send the data to the queue, then there is a need for mutual exclusion to protect these data structure queues.

Many Multicore SoCs provide queues for software usage.  These queues would eliminate the need for software queues and hence eliminate the mutual exclusion problem, there by improving performance.   Some Multicore SoCs also provide facility to group multiple queues together into a queue group which allows sending and receiving applications to enqueue priority items and dequeue based on priority.  These queues can be used even among different processes or virtual machines as long as shared memory is used for items that get enqueued and dequeued.  Some Multicore SoCs even went a step further to provide 'copy' feature which avoids shared memory and there by providing good isolation. This feature makes a copy of these items from source process to internal managed memory by Multicore SoCs and copy to the destination process memory as part of dequeue operation.

Always use this feature if it is available in your Multicore SoC.

Eliminate the usage of Software Free pools 

Networking applications use free pools of memory blocks for memory management.  These free pools are used to allocate/free session contexts, buffers etc..   Many software threads would require these facilities at different times. Software typically maintains the memory pools on per core basis to avoid mutual exclusion operations on per allocation basis.  Since there is a possibility of asymmetric usage of pools by different threads, yet times there is a possibility of memory allocation failures even though there are free memory blocks in other threads' pools.   To avoid this, software does complex operations during these scenarios of moving memory blocks from one pool to another through global queues.   Many Multicore SoCs provide 'free pool' functionality in hardware.  Allocation and free can be done by any thread at any time without mutual exclusion operations. Use this facility whenever it is available.  It saves some core cycles.  More than that is provides efficient usage of memory blocks.

Use Multicore SoC acceleration features to improve performance

There are many acceleration features that are available in Multicore SoCs.  Try to take advantage of them.  I classify acceleration functions in Multicore SoCs into three buckets -  Ingress In-flow acceleration,  In-flight acceleration and Egress in-flow acceleration.

Ingress In-flow acceleration:  Acceleration functions that are done by Multicore SoCs in the hardware on the packets before they are handed over to software are called Ingress In-flow accelerations.  Some of the features, I am aware, in Multicore SoCs are:

• Parsing of Packet fields :  Some Multicore SoCs parse the headers and make those fields available to the software along with the packet.  Software needing the fields can eliminate the parsing of fields.   These SoCs also provide facility for software to choose the fields to be made available along with the packet.  They also provide facilities for software to create parser to extract fields from proprietary headers or from non pre-defined headers.  Try to take advantage of this feature.
• Distribution of packets across threads:  This is basic feature required in Multicore environments.  Packets needs to be distributed to different software threads.  Many Multicore SoCs also ensure that packets belonging to one flow go to one software thread at any time to ensure that packets will not get mis-ordered within a flow.  As described above,  multiple queues would be used by hardware to place the packets.  Selection of queue is based on hash value calculated on the set of software programmable fields.  As a software developer, take advantage of this feature rather than implementing the distribution in software.
• Packet Integrity checks & Processing offloads:  Many Multicore SoCs do quite a bit of integrity checks on the packet as  listed below.  Ensure that your software don't do them again to save some core cycles.
• IP Checksum verification.
• TCP, UDP checksum verification.
• Ensuring that the headers are there in full.
• Ensure that size of packet is not less than the size indicated in the headers.
• Invalid field values.
• IPsec inbound processing.
• Reassembly of fragments
• LRO/GRO as described above.
• Packet coalescing as described above.
• Many more.
• Policing :  This feature can police the traffic and reduce the amount of traffic that is seen by the software.  If your software requires policing of some particular traffic to stop cores from getting overwhelmed, this feature can be used rather than doing it in the lowest layers of software.
• Congestion Management :  This feature ensures that the number of buffers used up by the hardware won't go up exponentially. Without this feature, cores may not find buffers to send out the packets if all buffers are used up by the receiving hardware. This situation typically happens when the core is doing lot of processing and hence slow in dequeuing while lot more packets are coming in.  Many Multicore SoCs also have facility to generate pause frames in case of congestion. 

Egress In-flow acceleration:   Acceleration functions that are done in the hardware once the packets are handed over to it by software to send the packets out are called Egress in-flow acceleration functions.  Some of the Egress in-flow acceleration functions are given below.  If these are available, take advantage of them in your software as these can reduce significant number of cycles in the core.

• Shaping and Scheduling :  High priority packets are sent out within the shaped bandwidth.  Many Multicore SoCs provide facilities to program the effective bandwidth. These SoCs shape the traffic with this bandwidth. Packets which are queued to it by software would be scheduled based on the priority of the packets.  Some SoCs even provide multiple scheduling algorithms and provide facility for software to choose the algorithm on per physical or logical port.  Some SoCs even provide hierarchical scheduling and shaping.  Take advantage of this in your software if you require shaping and scheduling of the traffic.
• Checksum Generation for IP and TCP/UDP transport packets :  Checksum generation, especially for locally generated TCP and UDP packets is very expensive.   Use the facilities provided by hardware.  
• Ipsec Outbound processing :  Some Multicore SoCs provide this functionality in hardware.  If you require Ipsec processing,  use this facility to save large number of cycles on per packet basis.
• TCP Segmentation and IP Fragmentation :  Some Multicore SoCs provide this functionality.  TCP segmentation performs well for local generated packets. Use this functionality to get best out of your Multicore.

In-flight Acceleration :   Acceleration functions provided by hardware that can be used during packet processing are called In-flight acceleration functions.  Crypto,  Crypto with protocol offload,  Pattern Matching,  XML acceleration are some of the acceleration functions that come in this category.  Here the packet/data for acceleration is handed over to the hardware acceleration functions by software. Software reads the results at later time when the results are ready.  Take advantage of these feature in your software wherever they are available . Some Multicore SoCs differentiate themselves by doing lot more in the acceleration functions.  For example,  some Multicore SoCs do protocol offload along with crypto such as Ipsec ESP,  SSL record layer protocol , SRTP and MACSec offloads which do beyond crypto offload.

I see many times people asking me a question on how to use the acceleration functions.  I had detailed this long time back here. Please see the details there and there.

Software Directed Ingress In-flow accelerations:

As described before, Ingress in-flow acceleration is applied before the packets are given to the software. Packets that are received on integrated Etherent controllers go through this acceleration.  But many times this acceleration is required from software too.  Take the example of Ipsec, SSL or any tunneling protocol.  Once the software processes these packets, that is once it gets hold of inner packets,  software would like ingress in-flow acceleration to be applied on the inner packets for distribution across cores and other acceleration functions.  To facilitate these kinds of scenarios, some Multicore SoCs provide concept of 'offline port' which allows software to reserve the offline ports and send the traffic for ingress in-flow acceleration.  Some software features that can take advantage of this feature are:

• Tunneled traffic as described above to let the inner packets to go through the ingress in-flow acceleration,
• IP reassembled traffic - Once the fragments are reassembled, it would have all 5-tuples which can be used to distribute the traffic through offline port.
• L2 encapsulated packets - Such as IP packet from PPP, FR etc..
• Ethernet controllers on PCI and Traffic from Wireless interfaces :  Here the traffic might need to be read by the software and Ingress in-flow acceleration might not have been implemented for these features. Software after getting hold of packets can be directed to in-flow acceleration functions through offline ports.

Use Multicore core features wherever they are available

Multicore SoCs from different vendors have different core architecture. Some Multicore SoCs are based on power pc, some based on MIPS core and Intel Multicore is based on x86 processors. Multicore SoC vendors provide different features to improve performance of Multicore applications.  Whenever they are available, software should make use of them to get the best performance out of cores.  Some of the features that I am aware of are listed below.

Single Instruction & Multiple Data instructions (SIMD)

Multicore SoCs from Freescale and Intel have this block in their cores.   This feature in the cores allows software do a given operation on the multiple data elements.  This kind of parallelism is called 'Data level parallelism'.  'Add' operation in typical cores is performance either on 32 bit or at the most 64 bit operands.  Current generation of SIMD do this operation on 128 bit operands. They also provide flexibility to do multiple 16 bit, 32 bit add operations on different parts of data simultaneously.  SIMD greatly helps in operations which involve arithmetic, bit, copy, compare operations on large amount of data.  Any operation that is done in a loop can be accelerated using SIMD.   In networking world,  SIMD is helpful in following cases:

• Memory compare, copy,  clear operations.
• String compare, copy, tokenization and other string operations.
• WFQ scheduling of QoS, where multiple queues need to be checked to figure out which queues need to be scheduled based on sequence number property of queues.  If the sequence numbers are arranged in array form, then SIMD can be used very effectively.
• Crypto operations.
• Big Number arithmetic which is useful in RSA, DSA and DH operations.
• XML Parsing and schema validations.
• Search algorithms -  Accelerating compare operation to find matching entry from collision elements in a hash list.
• Check-sum verification and generation:  In some cases Ingress and Egress In-flow accelerations can't be used to verify and generate the checksums.  One example is,  TCP and UDP packets that come in IPsec tunnel.   Since the packets are in encrypted form,  ingress and egress accelerators will not be able to verify and generate checksums in inner packets.  Even packets that get encapsulated in tunnels will not be able to take advantage of Ingress & Egress in-flow accelerations.  Checksum verifications and generations need to be done in software by cores.  SIMD would help in those cases tremendously.
• CRC verification and generation:  These algorithms are not very expensive to have In-flight acceleration and not inexpensive for core to do.  SIMD in these cases help as it does not involve any architecture changes to the software and still get lot better performance over the cores which don't have SIMD.

Normally SIMD based cores give at least 50% more performance improvement for typical workloads.  So, as a software developer, figure out the ones that can be improved using SIMD and modify the code to improve performance of your application.

Speculative Hardware Data Prefetching & Software Directed Prefetching

This feature fetches the next cache line worth of data from the current memory access in the hopes that software would use next memory line.  Many core technologies provide control on enabling and disabling this at run time.  Software can take advantage of this while doing memory copy, set and compare operations.  Any data is arranged in linear fashion in the memory (such as arrays) can get good boost of performance with this feature. Note that, if this feature is not used selectively and carefully, it might even give degradation in performance. Be careful in using this feature.

Many cores also provide special instruction to warm the cache given a memory address. Software developers know the kind of processing (next module) and many times next module session context is also known. In those cases, software can be developed such a way that next module session is prefetched while packet processing happens in current module.  When the next module gets the control of the packet, it already has session context in the cache which avoids getting it DDR in serial fashion.  My experience is that using software directed prefetching gives very good results.  This also ensures that the performance does not go down even with large number of sessions.

Some Multicore SoCs provide support for Cache warming on the incoming packets.  As part of making packets ready for the software, these SoCs warm the cache with some part of packet content,  annotation data containing parsed fields and software issued context data.   When the software dequeues the packet, most of the information required to process the packet of the module that is getting hold of packet is in place in the cache, thereby, avoiding on-demand DDR access.  Software can program its context on per queue basis.  Note that, this feature is useful for the first module that receives the packet.  Actually that is good enough as this module can prefetch the next module context while the packet is being processed in the current module.  As long as each modules does this, there is no performance degradation even with high capacity. 

As described before,  hardware queues may not have one-to-one correspondence with the receiving module session contexts.  A queue might be having packets for multiple session contexts. Many times, software maintains the sessions in the hash table with large number of hash buckets.  All collision sessions are arranged in linked list or RB tree.  Software can ensure that there are as many queues as number of hash buckets and program the first collision element in the queue.  If the matching context is not same as the one that was programmed, then one might not get the full benefit of cache warming by the hardware. But if there are 4 collision elements and the traffic across these four are same, cache warming would come in handy 25% of the time. Some software developers might even store the collision elements in an array and program the array to the queue.

Software directed prefetch works very well as long as there is one-to-one correspondence between current module session context and next module session context.  In this case,  current module session context can cache the reference to the next module session context and use this to do prefetch operation.  This scheme also work fine if next module context is super set of multiple current module contexts.  But it does not work well if the next module context is finer granular.  Example:  Ipsec SA transfer packets from  multiple firewall/NAT sessions.  In this case, 'Software Directed Ingress In-flow acceleration' method can be used to direct the hardware to send the packet to next module.  This method not only provides cache warming, but also distributes the processing to multiple cores.


Hardware Page Table walk:

Some cores provide nested hardware page table walk to find out the physical address given the virtual address.  This is really useful for user space applications in Linux kind of operating systems.  Hardware page table walk feature is expected to be taken care by operating system vendors.  But unfortunately many OS vendors are not taking advantage of this feature.  As a software developer, if your Multicore SoC provide this feature, don't forget to ask your OS vendors to take advantage of this.  This will ensure that your performance does not go down when you move your application from Bare-metal environment (where the TLB are fixed and there is no page walk required) to Linux user space.

I hope it helps.