Key-Value Store using Memcached - Microservers & Embedded Multicore processors

 
What is Microserver?

It is explained multiple ways in the Industry.  Intel defines it as ""We define it as any server with a large number of nodes, usually with a single socket or multiple low-power processors and shared infrastructure,".   Serverwatch article on Microservers - small footprint, powerful punch article characterizes it as "The difference between microservers and server blades is very similar to the difference between a netbook and a full-fledged laptop: A netbook is compact, consumes little power and is suitable for a limited range of light duties, while a laptop is bigger and more power hungry but can generally handle anything that is thrown at it".  

What I understand finally is that,  these are like normal server chassis with multiple blades except that these blades (some call it as sleds in the context of Microservers) have low power Multicore processors such as ATOM processor from Intel.   Another difference is that there is no control card in the Microserver chassis to control the sleds.  They run independent of each other connected by 10G switch fabric. 

Why Microservers?

According to Serverwatch article,  Microservers are intended for companies to become energy efficient.  All kinds of workloads don't require high processing power offered by high end server blades.  Simple webservers that serve pages,  file servers,  memcached applications,  Hadoop kind of applications don't require single thread processing power.  They can be serviced using multiple Microserver sleds or multiple low powered cores on each sled.

One might argue that,  high end blades can be virtualzied into multiple virtual machines to handle simple workloads.  Even though that is one option,  some believe that costs associated with virtualization software is high.  With similar energy savings between virtualization and Microservers,  some deployments seems to be favoring the Microserver approach due to cost savings and maintenance headaches associated with virtualization.  Since each Microserver sled can be used to host one particular application (similar to virtual machines),  some see less interruption in services when there is a hardware failure.  In case of Microserver approach,  only the faulty sled needs to be replaced and during that time, only the application running on that sled will be affected.  In case of blade replacement,  services related to all applications running on different virtual machines may have interruption.

Embedded Multicore Processors:  Non-x86 based Multicore processors are increasingly being considered in Microservers mainly for following reasons.

• Low power.
• Performance is achieved using many cores.
• Integration of Peripherals -  Security Co-processors,   Regex pattern matching acceleration,  Compression acceleration and mainly inbuilt  multiple 10G/1G Ethernet MACs.
• 64-bit processors.
• JVM is increasingly available on Embedded Multicore processors too.
• SIMD acceleration as in x86 processors.

It appears that non-x86 based Multicore processors are being considered for some workloads more than others.  Some of the workloads that are more suitable for Embedded Multicore processors are:

• Key-Value stores such as memcached.
• Hadoop based distributed processing
• Front End Web Servers.

This article from facebook "Many-core-key-value store"  mainly talks about memcached and how non-x86 Multicore processors are playing a role.  Though this article mainly concentrates on performance comparison between x86 processors and Tilera Multicore processors,  I believe this argument of Embedded Multicore processor advantage over x86 is equally valid with other Embedded Multicore processors.

What is Memcached:

Excerpt from www.memcached.org:

"Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.
Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.
Memcached is simple yet powerful. Its simple design promotes quick deployment, ease of development, and solves many problems facing large data caches. Its API is available for most popular languages."


Some internal details of memcached:

• Memcached not only listens for the requests coming from applications running on the same system, but also services the requests coming from remote systems via UDP and TCP.
• Memcached maintain the data in the hash table.
• It provides operations to insert, delete and query the entries in/from the hash table.
• It appears that STORE operations are normally done over TCP connections and GET operation is normally done on the UDP connections.
• It does not use hard disk to store the hash table and associated data.  It uses main memory.  Hence no IO bottleneck.

There are large number of memcached users.   Please visit www.memcached.org for more information. It appears that Facebook, Amazon have thousands of servers running memcached application alone.

 Some Key performance Excerpts from the Facebook article:

• Two socket based Intel Xeon L5520 running at 2.27Ghz (total of 8 cores):  200,000 GET transactions/sec.   Power consumption:  140Watts
• Tilera 64 core processor running at 800Mhz:  335000 GET transactions/sec.  Power consumption: 90 watts.

 It appears that some changes were done to Memcached to achieve above performance on Tilera due to 32 bit architecture.

My view is that Embedded Multicore processors from other vendors will do better even. I believe that one should look for following features from Multicore processors to enable Memcached appliances.

• Large number of cores/threads.
• Fast hash generation (SIMD Engine would help).
• Traffic distribution across multiple threads.
• User space based bare-metal environment for UDP based GET transactions while TCP based transaction continue to happen using traditional TCP/IP stack in Operating systems.
• Stashing and Prefetching facilities in hardware to utilize the cache effectively.
• TCP Termination offload functionality in the HW:  TCP/UDP GRO/TSO,  Transport layer Checksum verification/generation, IP fragmentation/reassembly etc..
• Software:  RCU library to avoid contention.  Huge-tlb-fs  to reduce the TLB misses etc..

If Hadoop and Hadoop based applications to work on these types of appliances, one should look for JVM support on the processors.

Hibernate Versus JDBC

Long back I worked on a project in Java, Centralized Management System, to  manage multiple networking devices.  At that time  I remember having a  big debate on whether to use Hibernate for Java Object persistence or use JDBC directly.  We have finally decided to use Hibernate at that time.

I happened to come across this link which talks about 'when to use Hibernate' and also provides advantages and disadvantages of using hibernate.  I felt that it captures many points. Hence I thought of sharing this.

Please find it here : http://www.mindfiresolutions.com/mindfire/Java_Hibernate_JDBC.pdf

One point  to consider by developers when using  Hibernate:

• Different parts of application require different fields from a database table or from a set of related tables to be persistent.  Don't try to put all possible fields from database in one single java object class. Some database fields are more often required for your application than some other fields.  In those cases, you are better off using multiple persistent classes - for frequently used fields and for rarely used fields.  If you go with one single java persistent class, then you may be using more heap memory than necessary.   You know your application. Decide on the number of java persistent classes based on 'type of data',  'how often they are required' for applications etc.. For example, some data which is needed for auditing, which is once-in-a-while activity, may require separate java persistent class so that this object can be instantiated and removed on demand basis.  

Features I like in Hibernate (JPA):

• It separates out the database organization with the business logic.  Due to Object-Relational mapping,  business logic always deals with the java class.  Mapping of java class members with the fields in database tables is defined in the XML file.  Any change in the database organization requires change only in the XML file.  
• Change in the database server from vendor to another during field upgrade or during development has no or very less impact.
• Automatic Version  & Time stamping feature of Hibernate ensures that there is no unintentional update of database by one thread using older data.

Above link has lot more information on the benefits. 

Cloud Multi-Tenancy Support - Authentication, Authorization and Auditing requirements

Authentication and Authorization requirements are common in any web based applications.  Authentication is the process by which the application takes the user credentials, typically using 'log-in' forms and checks the existence of user by checking against user databases (LDAP,  RDBMS, RADIUS etc..).    Authorization is the process in which application allows the access to different parts of application based on type of user accessing it.   In typical application, authorization consists of - Assignment of roles to users as part of creating user in the user database (LDAP, RADIUS etc..).  Mapping the part of application (resource)  to the roles and providing set of permissions to the roles to access mapped resources.  Resources and related permissions for different roles is also called 'Access Control'. 

Many web application development frameworks have minimized the amount of development one needs to do for Authentication and Authorization.   For example, spring-security component of spring framework (http://static.springsource.org) takes care of Authentication & Authorization requirements of many web applications. That is,  for many web based applications,  web developers only need to concentrate on the business logic and can leave the Authentication and Authorization part to the spring-security.  

Cloud applications', mainly SaaS applications',  Authentication & Authorization requirements can't be met by spring-security (Version 3.0.5) as is.  But spring-security provides multiple hooks for extending it.

It is good to know the typical requirements of Cloud applications with respect to Authentication and Authorization first.  This is what this post is concentrating on.  I will try to discuss the how to bridge the gaps in spring-security to enable cloud application in future posts.

Authentication Requirements

Like any web applications,  Cloud applications also require some portions of the application resources to be accessible only to some set of users.  Authentication  is the process where it takes the user credentials using 'login' forms and validates the user by communicating with pre-created user database.   Cloud applications differ from the traditional web applications where cloud application provides services for multiple tenants. That is cloud application resources are instantiated for each tenant.  Rather than having separate server and separate application instance for each tenant,  cloud application services multiple tenants using one server and one application.  Typically, this is achieved using 'Tenant ID" in application and in database tables.  When single tenant based application gets converted to multi-tenant application,  one would see the introduction of 'Tenant ID" in each database table schema.   Note that the tenant is not equivalent to user.  Tenant is typically a company to which the SaaS application provides the services for.  Each tenant would have its own set of users - typically employees to access  its resources in the cloud application. 

Some of the characteristics of Cloud applications with respect to Authentication:

• Tenant identification :  In many cloud applications,  email address is taken as the login ID.  Email address consists of user name and domain name.  Domain name is, typically , used to identify the tenant instance.  If email address is  xyz@example.com, then xyz is user name and example.com is the domain name.
• Each tenant has their own authentication database (user database) :  Cloud application vendors are increasingly allowing  their customers (tenants) to provide  authentication server information.  It would eliminate the need for duplicating the user accounts in the cloud applications by companies (tenants). Cloud applications are expected to get the appropriate authentication server information from the domain name of the login and validate the user by communicating with the authentication server.  Since tenants might use different types of authentication servers and hence it becomes requirement for cloud application to support multiple authentication protocols. 

Any authentication framework for cloud applications should satisfy following requirements (this list is beyond what is available in spring-security in its 3.0.5 version)

•  Ability to identify the tenant from the login credentials
• From domain name of email address.   Note that  multiple domain names might be part of one tenant ID.  That is, one should not assume that there is one-to-one correspondence between domain name and tenant ID.   Multiple domain names for a given tenant ID can arise  for multiple reasons -  Mergers of companies where domain names of merged companies exist for few months after merger,  Multiple divisions of company may be represented in the domain name portion of the tenant.  
• Ability to identify the tenant ID using application specific mapping table - Domain name versus tenant ID.
• As a separate POST variable indicating the tenant ID.
• Ability to get the Authentication Server information from the RDBMS where the authentication server information is stored based on the tenant ID.  Authentication Server information based on the tenant ID is part of the application specific RDBMS table.
• A tenant might have multiple authentication servers. One can apply different strategies to select the authentication server.  Some of the strategies that are useful are:
•  Round-Robin:  Yet times, the tenant has multiple authentication servers either for load distribution and for high availability.  In those cases, this strategy might be chosen.  If this strategy chosen,  servers are chosen on round-robin fashion.
• Until authentication is successful :   Multiple authentication servers might be provided in order of priority.   If this strategy is chosen,  authentication servers are tried in the roder until authentication is successful or until all authentication servers are tried.
• Sub-domain based authentication server:  If this strategy is chosen,  then the set of authentication servers is chosen based on the domain name of email address.  That is,  domain name is not only used to determine the tenant ID, but also is used to get the list of authentication servers.  Then the above strategy can be applied on this list of authentication servers.
• Ability to support multiple authentication protocols:  As discussed above,  Cloud application vendor will not be able to mandate their customers (tenants) to go with one kind of authentication servers.  Though LDAP and Active Directory (which also can be accessed using LDAP) are most commonly used, it is necessary that framework allows cloud applications work with  multiple authentication protocols.   It is expected that the information regarding communication protocol such as x.509 certificate and Certificate chain in case of SSL protocols would be expected from the application specific tables in RDBMS.  This information can come along with authentication servers' information. 
• LDAPv3 with/without SSL:  I believe SSL is must, as one would not like to send the user credentials in clear on the wire.
• CAS (Centralized Authentication Service)
• SAMLv2  (IDP is fast becoming common choice of authentication in recent past).
• OpenID
• And more ...  (Though RADIUS is one popular protocol, I don't see it being used that often, may be it is due that RADIUS is based on  UDP and related security concerns).
• Firewall Traversal while communicating with authentication servers:  Some companies (tenants) though like cloud applications to authenticate their users with company authentication database, they don't like to open the firewall hole for cloud applications to make a connection to their servers.  Some company administrators are paranoid to create an inbound hole in their firewall. But they are fine with creating outbound hole. In those cases, it is required that framework provide a proxy that can be installed in the company premises behind firewall.  Proxy is expected to work as proxy between cloud applications and internal authentication servers.  Proxy is expected to make a persistent connection to the cloud application server all the time.  Cloud applications, rather than communicating with authentication server directly,  it sends the authentication messages to the proxy on the established connection and proxy in turn makes the connection to the authentication server and proxies the messages to the authentication server. Similarly, it receives the messages from the authentication servers and forwards them the cloud application server using pre-created connection. 

Authorization Requirements

Traditional single-tenant based web applications can work with authorization functionality provided by spring-security.   Method/Class based authorization and page level authorization is good enough for many single-tenant based web applications.  spring-security provides mechanism to associate method/page/class with granted authorities.  As we discussed earlier, each user is associated with the roles in the authentication database.  As part of authentication process, in addition to validation of user with the authentication database, the authentication process also gets the authorities (Roles) of the user and keeps it in the security context.  At later time during authorization phase, this information is used by spring-security.

As I understand spring-security in later versions introduced the concept of ACL and domain objects.  This seems to be promising.  Using this new framework,  application business logic can create ACL which consists of 'Domain Object',  'Roles' and 'Permissions'.  Domain object is reference to the application business level object.  'Roles' is list of roles that are allowed to access the domain object and 'Permissions' is set of bits indicating permissions given to the roles to access the domain object.  Though this seems to be promising for cloud applications,  it still does not meet the requirements of cloud applications.

Characteristics of Cloud applications with respect to Authorization:

• There are multiple types of users in Cloud applications
• Users of Cloud application vendor.   These users are typically administer the cloud application as a whole.  There are different users with different roles to operate different parts of the application.  
• Users of tenants:   Cloud applications, as discussed above, support multiple tenants.  Each tenant has multiple employees.  Based on the type and authority of each employee, access to different resources of the application would be permitted.    Each tenant might define different roles.  That is, role names across different tenants might not be same in the system.
•  Each user is configured with the roles in the authentication database.
•  Support for Collaboration :  Some (not all) cloud application require a resource of one tenant to be accessible to other tenants.  Hence some cloud application provide ACL for a given domain object with not only owner tenant ID, but also with respect to other tenants.  That is, ACL is expected to consist of multiple rules for a given domain object with each rule having tenant ID and permissions.

Any authorization framework for cloud applications should satisfy following requirements

•  Ability for application to specify (add/delete/modify) the ACL for needed domain objects:
• As discussed, each ACL record in the ACL table corresponds to one domain object.  To allow multi-tenancy,  Domain object needs to be represented by tenant ID,  application name of the domain  and domain object identification.  Each ACL itself consists of multiple rules - Access Control Rules.  They can be represented in a separate table with each row corresponds to one rule.  Rule consisting of  "Tenant ID"/'Group of tenant IDs",  "Roles of that tenant ID" that can access the domain object, "Permissions" on the domain object.  Due to Collaboration based Cloud applications,  tenant ID in the rule can be other tenants than the tenant of the domain object in the ACL.   To simplify the configuration of collaboration related AC rules,  tenant IDs may be grouped and groups can be reused across multiple AC rules of different ACLs.
• Ability for business logic to find out whether the authenticated principal and his/her authorities can access the domain object that is being accessed. 
• Performance considerations:  Since it is possible to have ACL for many domain objects,ACLs can become very big in the system.   It may be necessary to have reference to the ACL from the domain object. This could avoid costly query of the ACL table.  
• Ability to inherit the ACL from the parent domain object: 
• Cloud applications have different resources and configuration of ACL and associated rules can become very complex.  Inheritance of ACLs can help reduce administrator overhead.   Domain objects that are being configured with ACLs can inherit the ACLs of its parent or grand parent or any other resource. It does not seem to be good to allow inheritance of the ACLs from domain objects of other tenants.  To allow inheritance, ACL for the domain object can be configured with list of  'Inherit from" and the identification of the parent domain object - Application name of the domain and its identification of the domain object.  In addition to inheriting it from the other ACLs, it can also its own Access control rules. 
• Due to inheritance,  there could be multiple ACLs to search for a given domain object - Its own ACL,  multiple ACLs as it can inherit from multiple other domain objects.  Also Domain objects, from which ACLs are inherited from, may themselves be inheriting ACLs from others.  Due to multiple ACLs, order of ACL to find the permissions is important. It is always good to check its own ACL first and then first level inheritance and second level inheritance and so on.  

Auditing Requirements

Any configuration changes to some important resources of an application must be logged for later auditing.  Since cloud applications have multiple tenants,  tenant ID must be logged along with configuration audit logs. 
Configuration audit logging must have information related the exact changes that were applied to the resource.  Addition of any application record must have all the information (field names and values) in the audit log.  In case of deletion of any application record,. audit log  must have similar information as 'Add'.   Modify operation should result in audit log having old information and new information. 

It is also important that the audit log also contains Date & Time at which the update happened,  User name of the user that changed the configuration,  role that was used to give the permission. 

Internet Resources on Multi-tenancy and spring-security.:

I have found two valuable resources related to Multi-tenancy and spring-security.  It gives a good picture on how spring-security can be enhanced to support multi-tenancy.  Though they solve some of the requirements mentioned above, they don't  satisfy all of above requirements. But it gives  very good understanding of how spring-security can be customized.   This knowledge can be used to customize spring-security for above requirements.

Securing a multitenant SaaS application 
Extend Spring Security to Protect Multi-tenant SaaS Applications

Experience with installing VMware Server 2.0 on windows 7

I was helping a friend to install Ubuntu 10.10 Linux on Windows 7 using VMware server 2.0.

I  had few hiccups, but finally could do it.

• Installation of VMware Server was a breeze.  No issue found in downloading and installing.  Just follow the instructions given in VMware site. 
• After VMware Server was installed, you would see few entries  at "Start"  of windows 7,  All Programs->VMWare->VMware Server->VMware Server Home Page.   Click on it.   It opens your browser.
• It asks for user name and password.  This is where I was stuck.  I was giving all my windows 7 user names and passwords, it could not succeed.  Finally, I found from searching VMware site that VMware server always expects the user name "Administrator".  By default windows 7 does not enable administrator account with "Administrator" name.  To do this, you need to execute following command at the command line prompt. 
• Command window must be opened with "Run as Administrator".  You can do this by going to "All Programs->Accessories".  Here you would see "Command Prompt".  Take your cursor there and press right click on that line.  You would see "Run as Administrator".  Click on that. It opens the command prompt window.
• There you should type "net user administrator /active:yes".  This would be enable hidden "Administrator" user.
• Next step is to assign the password to "Administrator" user.  Go to 'Control Panel->User Accounts and Family Safety->Add or remove user accounts".  Here you would see "Administrator" user. Click on it and add a password to it.
• Now you should be able to login in to the VMware server using "Administrator" and password you have set for the user.
• Then I was trying to install Ubuntu.  First step is to create a virtual machine. You could do this using "Virtual machine" menu.  Click on "Create Virtual Machine" item.  It takes you to set of screens to assign the memory,  hard drive space,  network etc..  These are all simple and you could follow the documentation.  As far as the Ubuntu image is concerned, I thought I could have iso image in my windows hard drive somewhere and could point to that. But I was not successful in giving the path for ISO image anywhere.  Finally I decided to burn the ISO image on CD and selected 'DVD' option.
• So far so good.
• Once the virtual machine is created, I started it.  I wanted to see the console of virtual machine. Selected "console" menu.   It asked to install a plugin.  Clicked on it, it installed the plugin. Then it asked me to click on anywhere on the window to open the console.  Unfortunately, I keep getting error that "request timed out".  Finally, I found that Firefox can't be used to open the console.  So, make sure that you always use "Internet Explorer".  Once I started using Internet Explorer, I could see new console window getting opened.
• After that,  ubuntu installation started on that console window and now I have ubuntu working on Windows 7 using VMware 2.0

Hope it helps.

Srini

IGMP Filtering - Developer tips

IGMP protocol allows hosts to report their interest in Multicast address membership with adjacent routers.  These adjacent routers in turn propagate the consolidate membership with upstream routers using PIM-SM or using IGMP proxy functionality.

IGMP protocol sits right on top of IP layer, at the same level as ICMP, UDP and TCP.  IGMPv1 and IGMPv2 protocols are older protocols to IGMPv3.  IGMPv1 and IGMPv2 join the multicast membership by providing Multicast address in IGMP report message.  IGMPv3 takes one step further and even ask for membership with respect to source.  That is, there could be  multiple Multicast sources  sending same streams using one Multicast address. Hosts have choice of receiving the Multicast traffic only from some specific sources - It can specifically include the source addresses in the report message or can exclude some specific sources.

Typical firewalls today don't have capability to restrict the hosts on a specific interface from joining specific Multicast stream.  Firewalls today can allow or deny the IGMP packets on a specific interface though. But this is not sufficient.  It should be possible to restrict a given multicast stream on a specific interface.  This is possible only if firewall on the router deny only IGMP reports having restricted Multicast addresses.

IGMP filtering functionality in routers expected to provide following functions.  Some vendors call this 'IGMP filtering for Multicast authentication'.  I never understood why they use term 'Authentication' though.

• Provide facility for admin user on per interface basis 'Allow Only List' or 'Disallow list'.  Each list contains multiple records. 
• Multicast Address,  Source address.
• Source address can be 'ALL'.  

Some implementation considerations:

It is always good to implement this module separate from IGMP Router or IGMP Proxy modules.  In case of IGMPv1 and IGMPv2,  each membership report message only contains one Multicast address.  If this multicast address is to be restricted,  then the complete message can be dropped.  In case of IGMPv3,  one membership report message can contain multiple Group records with each group record having multicast address, source addresses and qualifier to all the source addresses listed in the group record - include or exclude.  IGMP filter module needs to do quite a bit of work to identify the group records and corresponding sources and remove only the affected ones from the message.  Then rest of the message should be allowed to pass through.  If the complete Multicast address is restricted as per configuration, then the complete Group record from the IGMP report message can be removed.  It is not as simple as though.  From the configuration, some times only specific source addresses are to be removed from the group record in the message. That is, IGMP filter is expected to selectively remove not only group records, but also source addresses selectively in the group records.

Clustering of devices with traffic distribution by L2 Switch - One limitation & Mitigation

In my last post on "Data Center/Enterprises Clustering of Devices"  I discussed on how L2 switches are enabling device equipment vendors to provide cluster solution to take up the increasing load on the networks.  Many L2 switches are capable of analyzing multiple different types of  layer 2 headers to get to the inner IP packet and use inner IP packet source and destination IP address fields to distribute the traffic across multiple devices in the cluster using hash distribution.  L2 switches typically understand Ethernet and MPLS related headers such as Ethernet DIX,  LLC/SNAP,  802.1Q VLAN headers and MPLS Label headers.  That is, L2 switch can get hold of IP packets if packets are sent over above mentioned L2 headers. 

In some deployments, IP packets may be encapsulated in multiple L2 headers or L3 tunnels.  Some examples are : PW (Pseudo wire) header,  Ethernet over Ethernet using PW,  GRE/UDP,  GTP/UDP, IPinIP,  Mobile IP and may more.  L2 switches in the market today are not capable of understanding these headers to to get to the inner IP header.   In these cases, distribution based on inner IP header fileds will not be possible.  In these deployments, L2 switches may need to resort to distribution based on L2 header fields such as source and destination MAC or tunnel IP header fields.  Unfortunately,  distribution based on these fields may not be good at all.  If you take an example of this cluster being places between two routers,  MAC addressees of every packet traversing the cluster will be same.  Hence any distribution based on the MAC addresses would go to only one device in the cluster.  Similar would be the case, if distribution depends on the tunnel IP header fields. 

Switches have one capability though.  They can generate the hash based on calculated CRC on some part of the packet or CRC of the Ethernet packet.   CRC of the Ethernet packet can be assumed to provide good distribution as CRC on Ethernet packet is based on the complete Frame payload, that is, including the inner IP packet payload. Switches have capability to take few bits of CRC to distribute the packets across multiple cluster devices.  But the issue with this is that packets belonging to same connection would go to different cluster devices.  Base on my earlier post, cluster devices assume that packets belonging to a connection would always land on the same device. This assumption is no longer true if the cluster solution is being deployed in above mentioned environments.  These types of environments are not common at all in Data Center and Enterprises environments.  So, this problem may not be there in many instances. But service provider environments, multiple L2 and tunnel header situations are not uncommon.

How do cluster solutions work in these environments?

In these environments, if CRC based distribution is used,  switches are really doing packet sprinkling across multiple devices.  In these cluster devices should have additional intelligence.  

• Cluster devices should be able to get past the headers to get to the inner IP header.
• Cluster devices among themselves should have understanding of session distribution. One simple method is to do what switches were doing.  That is, they can generate the hash on the IP header fields (source and destination IP) and figure out the device which got the packet is the one which needs to server based on the hash value.  If it is,  it should continue processing the packet. If it is not, then it should give the packet to the device that owns the hash value.  

There could be good amount of traffic among cluster devices.  They can use the switch as their back plane to send and receive the traffic among them.  To avoid other device doing same thing, that is getting hold of the inner IP header and hashing on the inner IP header fields again,  the sending device can send this information along with the packet and receiving device can avoid doing same operations again.

As indicated above,  only few deployments where L2 switch does not do inner IP header based distribution.  If same cluster solution being provided for all kinds of deployments, then it is good for network equipment vendors of cluster solution to provide configuration on the cluster whether the packets which are being distributed by L2 switch is packet sprinkler or intelligent IP flow based sprinkling.  If it is packet based sprinkling, then additional logic in devices can kick in to figure out the real destination device.

Data Center/Enterprises - Clustering of Network devices

Throughput requirements of Data center/Enterprise network equipment are going up with increased traffic in data centers and Enterprises.  In addition,  computational requirements of network equipment are also going up.  Some examples of why more computation power is required.

• Intrusion Detection/Prevention now requires almost 3 - 4 times the  computation power on per Mbps of traffic than what was required few years back.  I guess it is mainly due to sophisticated nature of attacks and evasion techniques adopted by attackers.  Javascript analysis itself takes 10 times computational power  than the typical DPI based pattern matching.  Javascript analysis requires proxy based functionality to get hold of the javascript and script analysis for attack detection.  These two tasks require lot more CPU cycles than typical pattern matching.
• Traditional Server Load Balancers (SLB) used to select the internal server based on the IP, UDP/TCP header values.  Next generation server load balancers (SLB) called ADCs do deep packet inspection, such as HTTP, SIP URL,  HTTP Request headers,  to select the internal server to send the load.  DPI requires more CPU cycles.
• Application Firewalls such as Web Application Firewalls and SIP firewalls not only  do the deep packet inspection, but also deep data inspection and that  requires more horse power from CPUs.
• DDOS prevention requires real time analysis of not only packet-by-packet analysis, but also sessions and application protocol level analysis across sessions to identify the attacks.  Many DDOS attacks on per session basis look exactly same as the normal traffic.  Hence the analysis across sessions is required for detecting the anomaly.  This capability requires not only lot of memory, but also good amount of computational power.

Multicore processors are helping some extent in solving performance issues.  Clustering of multiple Multicore SoCs are becoming necessary to solve above performance issues in Data Center and Large Enterprise markets.  Typically,  multiple blades, each using Multicore SoCs, running the same application are clustered to take up the load. L2 switches are increasingly used to front end the cluster.  L2 switches now can be configured to balance the load across multiple devices of cluster.  One might see the cluster and L2 switch in one enclosure giving a feeling that it is one big box providing tens of gigabits of performance.

What features of L2 switch are enabling clustering?

• Distribution of sessions across multiple devices in cluster:  Majority of L2 switches have capability to distribute the  traffic coming from incoming ports (Data Ports) across multiple ports (Device ports).  By connecting devices in the cluster to these ports, then each device gets the traffic that was redirected to that port. But many of the network devices expect that all packets of any given session go to the same device.  For example, all packets belonging to one HTTP connection should go to one device.  If packets of the sessions are distributed across multiple devices,  they will not be able to do their operation of analysis, proxy etc..    A given connection traffic involves both Client to Server and Server to Client traffic.  Though L2 switches don't have session intelligence, due to the hash based distribution mechanism they adopt,  same hash value gets generated for session traffic whether it is C-S or S-C traffic of a connection.   Some cautions:
• L2 switches don't do IP reassembly.  Due to this, hash generated for first fragment of a packet can be different from the non-initial fragments if the hash generation block is configured with L4 fields (TCP source and destination ports).  So, it is advisable to configure the hash block with IP addresses and IP protocol.   This may give rise to unequal distribution. But with large number of sessions in DC, this may not be a big limitation.  
• Some application sessions require multiple connections. Example:  SIP (Session Initiation Protocol).   SIP voice call typically involves three connections - SIP control connection,  RTP for voice/video data and RTCP for control frames.  Many devices expect that all three connections land on the same device.  If all three connections have same source, destination and protocol fields, then all packets of SIP application session would be sent to the same device by the switch.  But, RTP and RTCP IP addresses may be different from the IP addresses of SIP control connection.  If your device needs to support this,  then it is responsibility of cluster devices.  Cluster devices need to have intelligence of ownership of these kinds of  application sessions. If a device receives packets belonging to application that is owned by some other device, it needs to redirect the traffic to that  device that owns the SIP session.
• As indicated implicitly above,  L2 switch port are divided into - Network ports (Data ports) that connect to the DC/Enterprise networks and Device ports where the cluster of devices are connected.  With large density of ports in current generation of switches, some ports even can be dedicated to inter-device communication, there by avoiding any other back plane such as infiniband or some other L2 switch fabric.  L2 switches and devices providing ETS (802.1qaz)  and 10G ports'  support can use the same port for both inter-cluster communication as well as for network traffic. 

New Generation Configuration Framework

Even though there are multiple devices in cluster,  it is required that admin user configures the cluster only once.  Admin users should not be expected to configure each device in the cluster.   Fortunately new generation of configuration framework are designed to handle cluster configuration. 

New generation configuration frameworks support the mechanism to ensure that configuration is same across the devices in the cluster.  Increasingly,  configuration architecture supports central management system which takes care synchronization of configuration across devices on per operation basis.

Network devices maintain several statistics. With multiple devices in the cluster, each device maintains its own set of statistics. Admin user typically expects to see the consolidated list of statistic counter values across all devices in the cluster. Again, new configuration frameworks reads the statistics from each device, consolidates them and show the consolidated output. 

On image upgrade : When new image version is available,   new configuration frameworks allow admin users to upgrade the image only once for the cluster. All devices in the cluster would get the image from the central configuration framework.

With these advancements in L2 switches and configuration frameworks,  clustering is again back in the networks.

Data Center Equipment & Offload card requirements

I  have been attending some web conferences focusing on Data Center equipment.  From last few years,  there are more and more discussions on offloading some processing out of current data center equipment. Before getting details on offload functions,  it is good to revisit some of data center equipment.

Data centers, whether they are private, public or cloud,  have following equipment.

• L2/L3 Switches and Routers
• Server Load balancers or Application Delivery controllers.
• WAN Optimization devices
• Network Security Devices.
• Monitoring Servers for network visibility, Surveillance and others.
• Application Servers (Web, EMAIL etc..)
• Database Servers
• Storage Devices.

At this time, all servers are x86 servers.   I gather that, many of the network equipment such as SLB, ADCs, WOC, UTMs are also implemented on x86 servers with Crypto, Compressions and Pattern matching done on add-on PCIe cards.  Some vendors of these equipment are seriously considering moving to embedded multicore SoCs.   There are two camps.  One camp of vendors want to continue with x86 for their applications, but use embedded Multicore SoCs for offloading some functions beyond algorithmic offloads.  Other camp of vendors would like to replace multiple x86 processors with multiple Embedded Multicore SoCs.   

This post talks about networking functions that equipment vendors looking to see in offload cards. There are two types of offload cards that are becoming popular - PCIe cards that can go into x86 based network equipment and  proprietary cards that go into L2/L3 switch. Both kinds of cards are expected to take away some processing out of networking equipment. If it is PCIe card, more offload functions are possible due to proximity with the target application.  In this post,  I am going to list down the offload functions that are possible and expected by the networking equipment vendors.

PCIe offload cards act NIC card to the x86 equipment while doing offloading of some networking functions.  Hence some call them  intelligent NIC cards. Offload card receives all the packets from the network, it processes offload functions and then sends the result packets to the server via PCIe interface. These packets are presented to the server TCP/IP stack via PCIe Ethernet driver.  Similarly outbound packets would come to offload card from the server.  Offload card does offload functions necessary on egress traffic and sends the result packets out on the network.

List of offload functions in different categories are given below.

Algorithmic Acceleration:  This is already being done and requires no further explanation.  Algorithmic acceleration now typically involves

• Public Crypto and Symmetric Crypto acceleration
• Symmetric Crypto acceleration with Protocol offload such as SSL Record layer : Though this is provided by offline cards,  increasingly network applications requiring SSL are using AES-NI SIMD functions provided Intel SSE4.  Many PCIe based crypto accelerators are mainly used for public crypto acceleration.
• Compression/Decompression  acceleration.
• Regular expression Search acceleration.

Basic Networking offloads functions:

• Offloading of ARP for IPv4 and ICMP based neighbor discover in case of IPv6:  As networks in the data center becoming flatter, there is more and more ARP broadcast traffic in the network.  TCP/IP stack of network equipment receives all broadcast packets.  Due to increased broadcast traffic,  CPU cycles are being spent in processing interrupts, and ARP processing.  Lot of times, ARP requests may not even be satisfied as there is no neighborhood entry.  Offload cards are expected to remove this overhead by responding to neighborhood requests by itself without involving the server TCP/IP stack.  How does it work?
• Server TCP/IP stack is expected to create/delete neighborhood entries with local IP addresses and associated MAC addresses  in the offload cards whenever new entries are created locally and whenever they are deleted.  Offload card maintains the neighborhood entries database on per physical and logic port basis (in case of VLAN based interface).
• When offload card receives the neighborhood request, it checks  its database first. If there is no matching cache entry, ARP packet is dropped.  If there is any matching entry, it processes the request and reply is sent to the network without involving server.  
• Offloading of packet integrity checks and dropping of invalid inbound packets:  Any TCP/IP stack including servers TCP/IP stack spend some CPU cycles checking whether the packet received is good.  These cycles can be saved if offload cards do the checks.  Some of the checks that offload card can do are:
• Check the SMAC of the packet :  If the source MAC is one of the local MAC addresses, then packet can be dropped by offload engine.
• Check the IP length with the packet size:  If the IP length is more than the packet size,  it is a bad packet and packet can be dropped.  If the IP length in header is less than the packet size, then packet needs to be truncated to length in IP header.
• Check the checksum of IP packet:  If the checksum  is bad, then offload card can drop the packet.
• Check the TTL:  If TTL is 0, then the packet can be dropped by offload engine.
• Check the SIP and DIP of IP header:  If both the addresses are same, then packet can be dropped by offload card.
• Check the SIP of IP header with local IP addresses:  If the source IP is one of local server IP addresses, then the packet can be dropped by offload card. Similarly, if the source IP addressis one of the local broadcast addresses or multicast address, packet can be dropped by the offload card.
• Transport header length checks:  If the length of the packet is not as per the length in the transport header, then the packet can be dropped by the offload card.
• Transport header checksum checks:  If the checksum of the packet does not match with checksum in the header, packet can be dropped by the offload card.
• When the checksums are verified,  to avoid similar checks in the server TCP/IP stack,  offload card can communicate the successful checksum verification indication along with each packet to the server TCP/IP stack.  Many server TCP/IP stacks such as Linux has capability to indicate this information in their buffers.  For example, Linux TCP/IP stack has capability to avoid checksum verification if some special fields are set in SKB.
• Offloading of IP reassembly on inbound packets:   Offload Engines by doing IP reassembly and sending one single packet to the server TCP/IP stack can save good number of CPU cycles. Any server terminating any kind of tunnel (such as IPsec, GTP, GRE, IPinIP etc..) would tremendously would benefit from this offload.  Offload cards are expected to do reassembly of IP packets that involve two packets at the minimum as many tunnel protocols make two fragments from the IP packet. IP reassembly function is not only required to save some CPU cycles in server, but also required to do other offloads such as 'Traffic policing',  'stateless and stateful filtering',  'inline DPI offload'  as they require access to transport selectors.  These offloads will not work well on packets if they don't have transport selectors.
• Offloading of IP fragmentation on outbound packets:  This offload again saves CPU cycles by moving this function from server TCP/IP stack to offload engine on outbound packets from the server.  Server TCP/IP stack can give PMTU value along with the packet.  Offload card can fragment the packet as per PMTU and send them out. By offloading this functionality to offload card,  offload card can do better job of traffic shaping if it involves transport selectors.  IP Fragmentation can be done after the traffic shaping there by allowing traffic shaper to look at full IP packet.
• Offloading of hash calculation  on inbound packets:  Many networking applications running in x86 server require hash result to index into  hash buckets of their session blocks.  Bob Jenkins,  CRC-32 and CRC-64 are some of common hash algorithms used on packet fields.  Offload cards are expected to take configuration from x86 server networking applications.  Configuration include 'hash algorithm', 'packet fields' and their order. Offload cards, then expected to, extract the fields, do the hash on the fields and send the hash result along with the packet to the server TCP/IP stack. 
• Packet parsing & Field extraction on inbound packets :  Almost all networking application require to extract some common fields from the packet.  Cycles required to extract the fields can be saved by offloading this part to the offload cards.  Server networking applications can program the offload card on the fields it is interested in.  Offload cards based on configuration extracts relevant fields and provide them along with the received packet.

Inbound Packet filtering

Non-security network equipment devices would normally service certain protocols.  Unrelated traffic that goes to the devices would only decrease the performance of the device. Offload cards can filter out the packets.  Device software can program the rules to allow only the traffic required to go to the devices. Offload cards can reject the traffic that does not match the rules programmed.

Traffic Policing of inbound packets

Network equipment devices run complex operations on the packets or data. For example, ADC/WOC/Security devices do SSL, TCP, HTTP and other application protocol proxying to act on the data.  Proxy operations are expensive in nature.  Even the best of equipment can't process more than few gbps. If more traffic is sent towards the device, packets would be dropped at some point and  it only worsens the performance of device.  Devices are typically rated for some performance for different types of traffic workloads.  Offloads cards are expected to provide traffic policing on different types of traffic and discard the packets that exceed the policing parameters.  Device software is expected to program the offload cards with different types of protocols and traffic policing parameters.

TCP stateless offload :

Based on some conversations with DC equipment vendors as well as offload card vendors,  it appears that majority of them don't like to have TCP termination in offload cards and corresponding TCP/IP stack bypass in the x86 device.  It appears that device vendors are happy with device Linux or NetBSD operating system's TCP/IP stack.  Only peripheral changes to the TCP/IP stack and new Ethernet driver is only thing allowed by these vendors due to offload cards.  Maturity of Linux TCP/IP stack,  constant upgrade of TCP/IP stack with latest standards (RFCs),  avoiding application porting or testing of system are some of the reasons cited.  

Due to this, offload cards are expected to provide only TCP stateless offload. TCP stateless offload functions increase the performance of proxy applications.

• Offload of hash function :  This is described above under 'Basic networking offload'.  TCP layer of device TCP/IP stack uses hash function to find the TCP control block (TCB).  If offload card does this, then device TCP/IP stack can skip this and save some CPU cycles.
• Large Receive Offload :  As described in the post "Linux TCP Large Receive Offload optimization to increase performance" LRO feature reduces the number of packets that are seen by the device TCP/IP stack. Lesser the number of packets, better the performance of TCP/IP stack. Offload cards can do this by implementing LRO function by aggregating multiple consecutive TCP packets of a flow of a connection into a single large packet.
• Transmit Segmentation offload :  Similar to LRO, this feature reduces the number of packets that traverse the device TCP/IP stack, but this time on the transmit packets.  Device TCP stack normally segments the packets based on MSS which is PMTU-TCPheader length - IP headerlength.   This would increase the number of packets on transmit direction and also some cycles are spent in segmenting the big data.  TSO functionality in offload cards can reduce the number of packets and also save CPU cycles from segmenting the packet in the device.  Device TCP stack can provide the big packet along with segment size and PMTU to the offload card. Offload card along with TSO and IP fragmentation functionality can segment and if required fragment the packets before sending them out on to the network.
• ACK offload : As we know,  TCP/IP stacks are expected to send at least one ACK for every two TCP segments it receives.  Hence, there would be large number of ACKs.  Similar to TSO,  reducing the number of ACKs traversing the device TCP/IP stack can increase the performance of device.  Device TCP/IP stack generates the template ACK with the number of ACKs to be generated and corresponding sequence/ACK number information to the offload card.  Offload card, then, can generate multiple ACKs from the template ACK.

Traffic Shaping of outbound packets

Many network equipment device types do some sort of traffic shaping on the outbound packets.  Traffic shaping is almost the last step on the outbound packets.  QoS or traffic shaping is quite the same in all devices and there is really not much a value addition by the device software.  Hence this function can be done by the offload cards without impacting any device functionality.  Traffic shaping policies can be programmed in the offload cards by the device software.  Offload cards can act on the outbound packets based on the configuration.  It is important to that offload cards support multiple scheduling algorithms and hierarchical shaping.  This function in offload cards can save good number of CPU cycles and hence more CPU power can be made available to the device applications.

IPsec termination offload

Some network equipment devices such as network security devices implement IPsec VPN.  For inbound packets,  IPsec is done in the early stages of packet processing. For outbound packets, it is done one step before traffic shaping.  Hence this is also good candidate for offloading along with the traffic shaping.  By offloading IPsec inbound SA processing,  device software only sees the clear packets.  Similarly,  by offloading Ipsec outbound SA processing, device software does not need to do the expensive IPsec outbound processing.   Device software is expected to program the tunnels (pairs of Inbound and outbound SAs) upon IKE negotiation and delete them when life time expires or when there explicit delete indication from the remote gateway.

Offload cards are expected to provide IPsec termination offload function.  Offload cards not only do outbound and inbound SA processing, they also expected to do inbound policy verification on both decapsulated inbound packets and also on the original clear packets.  On the outbound side,  offload function can expect device software to provide outbound SA reference along with the packet. 

Inline DPI offload on inbound packets

Network equipment devices having IDS, IPS and Application detection functionality require some kind of pattern match on the packets.  These applications analyze the TCP, UDP, IP payloads for some detections and for others it may require processed data of L7 protocols.  Offload cards have access to TCP, UDP and IP payloads and may not have L7 intelligence.  At least for TCP, UDP and IP payload based detection, offload cards can be used.  Device software programs the patterns in the offload card and its regular expression engine.  Offload card after doing all its operations on inbound packets such as 'IP Reassembly', 'LRO', 'Packet filtering', it does pattern match on the payload and sends the results along with the packet.  Each result is expected to have pattern ID, offset in the payload.  Device software can avoid pattern matching on basic protocol payloads and there by utilizing saved CPU cycles on other processing.

Multicore Distribution

Network equipment devices normally use multicore x86 processors.  Without any support of offload cards,  distribution of packets to multiple cores happen either via software based interrupt distribution or by affining interrupts to different cores.   Some NICs have capability of distributing the load across multiple large number of descriptor rings with each descriptor ring associated with one interrupt. When these kinds of NICs are replaced with the offload card, it is expected that offload cards also provide similar capability of distributing the inbound packet load across multiple descriptor rings with associated interrupt.   In addition to this,  offload cards are expected to place traffic of a given connection (whether it is client-to-server or server-to-client) on to the same descriptor ring.  That is hash value calculated on 5 tuples should be same whether it is C-S and S-C traffic of a connection.  As we know,  SIP, DIP are in opposite order in IP header between C-S and S-C traffic a connection. Same is true with TCP and UDP ports.  Some hash algorithms generate different values if the order of input to the hash algorthm  is different. To ensure that same hash is generated,   offload cards are expected to arrange the input to hash in  fashion that input is same for both C-S and S-C traffic.  One method of doing this is to pass the input based on the value of the field.  That is, always lesser value of SIP and DIP can be passed first to the  hash algorithm before passing higher value among SIP and DIP.  Same needs to be for SP and DP too.  This will ensure that same hash value is generated.

Device Ethernet Driver and offload card should have understanding of number of receive descriptor rings and accordingly the offload cards uses the hash result to map to one of the descriptor rings.

Virtualization Distribution

Some network equipment devices support virtualization with each VM serving different DC customer traffic. VLANs are typically used to distinguish different customer traffic.  Using IOMMU functionality of x86 and PCIe SR_IOV functionality, offload cards are expected to send the inbound traffic directly to the VMs memory without involving the hypervisor layer.  IOMMU also allows direct mapping of different descriptor rings to the user space processes.  For more information on IOMMU works, please refer to the AMD paper on IOMMU specification.

Multiple cores might have been assigned to VMs.  In those cases, this distribution should be combined with the 'Multicore Distribution'.

Miscellaneous

Since offload cards are intelligent NICs and they are part of Data center environment,  data center bridging functionality is expected to be provided by NIC cards, even though these are not offload functions.  Some of the functions expected are:

802.1qaz:  Enhanced Transmission Selection

802.1qbb: Priority based flow control

802.3bd:  MAC control frame for priority based flow control.
802.1 Qau: Congestion notification for end points to take action.

802.1qbg and 802.1qbh :  Virtual bridge (VEPA, VN-Tag)

I hope it helps. If you find any functions that can be offloaded, please drop a comment.

What are Traffic Monitoring Enabler Switches?

There is increasing trend of  Traffic Monitoring Enabler Switches (TMES) in Enterprise, Data Center and Service provider environments.

Need for TMES:

Traffic monitoring devices are increasingly becoming requirement for networks in Enterprise, Data Center and Service provider environments.  There are multiple types of monitoring devices are being deployed in networks.

• Traffic Monitoring for intrusion detection:  Security is very important aspect of Enterprise networks.  Intrusion detection is one of the components of comprehensive network security.  IDS devices listen for the traffic passively and do the intrusion analysis on the traffic.  Intrusion attempts and intrusion events are sent to the administrators for out-of-band action.  IDS devices also can be configured to send TCP resets in case of intrusion detection to stop any more traffic going on the TCP connection.  IDS devices also can be configured to block certain traffic for certain amount of time by informing local firewall devices. 
• Surveillance:  Due to government regulations, all important data needs to be recorded.  Surveillance monitoring devices again listen for the traffic passively and record them in persistent storage for later interpretation.  Surveillance devices also provide capability to recreate the sessions such as Email conversations, file transfer conversations,   Voice and video conversations from the recorded traffic.  Some surveillance devices also provide run time recreation of conversations too.  
• Network Visibility :  These monitoring devices capture the traffic passively and provide complete network visibility of the traffic. They provide capabilities such as 'Identification of applications such as P2P, IM,  Social networking and many more ',  'Bandwidth usage of different applications,  networks'  and provide analysis for network administrators with valuable information to maintain networks and bandwidth to make  Enterprise critical applications work always.
• Traffic Trace:   Traffic trace devices help network administrators to find the bottlenecks in different network segments.    These devices tap the traffic at multiple locations in the network and provide the trace capability for finding out the issues in network such as misconfiguration of different devices in network,  choke points etc..

Network administrators face following challenges to deploy multiple monitoring devices.

• Few SPAN ports in existing L2 switch infrastructure:  Many L2 switch vendors provide one or at the most two SPAN ports.  L2 switches replicate the packets to SPAN ports.  Since there are only two SPAN ports at the most,  only two types of monitoring devices can be connected.  This is one big limitation network administrators face.  
• Multiple network tap points :  In complex network infrastructure, there are multiple points where monitoring is required.  Placing  multiple monitoring devices at each point is too expensive. Network administrators would like to use same monitoring devices to capture traffic at multiple locations. 
• Capacity limitations of monitoring devices:  With increasing bandwidth in the networks, it is possible that one monitoring device may not be able to cope with the traffic.  Administrators would like to multiple monitoring devices of same type to capture the traffic with some external component doing load balancing the sessions across multiple monitoring devices.
• High Capacity Monitoring devices :  There could be instances where monitoring device can take more load. In these cases, one monitoring device can take load from several tap points.  Administrators look for facility to aggregate the traffic from multiple points to one or few monitoring devices of same type.
• Non-Switch capture points :  Network administrator may like monitoring of traffic in a point where there are no switches -  Router to Server,  Wireless LAN Access Point to Access Concentrator etc..   Since there is no switch, there are no SPAN ports.  Network administrators look for some mechanism such as Inline TAP functionality to capture the traffic for monitoring.

What is TMES?

TMES is a switch device with monitoring enabling intelligence to allow connectivity of multiple monitoring devices of different types without any major changes to the existing network infrastructure.

This device taps the traffic from SPAN ports of existing switches in the network and direct the traffic to attached monitoring devices.

TMES allows:

• Centralization of monitoring devices.
• Filtering of the traffic.
• Balancing of traffic to multiple monitoring devices of a given type.
• Replication of traffic to different types of monitoring devices.
• Aggregation of traffic from multiple points to same set of monitoring devices.
• Truncation of data 
• Data manipulation & masking  of the sensitive content of the traffic being sent to monitoring devices.
• Inline TAP functionality to allow capture points where there are no SPAN ports.
• Time Stamp functionality
• Stripping off  Layer 2 and Tunnel headers that are unrecognized by monitoring devices.
• Conditioning of the burst traffic going to the monitoring devices.

Centralization of Monitoring Devices: 

Without TMES,  monitoring devices need to be placed at different locations in the network.  With TMES,  TAP points are connected to TMES and monitoring devices are connected to only TMES ports.

Filtering of Traffic:

This feature of TMES allows filtering of unwanted traffic to a given monitoring device.  Monitoring devices are normally listen for traffic in promiscuous mode. That is, monitoring device gets all the traffic that is going on the wire.  But all the traffic is not interesting to the monitoring device.   Typically monitoring device itself does the filtering.  By offloading filtering out of it,  it saves valuable cycles in receiving the traffic (interrupts) and filtering the traffic.  TMES takes this load out of monitoring device and thereby increase the capacity of monitoring devices.

Filtering of traffic should not only be restricted to unicast. It should be made available even for Multicast and broadcast packets.

Balancing the traffic to Multiple Monitoring devices of a given type of monitoring:

If the amount of traffic that needs to be recorded is very high, then multiple monitoring devices will be deployed.   TMES allows multiple monitoring devices to take the load.  TMES load balances the sessions (not packets) to multiple monitoring devices based on performance of monitoring device.  By balancing based on sessions,  TMES ensures that all the traffic for a given connection go to one monitor device.

Replication of Traffic

When there are different types of monitoring devices, each device is expected to get the traffic.  As discussed above,  traditional L2 switches have at the most two SPAN ports.  TMES is expected to replicate the traffic as many number of times as number of different monitoring device types and send the replicated traffic to the monitoring devices.


Combining the replication feature with load balancing:  Assume that a deployment requires the traffic to be sent to two types of monitoring devices -  IDS and Surveillance. This deployment requires  that the 6Gbps bandwidth traffic to be analyzed and recorded.  If IDS and Surveillance devices can analyze and record only 2Gbps bandwidth, then the deployment requires 3 IDS devices and 3 Surveillance devices.  In this case, TMES is expected to replicate the original traffic twice - One for IDS devices and another for Surveillance devices.  Then TMES is expected to balance the one set of replicated packets to go to one of 3 IDS devices and second set to go to one of three Surveillance devices.  

Aggregation of traffic from multiple points to same set of monitoring devices

As discussed in 'Centralization of Monitoring devices',  traffic from different locations of the network can be tapped.   TMES is expected to provide multiple ports to receive the traffic from multiple locations in the network, filter the traffic, replicate and balance the traffic across monitoring devices.  It is possible that traffic of a given connection might be going through multiple points and hence there could be duplication of traffic coming to the TMES.  It is also possible that duplicated traffic might be going to the same monitoring device.  Monitoring device might get confused with duplicated traffic.  To avoid this scenario,  TMES is expected to mark the packets based on the incoming port on TMFS (that is the capture point) such as adding different VLAN ID based on the capture point or adding an IP option etc..    This would allow monitoring device to distinguish the same connection traffic across different capture points.




Truncation or Slicing of packets


Some monitoring device types such as traffic measuring devices don't require complete data of the packets to come in. By slicing the packet to smaller packet would increase the performance of those monitoring devices. TMESs are expected to provide this functionality before sending the packets to the monitoring devices. Truncate value is with respect to payload of TCP, UDP etc..  Some monitoring devices are only interested in headers upto layer 4.  In this case, truncation value can be 0.  Some monitoring devices may expect to see few bytes of payload. TMESes are expected to provide this flexibility of configuring truncate value.

Truncation of packet content should not reflect in the IP payload size. It should be kept intact to ensure that monitoring devices can figure out the original data length even though it receives truncated packets.



Data  masking 

Based on type of monitoring devices,  administrator may like to mask some sensitive information such as credit cards,  user names and passwords from being recorded.   TMES is expected to provide this functionality  of pattern match and mask the content there by removing privacy concerns.

TMES also might support Data replacement (DR).   DR feature might increase the size of the data. Though it is not a big issue for UDP type of sessions, it requires good amount of handling for TCP connections.  As we all know TCP sequence number represent the bytes, not packets.  So, any changes in the data size requires sequence number update.  It not only requires sequence number update in the affected packet, but also further packets going on the session.  All new packets would undergo the sequence number update. Similarly, ACK sequence number of reverse packets also should be updated while sending the packets to the monitoring devices.

When DR feature is combined with the 'Replication' feature,  this delta sequence number update can be different for different replicated packets.  Delta sequence number update feature is required to ensure that monitoring devices find the packet consistency with respect to sequence numbers and the data.

Some TMES vendors call this as part of DPI feature.


Inline TAP functionality:



Many places in the network might not have L2 switch to get hold of traffic from SPAN ports. If the traffic needs to be monitored from those points, then one choice is to place L2 switch and pass the traffic to the monitoring devices through the SPAN ports.  If the capture points are high, then there is a need for placing multiple L2 switches.  Inline TAP functionality is expected to be there in the TMES.  Two ports of TMES are required to TAP the traffic from these capture points.  These two ports will act as L2 switch while replicating traffic for monitoring devices.  Baiscally, TMES is expected to act as L2 swtich for these capture points. Since there are many capture points, TMES essentially become Multi-switch device with each logical switch having two ports.

Time Stamping of Packets

Analysis of traffic that was recorded acorss multiple monitoring devices would be a requirement in general.  It means that the recording devices should have same clock reference so that analysis engine knows the order in which the packets were received.  Yet times, it is not practical to assume that the monitoring devices will have expensive clock synchronization mechanisms.  Since TMES is becoming a central location to get hold of traffic and redirecting them to monitoring devices, TMES is expected to add time stamp in each packet that is being sent to the monitoring devices. 

IP protocol provides an option called 'Internet TimeStamp'.  This option expects TMES to fill its IP address and timestamp in milliseconds with midnight UT.  

Stripping of L2 and Tunnel headers


Many monitoring devices don't understand complicated L2 headers such as MPLS, PPPoE and tunnel headers such as PPTP (GRE),  GTP (in case of wireless core networks),  L2TP-Data,  IP-in-IP,  IPv6 in IPv4 (Toredo,  6-to-4, 6-in-4) and many more.  Monitoring devices are primarily interested in inner packets. TMESs are expected to provide stripping functionality and provide basic IP packets to the monitoring devices.  Since monitoring devices expect to see some known L2 header,  TMESes typically are expected to strip off tunnel headers and complicated L2 headers and keep Ethernet header intact.  If Ethernet header is not present,  TMESes are expected to add dummy Ethernet header to satisfy the monitoring device reception.


Traffic Conditioning

Monitroing devices are normally rated for certain amount of Mbps. Yet times, there could be bursts in the traffic, even though overall average traffic rate is within the device rating.  To avoid any packet drop due  to brusts,  TMES is expected to condition the traffic going to monitoring device.

Players :

I came across few vendors who are providing solutions meeting most of abvoe requirements.

Gigamon :  http://www.gigamon.com/
Anue Systems:  http://www.anuesystems.com/
NetOptics:  http://www.netoptics.com


I believe this market is yet to mature and there is lot of good upside potential.

There is good need for monitoring devices and hence the need for TMES will only go up in coming years.

User space Packet processing applications - Execution Engine differences with processors

Please read this post to understand Execution Engine.

Many processors with descriptor based IO devices have their own interrupts.  For each device, there is corresponding UIO device.  Hence software poll based EE provides 'file descriptor' based interface to register, deregister and get hold of indication through callbacks.  EE applications are expected to read the packets from the the hardware by themselves and do rest of the processing.

As discussed in UIO related posts,  we have discussed ways to share the interrupts across devices. As long as UIO related application kernel driver knows the type of event for which interrupt is generated, appropriate UIO FD is woken up and  things will work fine.

Non-descriptor based IO is becoming quite common in recent Multicore processors.  Hardware events (packets from the Ethernet controllers,  acceleration results from the acceleration engines) are given to the software through set of HW interfaces.  Selection of  HW interface by the hardware is based on some load balancing algorithms or based on some software inputs.  But the point is that, the events which are being given to the software through one HW interface are from multiple hardware IO sources.  Each HW interface is normally associated with one interrupt.  One might say that this can be treated as interrupt being shared across multiple devices.  But, some of  the Multicore processors don't have facility to know the reason for HW interrupt.  Nor they have facility to know the event type of first pending event in HW interface.  Unless the event is dequeued from the HW interface, it is impossible to know the type of event.  Also, due to interrupt coalescing requirements, a given interrupt instance might represent multiple events of different IO source devices.  Due to this behavior,  there may be only one  UIO device for multiple IO devices.  Hence responsibility of demultiplexing these events to right EE application falls on the EE itself.  EE needs to read the event and find out the right application and call the appropriate callback function registered to it. Let us call this functionality in EE as 'EE Event DeMux'.

In Descriptor based systems,  EE applications are expected to read the HW events (packets & acceleration results) by each EE application.  Callback function invocation only provides indication for EE application to read the events from associated hardware descriptors.  In case of 'EE Event DeMux',  the event is already read by the EE itself.  Hence, event is expected to be passed to the callback function.

'EE Event DeMux' submodule registers itself with the rest of EE module to get UIO indication in case of software poll method.  In case of  hardware poll,  'EE Event DeMux' in invoked by the  hardware poll function.

Multicore processors normally provides HW interface for multiple IO devices for the devices which are part of the Multicore processors.  External devices such as PCI and other HW bus based IO devices are still implemented using descriptor based mechanism.  Software poll based EE should not assume that all devices are satisfied using 'EE Event DeMux'.   As far as core Software poll system is concerned,  'EE Event DeMux' is another EE application.  Hardware Poll based method, if they need to use descriptor based HW interfaces, then the hardware poll should also poll descriptor based HW interfaces.

When 'EE Event DeMux' is used by EE applications (such as Ethernet Driver,  Accelerator drivers,), it is necessary that 'EE Event DeMux' considers following requirements.

• It should have its own 'Quota' as number of maximum events it is going to read from the HW interface as part of the callback function invocation by the EE core. Once it reads the 'Quota' number of events or if there are no more events, then it should return back to the 'Core EE' module.  
• Since this the module which demuxes to some EE applications, it should provide its own register/De-register functions.
• When 'Core EE' module invokes this module callback function due to interrupt generation or due to hardware poll,  as described above, it is expected to read at the most 'quota' number of events. While giving the control back to the  'Core EE', it is expected to call EE applications that there are no more events in this iteration. Some EE applications might register to get this indication.  For example, Ethernet driver application might register for this to do the 'Generic Receive Offload' function.  GRO functionality requires to know when to give up while doing TCP coalescing functionality.  In case descriptor based drivers, this issue does not arise as each Ethernet driver as part of callback invocation by the EE itself reads the events and knows when to give up.

Thanks for reading my earlier post.  I hope this helps.

User space Packet processing applications - Execution Engine

If you plan to port your data plane network processing application from Linux kernel space to user space,  first thing you would think is how you can port your software to user space with minimal changes to your software.  Execution Engine is the first thing one would think of.

Many kernel based networking applications don't create their own threads.  They work with the threads which are already present in the kernel.   For example,  packet processing applications such as Firewall, IDS/IPS, Ipsec VPN work in the context of Kernel TCP/IP stack.  This is mainly done for performance reasons.  Additional threads for these applications result in multiple context switches.  Also, it results into pipeline processing as packets handover from one execution context to another execution context.  Pipelining requires queues involving enqueue, dequeue operations which take some core cycles.  Also, it results into flow control issues when one thread processing is more than other threads. 

Essentially,  Linux kernel itself provides execution contexts and networking packet processing applications work within these contexts.  Linux TCP/IP stack itself works in softirq context.  SoftIRQ processing in normal kernel runs from both IRQ context as well as softirqd context.  I would say 90% of the time SoftIRQ processing happens in the IRQ context.  In PREEMPT_RT patched kernel, network IRQs are mapped to the IRQ threads.  In any case,  the context at which network packet processing applications run is unknown to the applications.  Since Linux kernel execution contexts are per core basis,  there are  less shared data structure and hence less locking requirement.  Kernel and underlying hardware also provides mechanism to balance the traffic across different execution contexts with flow granularity. In cases where hardware does not provide any load balancing functionality,  IRQs are dedicated to different execution contexts.  If there are 4 Ethernet devices and 2 cores (hence two execution contexts),  four receive interrupts of Ethernet controllers are assigned equally to two execution contexts.. If the traffic from all four Ethernet devices is same or similar, then both the cores are used effectively.  

Execution Engine in user space packet processing applications, if made similar to the Kernel execution contexts, then application porting becomes simpler.  Execution Engine (EE) can be considered part of infrastructure to enable DP processing in user space.  EE design should consider following.

• There could be multiple data plane processing applications in user space.  Each DP daemon may be assigned to run in fixed set of cores - core mask may be provided at startup time.
• If DP daemon is not associated with any core mask, then it should assume that the daemon may be run by all cores. That is, it should considered as if core mask contains all core bits set.
• Set of cores are dedicated to the daemon. That is, those cores don't do anything else other than doing DP processing of the DP daemon.  This facility is typically used to ensure in Multicore processors providing hardware poll.  Recent generation of Multicore processors have facility to provide incoming events and acceleration results through single portal (or station or work group).  Since the core is dedicated, there is no software polling is required.  That is hardware polling can be used if underlying hardware supports it and if the core(s) are dedicated to the process.

It appears that number threads in the process equaling the number of cores assigned to the process provides the best performance. Also, this also provides great similarity with kernel execution contexts. With the above background,  I believe EE needs to have following capabilities:

• Provide capability to assign the core mask. 
• Provide capability to indicate whether the cores are dedicated or assigned.
• If no core mask is provided, it should have capability to read the number of cores in the system and should assume that all the cores are given in core-mask.
• Provide capability to use software poll or hardware poll.  Hardware poll should be validated and accepted only if underlying hardware supports it and only if the cores are dedicated to it. Hardware polling has performance advantages as it does not require interrupt generation and interrupt processing. But the disadvantage is that the core is not used for anything else.  One should weigh the options based on the application performance requirements. 
• API it exposes for its applications should be same whether the execution engine uses software poll (such as epoll()) or hardware poll.

Typically capabilities are provided through command line parameters or via some configuration file.  EE is expected to create as many threads as number of cores in the core mask.  Each thread should provide following functionality:

• Software timers functionality - EE should provide following functionality.
• Creation and deletion of timer block
• Starting, stopping, restarting timers in each timer block.
• Each application can create one ore more timer blocks and use large number of timers in each timer block.
• As in Kernel,  it is required that EE provides cascaded timer wheels for each timer block.
• Facility for applications to register/De register for events and passing the events.
• API function (EEGetPollType()) to return the type of poll - Software or Hardware : This function would be used by EE applications to use file descriptors such as UIO and other application oriented kernel drivers for software poll or use hardware facilities for hardware poll 
• Register Event Receiver :  EE applications can use this function to register the FD, READ/Write/Error, associated callback function pointer and callback argument.
• Deregister Event receiver:  EE applications can call this to de register the event receiver which was registered using 'Register' function.  
• Variations above API will need to be provided by EE if it is configured with hardware poll. Since each hardware device has its own way of representing this,  there may as many number of API sets. Some part of each EE application have hardware specific initialization code and calls the right set of API functions.
• Note that one thread handles multiple devices (multiple file descriptors in case of software poll). Every time epoll() comes out,  callback functions of ready FDs would need be called. These functions which are provided by the EE applications are expected to get hold of packets in case of Ethernet controllers, acceleration results in case of  acceleration devices or other kinds of events from different kinds of devices. From the UIO discussion,  if the applications use UIO based interrupts to wakeup the thread, then it is expected that all events are read from the device to reduce the number of wakeups (UIO coalescing capability).  Some EE application might be reading lot of events. For each event it reads, it is going to call its own application function.  These applications can be very heavy too. Due to this,  if there are multiple FDs are ready,  one EE application may take very long time before it returns back to the EE.  This results into unfair assignment of EE thread to FDs which are also ready.  This unfairness might even result into packet drops or increase the jitter if high priority traffic is pending to be read in other devices.  To ensure the fairness,   it is expected that EE applications process only 'quota' number of events at any time before returning back to the EE.  'Quota' is tunable parameter and can be different for different types of devices.  EE is expected to callback the same application after it runs through all other ready file descriptors.  Until all ready EE applications indicate that they don't have anything to process, EE should not be calling ePoll().  To allow EE to know whether to call the application callbacks again,  there should be some protocol.  Each EE application can indicate to the EE while returning from the callback function on whether it processed all events.  EE based on this indication will decide to call the EE application again or not before it goes to the epoll() again.  Note that epoll() is expensive call and hence it is better if all events are processed in fairness fashion before epoll() is called again.   In case of hardware poll based configuration,  this kind of facility is not required as polling is not expensive. Also Multicore SoCs implementing the single portal for all events have fairness capabilities built in.   Since the callback function definition is same for both software and hardware poll based systems,  these parameters exist, but they are not used by hardware poll based systems.

EE before creating threads should initialize itself and then create threads. Once created it should load the shared libraries of its applications one by one.  For each EE application library, it is expected to call 'init' function by getting hold of address of 'init()' symbol.  Init() function is expected to initialize its own module.  Each EE packet processing threads is expected to call another function of EE application. Let us call this symbol name is 'EEAppContextInit()'.   EEAppContextInit function expected to real initialization such as opening UIO and other character device drivers and registering with the software poll() system.

EE also would need to call 'EEAppFinish()' function when the EE is killed.  EEAppFinish does whatever graceful shutdown required for its module.

Each thread, if it is software based poll, does epoll on all the FDs registered so far.  Polling happens in the while() loop.  epoll() can take the timeout argument. Timeout argument must be next lowest timer expiry timeout of all software timer blocks.  When epoll() returns, it should call the software timer library for any timer expiry processing. In case of hardware poll,  specific hardware specific poll function would need to be used.

In addition to above functions,  EE typically needs to emulate other capabilities provided by Linux Kernel for its applications such as - Memory Pool library,  Packet descriptor buffer library,  Mutual exclusion facilities using Futexes and user space RCU  etc.. 

With these above capabilities, EE can jump start the application development. This kind of functionality only requires changes at very few places in the applications.

Hope it helps.