L4 Switching & IPSec Requirements & Openflow extensions

 

L4 Switching - Introduction

L4 switching typically involves connection tracking,  NAT and common attack checks.  Stateful inspection firewalls,  NAT, URL filtering  and SLB (Server Load Balancing) are some of the middle-box functions that take advantage of L4 switching.  These middle-box functions inspect first few packets of every connection (TCP/UDP/ICMP etc..) and offload the connection processing to some fast path entity to do L4 switching.  Normally both fastpath and normal path functionality reside in the same box/device.

Openflow, promoted by industry big names,  is one of the protocols that separates control plane from data plane where control planes of multiple switches implemented in one logical central controller and leaving the data plane alone at devices, which are programmable to work in a specified way,  thereby making devices simple.

Middle-box functions described above can also be separated - Normal path (service plane) and fast path (L4 switchiing).  Normal path works on first packet or at the most first few packets and fast path works on the connection for rest of the packets in the connection. By implementing normal path (Service plane) in centralized logical controllers and leaving the fastpath (L4 switching) at the physical/logical device level,  similar benefits of CP/DP separation can be achieved.  Benefits include

• Programmable devices where device personality can be changed by controller applications.  Now,  Openflow switch can be made as L2 switch, L3 switch and/or L4 switch.
• By removing major software logic (in this case, it is service plane) from device to the central location, cost of ownership goes down for end customers.
• By centralizing the service plane,  operation efficiency can be improved significantly
• Ease-of software upgrades
• Configure/Manage from a central location.
• Granular flow control.
• Visibility across all devices.
• Comprehensive traffic visualization

L4 Switching - Use cases:

• Branch office connectivity with corporate head quarters :   Instead of having firewall, URL filtering and Policy control on every branch office device,  it can be centralized at corporate office. First few packets of every connection could go to main office.  Main office controller decides on the connection. If the connection is allowed,  it lets the openflow switch in the branch office to forward rest of the traffic on that connection.  This method only requires simple openflow switches in the branch office and all intelligence can be centralized at one location, that is, at main office. Thus, it reduces the need for skilled administrator at every branch office.
• Server Load Balancing in Data Centers:   Today data-centers have expensive server load balancers to distribute the incoming connections to multiple servers.  SLB devices are big boxes today for two reasons -  Normal path processing that selects the best server for every new connection and  fastpath packet processing are combined into one box/device.   By offloading packet processing to inexpensive openflow switches,  SLB devices only need to worry about normal path processing, which could be done in lesser expensive boxes and even using commodity PC hardware.  
• Managed Service providers providing URL filtering capability to home & SME offices :  Many homes today use PC based software to do URL filtering  to provide safe internet experience to kids.  SME administrators require URL filtering service to increase the productivity of their employees by preventing them to access recreational sites and also to prevent any malware contamination.  Again, instead of deploying URL filtering service at customer office premises,   service providers like to host this service centrally and manage centrally across multiple of their customers for operational efficiency.  Openflow controller implementing URL filtering service can get hold of packets until URL is fetched, find the category of URL, apply policy and decide on whether to continue the connection.  If the connection is to  be continued, then it can program the openflow switch in customer premises to forward rest of the packets.  
• Though URL filtering service is one use case given,  this concept of centralizing the intelligence at one central place and programming the openflow switches for fast path is equally applicable for "Session Border Controller" and "Application Detection using DPI" and other services. Whichever service needs to inspect first few packets of the connection is candidate for Service Plane/L4 Switching  separation using Openflow.

L4 Switching - Processing

L4 switching or fastpath entity are used to do connection level processing. Any connection consists of two flows -  Client-to-Server (C-S) flow and Server-to-Client (S-C) flow.  

Connections are typically 5-tuple based.  

Functionality of L4 switching:

• Connection Management :  
• Acts on the connection Add/Remove/Query/Modify requests from the service plane.
• Inactivity timeout :  Activity is observed on the connection. If no activity for some time (programmed during the connection entry creation),   connection entry is removed and notification is sent to the normal path.
• TCP Termination : L4 switching entity can remove the connection entry if it observes the both endpoints of TCP connections exchange FINs and ACKed. It can also remove the entry if TCP packet with RST is seen.
• Periodic notifications of the state collected to Service plane.
•  Packet processing logic involves
• Pre-flow lookup actions typically performed:
• Packet integrity checks :  Checksum verification of both IP and transport headers,  Ensuring that the length field values are consistent with packet sizes etc..
• IP Reassembly:  As the flows are identified by 5-tuples, it is required that packets are reassembled to determine the flow across fragments. 
• IP Reassembly related attack checks and remediation
• Ping-of-Death attack checks (IP fragment Overrun) - Checking for full IP packet never exceeds 64K.
• Normalization of the IP fragments to remove overlaps to protect from teardrop related vulnerabilities.
• Too many overlap fragments - Remove the IP reassembly context when too many overlaps observed.
• Too many IP fragments  & Very small IP fragment size:  Drop the reassembly context and associated IP fragments.
• Incomplete IP fragments resulting Denial of Service attacks - Limit the number of reassembly contexts on per IP address pair,  IP address etc..
• TCP Syn flood protection using TCP Syn Cookie mechanism
• Flow & Connection match:  Find the flow entry and corresponding connection entry.
• Post lookup actions
• Network Address Translation :  Translation of SIP, DIP, SP, DP.
• Sequence number translation in TCP connections :  This is normally needed when the service plane updates the TCP payload that decreases or increases the size or when the TCP syn cookie processing is applied on the TCP connection.
• Delta Sequence number updates in TCP connections :  This is normally required to ensure that right sequence number translation is applied, especially for retransmitted TCP packets.
• Sequence number attack checks :  Ensuring that sequence numbers of the packets going in one direction of the connection are within some particular range. This is to ensure that any attacker injecting the traffic is not honored.  This check is mainly required to stop TCP RST packets that are generated and sent by attackers.  As RST packet terminates the connection and thereby creating DoS attack, it is required that this check is done.
• Forwarding action :  To send the packet out by referring to routing tables and ARP tables.

IPSec - Introduction

IPSec is used to secure the IP packets. IP packets are encrypted to avoid leakage of data on the wire and also authenticated to ensure that the packets are indeed sent by the trusted party.  It runs in two modes - Tunnel mode and transport mode.   IP packets are encapsulated in another IP packet in tunnel mode.  No additional IP header in involved in transport mode. IPSec applies the security on per packet basis - Hence it is datagram oriented protocol.  Multiple encryption and authentication algorithms can be used to secure the packets.   Encryption keys and authentication keys are established on per tunnel basis by the Internet Key Exchange Protocol (IKE).   Essentially,  IPSec protocol suite contains IKE protocol and IPSec-PP (Packet processing).  

Traditionally,  implementations use proprietary mechanism between IKE and IPSec-PP as both of them sit in the same box.  New way of networking (SDN) centralizes control plane with distributed data paths.  IPSec is one good candidate for this.  When the control plane (IKE) is separated from the data path(IPSec-PP),   then there is a need for some standardization for communication between IKE and IPSec-PP.  Openflow is thought to be one of the protocols to separate out control plane and data plane.   But Openflow as defined in OF 1.3.x specification did not keep IPSec CP-DP separation in mind.  This blog post tries to provide extensions required in OF specification to enable IPsec CP-DP separation.

Openflow

ONF (Open Networking Foundataion) is standardizing Openflow protocol.  Controllers using OF protocol programs the OF switches with flows and instructions/actions to be performed on packets.  Openflow 1.3+ specification supports multiple tables which simplifies the controller programming as each table in the switch can be used for a purpose.

Contrary to what everybody says,  Openflow protocol is not really friendly let alone to SP/FP separation, but also to L3 CP/DP separation.  The way I look at OF today is that it is good for L2 CP/DP separation and also good for traffic steering kind of use cases, but it is not sufficient for L3 and L4 switching.  This post tries to give some of the features that are required in OF specifications to support L4 switching in particular and L3 switching to some extent.

Extensions Required to realize L4 switching & IPSec CP-DP separation

Generic extensions:

Table selection on packet-out message:

Problem Description
Current OF 1.3 specification does not give provision for controllers to start the processing of packet-out message from a specific table in the switch.  Currently,  controller can send the packet-out message with set of actions.    These actions are expected to be executed in the switch.  Typically 'OUTPUT' action is expected to be specified by the controller in the action list.  This action sends the packets out on the port given in "OUTPUT" action.  One facility is provided to run through OF pipeline though.  If the port in 'OUTPUT' action is specified to be a reserved port OFPP_TABLE,  then the packet-out message execution starts from table 0.

Normally controllers takes advantage of multiple tables in the OF switch by dedicating tables to purposes.  For example, when L4 switching  with L3 forwarding are used, multiple tables are used - A table to classify packets,  A table for Traffic Policing entries,  A table for PBR and few tables for routing tables,  a table for L4 connections and a table for next hop entries and a table for traffic shaping rules.   Some tables are populated pro-actively by the OF controller , hence miss packets are not expected by controllers.  Entries in some tables are populated re-actively.  Tables such as 'L4 connections',  'Next hop' entries are created reactively by the controllers.   Processing of the miss packets (packet-in) can result into a flow being created in the table which was the cause of miss packet. Then controllers likes to send back the packet to the OF switch and would like OF switch to start processing the packet as OF switch does when there is a matching entry. Controller programming does not get complicated  if that was possible.  As indicated before,  due to limitations of the specification,  controller can't ask switch to start from a specific table.   Due to lack of this feature in the switches,   controllers are forced to figure out the actions that would need to be performed and then program the action list in the packet-out message.  In our view,  that is quite complex task for the controller programmers.  As I understand,  many controller applications simply using packet-in message to create the fow, but drop the packet-in message.  This is not good at all.  Think of SYN packet getting lost.  It would take some time for client to retransmit the TCP SYN packet and that delay results into very bad user experience.

Also, in few cases,  applications (CP protocols such as OSPF, BGP)  generate packets that needs to be sent out on the data network.  Controllers typically sit in management network, not on the data network.  Hence these packets are to be sent to the data network via OF switches.  This can only be achieved by sending these packets as packet-out messages to the switch.  Yet times, these packets need to be exposed to only part of the table pipeline.  In these cases, it would be required to have ability for controllers to start the packet processing in the switch for these packets from a specific table given by controller.

Solution

Freescale Extension :

struct ofp_action_fsl_experimenter_goto_table {
      uint16_t type; /* OFPAT_EXPERIMENTER. */
      uint16_t len; /* Length is a multiple of 8. */
      uint32_t experimenter; /** FSL Experimenter ID **/
      uint16_t fsl_action_type; /** OFPAT_FSL_GOTO_TABLE **/
      uint16_t table_id;
}
OFP_ASSERT(sizeof(struct ofp_action_fsl_experimenter_goto_table == 8);

table_id :  When switch encounters this action,  switch starts processing the packet from the table specified by 'table_id'.

Though this action is normally present in the action_list of e packet-out messages, this action can be used in flow_mod actions too.  When there is GOTO_TABLE instruction and GOTO_TABLE action,  then the GOTO_TABLE action takes precedence and GOTO_TABLE instruction is ignored.

Notice that 'table_id' size is 2 bytes.  This was intentional.  We believe that OF specifications in future would need to support more than 256 tables.

Network Services deployments in Data Centers

Application Delivery controllers are already part of the many data centers where there are multiple servers to serve the content to end users. ADCs balance the incoming connections to multiple servers for high availability and also to ensure that load is shared.

Let us see where they fit in the data center architectures as listed in here. 

You might have heard the term 'Network Services Layer'.  This particular area contains network security devices, ADCs and WAN optimization devices.  Many traditional data centers don't have this layer, but newer data centers have this layer.  This layer works in conjunction with Core switching layer. 

Core switches are expected to be configured to pass the selective traffic to network services layer.  There is no change expected in access switch layer or SAN switch layer.  One can have security at these layers too for additional security,  but I guess it will be some more time before data center administrators add any additional security at access level.

Before getting into the details of capabilities of core switches, devices in network switch layer,  let us first visit the addressing of typical data centers:

Many time private IP addresses are used for servers.  Though public IP address can be assigned to server(s) if there is only one for a domain name or in case of DNS load balancing, it is my observation that many times only private IP addresses are assigned.  There are several reasons:

•  If there is a need in future to expand to multiple servers due to load or high availability considerations,  no changes would be necessary except for some configuration in ADCs and bring up new server (or virtual machine).
• Many times, servers need to communicate with other internal servers such as database servers, SANs, application servers etc.. .   Private network gives comfort of security as well as reduce the need for number of public IP addresses.
• For a given domain name,  multiple services may need to be exposed.  Also, for each service, different physical server or different virtual machine may be required.  Private IP addressing with ADCs translating to right private IP address on incoming connections help in facilitating this. If public IP address is used, then all services need to be put in same physical server or virtual machine.

Typically private centers have one domain name and multiple services. Public data centers have multiple domain names and multiple services in each domain.  Each domain name + service name combination may contain multiple servers serving the content/users.    You might have heard term called server farm.  A given server farm is identified by  IP address (resolved IP of domain name ) and Service (Ex: Port 80, 25 etc..). Server farm is configured with IP addresses of real servers serving the content.  ADCs do this job of selecting the least loaded server dynamically upon receiving client connection and awards the connection to selected real server.  In recent times, real servers are subdivided into multiple subsets and each server farm provides additional configuration (rules) to select the subset.  Then least loaded server is selected from this subset.  One example deployment may put  all videos, images and static content  in some servers (set1) and other servers serve the dynamic content (set2).   ADCs can be configured with rules to select set1 for URL containing some file extensions and set2 for everything else. I hope server farm concept is clear.

It is important to understand how ADCs are deployed first:

Small data centers don't have two tier architecture of switches. Jut a simple ADC is good enough. This acts as switch as well as Load balancer.  ADCs have L2 switch and one or more Ethernet MACs.  Ethernet MACs are connected to the network that gets connected to the WAN links and servers get connected to the L2 switch. Basically, in simple deployments ADC acts as access layer switch on server side and Core router/switch on the core network side.  Let us call this as 'simple data center'.

In complex data centers or in public data centers, two tier architectures are required. In this case, core switches are configured to pass the traffic coming from core network to the ADCs and traffic from server network to ADCs for load balancing purposes.  Traffic from server network might go to ADCs without any special configuration in core switches if the client to server packets were translated with SNAT. Due to this SNAT,  server to client traffic will have DIP as the ADC ip address and hence packets will go to ADCs without any special configuration in the core switches. Data centers are complex because it needs to handle large amount of traffic and/or has large number of server farms.  Due to this, some times one ADC may not be enough to take the load of all server farms.  Multiple ADC devices are used in those cases with each ADC handling traffic belonging to few server farms.  In these cases, core switches are expected to provide facilities to segment the traffic on server farm basis and redirect the traffic to ADCs for balancing the traffic across servers within server farm.  To give an example:  If a data center has 100 server farms and 10 ADCs, then the core switch should have capability of segmenting the traffic 100 ways and passing 10 sets of traffic to appropriate ADCs.   This is typically achieved via VLANs.  

As discussed above each server farm is identified on incoming traffic (from core network) by public IP address (resolved IP of domain name) and the port.  If there are 100 server farms, then 100 VLAN IDs are required.  Core switches can be configured to generate VLAN ID based on the incoming traffic.  Core switches have this capability called 'rules'.  Rule can be created with selectors (in this case destination IP and destination port) and action as 'redirect' with VLAN ID and Port on which to transmit.  Switches, when the traffic matches with this kind of rule, adds VLAN ID to the packet and transmits the packets on the port indicated in the rule.

Let us take an example of public data center.  Let us assume that there are two domain names - www.example1.com (P1 public IP address) and www.example2.com (P2 public IP address)   www.example1.com has two services - Port 80 and Port 25.   www.example2.com has one one service - Port 80.   Example1 company wanted four servers - P11, P12, P13, P14 to serve Port 80 content and 1 server for Port 25.  Example2 has two servers P21 and P22 to server port 80 content.  It was decided to use ADC1 and ADC2 with ADC1 handling two server farms of www.example1.com and ADC2 handling www.example2.com server farm.   Let us also assume that ADCs are deployed in two-arm mode.  On one arm it expects client-to-traffic to land and another arm it expects server to client traffic land.  Let us assume that VLAN11 and VLAN12 are for client and server traffic of www.example1.com. And VLAN21 and VLAN22 are for client and server traffic of www.example2.com.  Let us also assume that two physical ports PR11 & PR12 and PR21 & PR22 are used to connect to ADC1 and ADC2 respectively. 

In above deployment, core switch cluster is expected to be configured with following rules:

• On ports that are towards the Internet:
• Source IP : ANY  Destination IP: P1  Destination Port : 80 or 25 --------> Add VLAN11 and send on PR11.
• Source IP : ANY Destination IP: P2  Destination Port : ANY --------> Add VLAN21 and send on PR21.
• On ports that are toward the Servers
• Source IP:  P11 Subnet,  Destination IP: ANY, Source Port 80, 25 ----> Add VLAN12 and send on PR12.
• Source IP:  P21 Subnet,  Destination IP: ANY -------> Add VLAN22 and send on PR22.

ADCs/Server load balancing devices would need to have capability to instantiate VLAN interfaces based on VLAN ID.

Network Security Devices and WAN Optimization devices also can be deployed in similar way.  It is good if all functions are available in the same device. This would only require one set of VLAN configuration in the switch. If  these devices are independent to each other, then switch need to configure multiple VLANs to pass traffic from one device to another via switch.  If routing is allowed, then the one device can have route to another device and switch will pass the traffic from one system to another as any switch do. 

In above example, ADC1 is handling two server farms. But it is also possible to divide these two servers across two different ADCs even though they belong to same domain. As long as service is different, it should be possible.  It is also possible to use one ADC for multiple domain names too.  In this case, ADCs or network security devices provide 'virtual instance' to ensure that traffic of two different domain names are independent and isolated. I am too lazy to type in the configuration required on switches for these two configurations, but I think you got the picture :-)

Data Center Switch requirements for new Data Center Architectures

Traditionally data centers have three tiers of switches -  Core switches,  Aggregate switches and Access switches.

• Core Switches :  These switches connect to the network which are connected to the WAN links. This is farthest switch farm with respect to servers.
• Access Switches :  These switches are also called top-of-rack switches.  Servers (Web Servers, Email Servers, Application Servers, Database Servers and others for which data center is built) get connected to the ports of these switches. 
• Aggregation Switches:  Aggregation switches is intermediate switch layer which is sandwiched between Core and Access switch layers.  Aggregation switch aggregates the traffic between core and access layers.  Note that there could be lot of traffic among servers (Specifically among application, web and database servers). This traffic need not be seen by the core switches.  This traffic just need to be among the access layer switches.  Aggregation layer eliminates the traffic being seen by every switch.  Core switches only see the traffic going to/coming from WAN/Corporate network.  Aggregation layer also reduces the traffic among access layer switches.

It was necessary to have three tiers in earlier data center architectures due to

•  Large number of physical machines serving the content requires large number of Ethernet ports.  Due to poor density of the ports on the switches, multiple access layer switches were necessary. Multiple switches means there is lot more traffic across access layer switches. One more hierarchy of switches enable good throughput by eliminating mesh kind of access layer switches for intra switch traffic.

What are some of the changes in Data Centers? One big change is collapse of three tiers to two tiers. Aggregation layer is disappearing.  Let us see what is making this change.

• Virtualizaton technology is reducing the number of physical machines:  This implies that there are less number of ports.
• Traffic on each port is increasing : Virtualization and Mulitcore processor are enabling multiple applications in one physical machine.  It is not uncommon to see the requirement of multi-gig traffic on a single port.
• 10G and in future 40G/100G ports are facilitating the unified fabric for both kinds of traffic - Application traffic and SAN traffic, thus eliminating number of ports and interconnects.

These technologies are reducing the cost by reducing equipment, interconnects, by amount of power required and amount of cooling required. It also reduces the maintenance and hence reduction in cost.

What kind of features one would expect in the switches in new data centers:

• Latency of traffic should be very less:  By eliminating the aggregation layer itself reduces the latency. But that is not good enough for SAN traffic, Video and Voice workloads.  Non-blocking switching or cut-through switching is expected to support real time traffic such as Video, Voice etc..   Traditionally, switches oversubscribe the bandwidth, that is, switches are not capable of receiving and transmitting of traffic of all ports at the same time with full port bandwidth. Hence the packets get blocked.  In non-blocking switches,  they are expected to send and receive traffic equal to number ports * each port bandwidth.  If there are ten 1G ports, switches are expected to receive 10G traffic and send 10G traffic.  
•  802.1qbb (Priority based Flow Control):  When there is a congestion in the receiving node, 802.3x pause frame is generated normally. This makes all the traffic pause for some time. This standard allows pause frame generation on 802.1p priority levels. It lets the high priority traffic flow. Switches are expected to honor and generate theses kinds of frames.
• 802.1qaz (Enhanced Traffic Selection):  This standard allows the bandwidth allocation for different priority levels or group of priority levels.  It lets higher priority bandwidth to be consumed lower priority traffic if there is no higher priority traffic.  SAN traffic would need to be going with higher priority levels. This feature is also expected to be supported by data center switches.
• 802.1qau (Congestion Notification):  This standard allows end nodes to communicate the congestion notification.  It lets the end node receiving the congestion notification to apply rate limiting on the out traffic.  This feature is also expected to be supported by data center switches.
• Port Density should be high.
• Multi-Path support is required - I am not sure whether there are any standard at this time, but spanning tree is not used in these cases as it only provides one path. 
• VEPA Support would be required eventually. Due to VEPA,  it may need to support C-VLAN and P-VLANs.
• Large number of VLANs support is required to work with other network services such as ADCs, WAN Optimization and Network Security (Firewall, IPS, IPSec VPN etc..). 
• Ability to redirect the traffic not only based on L2 and L3 fields, but also L4 fields such as TCP, UDP Source and destination ports. 
• Any switch architecture should work with VM migration from one physical server to new physical server.
• Public Data Center networks require Virtual Instance kind of concept within the switches to reuse VLANs (across different subscribers) due to limited number of VLAN IDs.  

IPv6 deprecates NAT that is popular in IPv4 world - Is this really true?

I am not sure though.  There is no doubt that there are large number of IP addresses available in IPv6 world.  Though NAT was started due to the IPv4 address resource exhaustion, it has been used for many other purposes such as

•  Avoid renumbering of the majority of network in corporations when they change their service provider or when the companies get merged.  When the service provider is changed or companies get merged, only few devices such as routers,  firewalls & NAT devices , IPSec VPN devices and Server such as DNS need to be updated.  Rest of the network addressing can remain same as NAT devices front end the network to external world.
• When corporate networks have multiple ISP connections,  two scenarios are possible - There is load balancing of connections across multiple ISP connections and failover of connections to other ISP network in case of failure of primary ISP connection.  In both cases,  in IPv4 world, NAT devices take care of balancing and routing the traffic to appropriate ISP connections, that is, end hosts are not aware of this. End hosts are always assigned with fixed private IP addresses and NAT device takes care of translating those addresses with IP address of the ISP connection on which the packets are going to be transferred to.  In case of IPv6 network, without NAT support, each end host should take care of this on their own. This may be possible when every host runs routing protocol, but not sure whether it is practical.  By doing NAT in IPv6 world, this would be similar to IPv4 world and the burden of balancing and failing over the connection resides with NAT devices.
• Security by obscurity is one another thing IPv4 NAT is providing which will be lost in IPv6 if every end host communicates with the external world on its own.

To me, above problems can be solved in some fashion or other such as Provider Independent IP addresses. This might increase the burden on the Internet core routers as there would be large number of routes. Though at this time, I am not sure whether this is practical, but I am sure some solution would be found.

But main reason I would believe NAT would continue to exist is due to Server Load balancing devices (And ADC devices).   SLB devices are expected to find out the least loaded server and award the connection to that server.  It is expected that connections are destined to one IP address (This IP address is called Virtual IP address) and server identifies the least loaded real server and redirects the connection to that server. Redirection of the connection typically involves Destination NAT where the DIP of the original connection packets are replaced with the real server IP address and vice versa on SIP (Source IP) for packets going from real server to client.  Also to ensure that the packet from the server passes through the same SLB device, source NAT is also applied with 'source IP' replaced with IP address of SLB device.  Since 'source IP' seen by real server is same even though connections came from multiple clients, there is a possibility of two different clients using the same source port. Hence source port translation is also required at the SLB device.  Due to this, I believe all considerations which are applicable for IPv4 NAT are valid here in Ipv6 world too.

One can say that there are other alternatives to address translation such as DNS based load balancing where different DNS requests are given with different real server IP addresses. But this is not accurate and there is a possibility of some real servers getting loaded more than others.  Another solution that can be suggested is to do only MAC translation.  Though this works, but it works only if both SLB device and real servers are on the same link.
Due to limitations on different approaches, Load balancing with NAT would continue to be required in many deployments.  So, don't rule out NAT even when we all move to IPv6.

If you believe I am wrong here, please drop a comment.

Linux TCP Large Receive Offload optimization to increase performanace

In some network packet processing applications, number of packets being processed determine the performance. TCP is a streaming protocol and hence there is no packet boundary. Hence consecutive packets can be aggregated into few packets when the TCP packets are received at the lowest level.  More the packets that can be aggregated, higher the performance would be.  Applications that can benefit are:

• Any Proxy based applications (Application Delivery controller, WAN optimization,  Network Anti Virus)
• IDS/IPS 
• Firewall ALGs.
• Server Applications

I found one excellent paper describing two techniques to improve the TCP connection throughput performance -  Receive aggregation  and  Acknowledgment offload.  Please find it here. This paper also gives performance improvement with receive aggregation and without these optimization techniques.  Performance was improved from 3.4Gbps to 4.6Gbps, 35% increase.

Receive aggregation technique is already implemented in Linux 2.6 kernel. It is called Large Receive offload feature.  This feature is implemented in net/ipv4/inet_lro.c.

Receive aggregation technique is simple. It is used only when the NAPI functionality is applied on the Ethernet driver.  In NAPI enabled Ethernet drivers,  softirq receives the packets from the descriptors. Typically NAPI reads out all the packets from the receive descriptors (or until some threshold - quota). 

• Ethernet Driver normally sends up the packet to the stack using netif_receive_skb if the NAPI is enabled. In case of LRO,  packet is given to the LRO library using lro_receive_skb function.
• Find the matching flow. If no match, it creates new flow.
• LRO module figures out whether this packet is eligible for aggregation. Packet is non-eligible if any of following conditions apply.
• Non Padded frame (IP total packet length must be received packet length)
• Non-TCP packet.
• IP options are present
• IP ECN CE is set
• TCP segments has no data.
• CWR (Congestion Window Reduced) flag is set
• ECE (ECN Echo) flag is set
• SYN flag is set
• FIN flag is set
• URG flag is set
• PUSH flag is set
• RST flag is set
• ACK flag is not set
• Non TCP Timestamp option is present
• If the next sequence number expected is matches with the sequence number of this packet, packet is added to the existing packet sequence.  If not, packet is not eligible for aggregation.
• When the packet is found to be not eligible for aggregation, it is necessary to send buffered packets first to the stack before sending the current packet. This is done using lro_flush() function.
• If the packet is eligible for aggregation, it associated with existing packets by manipulating the skb.
• When the aggregation stops, it does following before sending the aggregated packet to the stack.
• Changes the ACK to the last packet ACK.
• Keeps the timestamp option of the last packet.
• Recalculates the IP checksum (now the packet became bigger).
• Partial Checksum update of TCP payload.
• When does the aggregation stop:
• When the configured aggregation limit reaches.
• When the total packet size is more than (64K-MTU).
• When the NAPI finishes all the packets in receive descriptors or reaches its quota.
• Ethernet Driver is expected to send all the packets so far buffered at the end of current NAPI instance. It does so by calling lro_flush_all.

ecryptfs - some thoughts on network device appliance usage

There are many cryptography file systems in Linux. But 'ecryptfs' is made into the kernel.org. Almost all distributions support ecryptfs today. Due to this, my belief is that many appliance vendors are using this file system to store the files encrypted.

Network appliances such as WAN optimization devices requiring storage of confidential information in secure form can use this file system to store files in encrypted form.

There are multiple utilities provided to configure file system with different security parameters. It supports x.509 certificates (RSA) for encrypting and decrypting the encryption key for each file that is being written in this file system. This file system is really a wrapper file system on top of existing file systems such as EXT2 and EXT3. You can have multiple of ecryptfs file systems on one existing file system. Create a directory and make it as 'ecryptfs'. Any file that is written on to this directory is encrypted. You can get ecryptfs utilities here.

It appears from the source code, each file that is being written into the crypto file system can have its own encryption key. It makes use of keyring facility provided by Linux Kernel to store keying material.

How can crypto file systems be used by appliances. Appliances unlike typical users are not attended all the time. They are expected to be up all the time. They are expected to start themselves in case of any failures.

'ecryptfs' when it gets started is expected to be provided with the key information such as RSA private key. To ensure that this private key is not visible for any thief assuming that laptop is stolen, this private key is normally encrypted with passphrase. This passphrase is a secret which is expected to be entered by the user when laptop is powered up. User providing the passphrase works fine for laptop cases. But this is not the case for appliances which are not monitored. If passphrase is not used, then RSA private key is in clear. If the appliance is stolen, then the private key can be read, therefore, the complete file system. I guess basic purpose of encrypted file system is lost. So, it is expected that RSA private key is not stored in the same appliance device. I suggest to have RSA key pair on external (but protected physically) machine and mounted on local machine. Since RSA key pair is not on the appliance, even if it stolen, information is secure.

Usage of 'ecryptfs' can be found here: http://maketecheasier.com/create-a-private-encrypted-folder-on-ubuntu-hardy-with-ecryptfs/2008/09/25

Secure Storage of cached and de-duplication data

WAN optimization devices store the data (content) in their own hard drives to process future requests. Some of this data can be confidential. More places the data is stored in clear, there are more opportunities of this data leaking out. In WAN optimization device case, if device is stolen, thieves can retrieve confidential data easily. To protect the privacy of data, it is necessary that the devices store the content in encrypted form. One thing one must ensure is that the DRE (Data Redundancy Elimination) efficiency should not go down even when encryption is applied. As you know Encryption algorithms similar to compression algorithms create dependency across the data in the file. That is some portion of previously encrypted block data is used to encrypt further data in the file. This will break the de-duplication efficiency dramatically. That is every time file gets modified by the user application, the file content change exponentially due to encryption even though the changes were made to the clear file were small.

'rsyncrypto' file encrypts the file such a way that there is no dependency among the encrypted blocks. Typically IV (Initialization Vector) is taken from the previous block and used along with the key to encrypt the new block. 'rsyncrypto' eliminates the IV being taken from the previous block and uses random IV for all blocks in the file. Though this may reduce some security effectiveness, it provides enough security effectiveness.

Backup market certainly can use this feature to secure data at rest while maintaining the de-duplication efficiency. This feature is particularly useful when external backup storage providers are used to backup the data. It is required that the users have control over keys used to encrypt the files and at no time backup storage providers have access to these keys at any time including while applying delta changes. This requirement mandates that the delta data is obtained on old and new encrypted files. So, 'rsyncrypto' utility is really useful. When the data needs to be retrieved from backup storage providers, user known keys would be used to decrypt the files.

I am not sure whether this technique is applicable for WAN de-duplication markets. WAN optimization devices need to serve the content locally without downloading the data from central WAN optimization device. That is, all WAN optimization devices should be able to get hold of clear data. Hence, I feel that WAN optimization devices would use 'Crypto file systems'. These file systems transparently encrypts all files in the file system. No knowledge of this is needed by WAN optimization feature in the device. This kind of secure storage appears to be fine as these WAN devices are typically administrated by same entity serving confidential data.

Validity of performace metrics?

Many security appliance vendors and ADC vendors provide performance metrics with respect to throughput and connection rate. But if you look deeply, none of these metrics are valid for actual deployments. At most, they are useful for developers to identify the bottlenecks in the software.

If you look at closely, throughput numbers are given with UDP traffic directed to some arbitrary port. Connection rate is given with respect to number of TCP connection establishments (3 way handshake) and termination by TCP RST packet. None of these are real deployment scenarios. How much of traffic is really UDP in real world? How many connections are really TCP with just connection establishment? How often the connections are terminated by TCP RST packets. So, the numbers being given by device vendors are useful for themselves to figure out the device or software bottlenecks.  But these are not sufficient to make end users to relate to their networks and traffic.

Many security devices and ADCs have application protocol capabilities.

• Firewall devices have ALGs (Application Level gateways) for each complex protocol to open holes in the firewall for dynamic data connections. Examples: FTP, SIP, MGCP, H.323, RTSP, DNS, L2TP, PPTP etc..
  
• IPS devices have application intelligent protocols. Similar examples as firewall.
• Anti Virus functionality typically implements proxies to collect the data for scanning. Example proxies: HTTP, HTTPS, FTP, SMTP, PoP3, IMAP etc..
• URL content filtering feature not only interprets HTTP protocol data, but also does quite a bit of content matching (regular expression search).
  
• ADC devices also contain application protocols such as HTTP, HTTPS, FTP, SIP, RTSP etc..

It is fair to expect the throughput and connection rate for all application protocols which are supported by the device.

Connection rate should not only involve TCP connection establishment, but also should have the data that is normally interpreted by the device. In addition, the size of data is reasonable. It is true that the size of data can be different in different deployments, but using some reasonable size for data during connection establishment is required.

Throughput numbers are also expected to be given with respect to application protocols. Whatever size of the data is chosen should be clearly documented. My suggestion to device vendors to give throughput numbers for different sizes of protocol data. Another aspect  is number of connections used in throughput measurements. I observed that device vendors use very less number of connections with large amount of data to get the best numbers. They do give good numbers as most of the connection state information resides in the cache. Any medium sized deployment would have more than few thousand of connections going through simultaneously.  So,  it would be good if device vendors give throughput numbers with large number of sessions.

So, don't go by marketing numbers. Ask for more performance data with respect to applications, number of flows etc.. You would be surprised to see the difference in results. Really..

Application Delivery Controllers versus Server Load balancing devices

Server Load Balancing (SLB):

Let us first revisit the "Server Load balancing" concept and its features. Server Load balancing was invented during initial stages of web in late 1990 and early 2000s. At that time the pages served were mostly static. So simple Layer4 load balancing was good enough to do balancing of TCP connections across multiple servers in the server farm. Load balancing device, typically, is assigned with Public IP address. When client connection lands on this public IP address, this device figures out the best server that can serve the connection and hands over the connection to the server by doing "Destination NAT", that is, destination IP address of remote client generated packets gets translated with IP address of server selected. To ensure that packets going to client from server are accepted by the client, response packets from the server would undergo source IP address translation. By this clients only see the public IP address in response packets. Now, let us discuss some SLB features.

• MAC Level redirection : When servers in the server farm have public IP address, this option can be used instead of doing "Destination NAT". Server Load balancing (SLB) device should be placed in the traffic path to ensure this mode works. In this mode, SLB device selects the server in the farm and modifies the destination MAC address with the server MAC address. Optionally it can also change the Source MAC with the SLB device MAC.
  
• Additional Source NAT : This mode is useful in assymetric routing topologies. When destination NAT is done on the client packets, it is required to do source IP translation on server to client traffic. For this to happen, server to client traffic must pass through the same SLB device that did the DNAT on the client traffic. Source NAT is done on the client traffic with the SLB device IP address to ensure that the traffic come to the SLB device even if the servers' routing tables are configured with some other default gateway.
  
• Multiple different algorithms to select the server: Based on SLB deployment and need, several different ways of scheduling connections to server can be chosen. Following are some of the methods.
  

• Round Robin : Each server is treated same from capability perspective and servers are chosen on round robin fashion. If there are 2 servers in the server farm, then alternate connections are given to each server.
  
• Weighted Round Robin : Typically each server is given a weight 1 to 10, 10 is assigned to server that can serve 10 connections for every 1 connection served by server that has weight 1. Until weight number of consecutive connections are assigned to a server, next server is not chosen.
  
• Least Connections : In both Round robin and Weigthted round robin methods, it is assumed that the all connections are almost equal with respect to duration of connection, amount of traffic that is sent on each connection and processing power used by each connection on the server. But in reality it is not true. In 'Least Connection' method new connections is given to server which is serving least number of connections at that time. SLB device maintains the count of active connections with each server to facilitate this scheduling.
  
• Weighted Least Connections : All servers can't be assumed to be equally powerful and this feature allows deploying servers of different processing power capabilities. Each server can be given a weight. If 2 and 5 weights are given to server 1 and server2, it means that server2 is powerful enough to serve 5/2 times the active connections simultaneously than server1. SLB device keeps this weight in mind in deciding the best server for new connections.
  
• Fastest Response time : Though 'Least connections' and 'Weighted Least connections' do their best to distribute the connections, there can be situations where many high processing connections are given to one server. Any new connections going that server can overwhelm that server and due to that connections may not be served well. Any connection that does deep data mining function could overwhelm the server. To take care of these situations, it is better to monitor the processing power left in each server for new connections. What is the better way than monitoring the response time of servers. Though it is not 100% way of finding the least utilized server with response time, it is close as long as monitoring for response time is happening continuously. Some SLB devices monitor the response time as part of health checking. That is not good enough as health checks happen once in a while. Response time measurements must be done on response to real client traffic.
  
• Fill-and-go-next : Green is the buzz word now - Save power and save on cooling costs. Round Robin and other methods above distribute the traffic to all servers. Even when there is less load across servers, they are all used and thereby not giving chance to any server to go to any power saving modes. When this mode is chosen, SLB device uses one server to full utilization before selecting next server. Typical configuration include the number of connections a server can handle and when to start warming the next server. For example if 10000 and 8000 numbers are given, after 8000 active connections on existing server, next connection is given to next server to bring that server up to the full power and both servers are used until 18000 active connections and then another server is warmed up and so on. This mode can be combined with Round Robin method where this is used to balance the traffic across all warmed up servers.
  

• Syn Flood Detection and Prevention (DDoS Prevention): This feature is common in servers from last few years. But detection and prevention at central place in SLB allows administrators not worry about feature availability in all servers or enabling this feature in servers. This feature prevents attackers from sending large number of SYN packets without completing the TCP connection establishment phase. SLB devices using this feature detect the SYN flood attack without consuming its resources using SYN-Cookie mechanism. Only when three way handshake is complete, connection is awarded to server in the server farm.
• Traffic Anomaly detection and Traffic Policing: DDoS attacks such as Syn flood detection is really thing of a past. Newer kind of attacks complete TCP connections and issue requests that consume CPU power there by creating Denial of Service for genuine users. Since these attacks complete the connection, it is possible to track the users by their IP addresses. Traffic anomaly detection feature allows administrators to configure the baseline traffic characteristics based on normal day activity. SLB devices based on this configuration detect any anomaly and can take corrective actions such as:

Throttling the traffic coming from identified IP addresses : Throttling the traffic can be on "Simultaneous active connections", "Connection Rate", "Packet/Byte Rate".


Blocking the traffic from identified IP addresses for certain time.


• Health Checking: SLBs are expected to use servers that are online. To ensure that, SLBs typically do health checks on servers to ensure that they are alive. Some types of the health checks that can be configured are:

ICMP Ping : Using this check SLB knows that the server machine is active.


TCP Connection Checks : These checks allow SLBs know that a given TCP Service is running. Note that it is possible that Machine is alive, but the user process listing for TCP connections is dead. This check allows SLBs to check the TCP Server liveness.


Application Health Checks: These checks allow SLB to know whether the application is running well on the TCP Service. Note that TCP Server may be running in the server, but application that acts on the application data could be different user process than the process that is listening for TCP connections. Hence it is important to have these health checks enabled in the SLB devices.


• Persistence: For some end applications, it is necessary that all connections coming from one client go to the same server. This is necessary if the application maintains some state across connections coming from the client. It necessitates temporary and dynamic binding state in SLB devices. When there is no matching bind state, one server is chosen for the connection and immediately binds the client IP address with the chosen server IP address. Any new connection from this client is given to same server without server selection process. To ensure that these bindings don't consume large amount of memory on SLB devices, these bindings are removed after certain amount of inactivity. As you can see later, binding are implemented using cookies in case of HTTP connections without having need to store binding state in the SLB. Cookie received in each request would indicate the bound server.

Application Delivery Controllers (ADC):

Server Load balancing based on L3-L4 ( Source IP, Destination IP and Ports) does not solve all issues with different kinds of applications running on the servers. It requires L7 intelligence in the SLB devices to meet the load balancing requirements of newer kind of applications. Since they have L7 intelligence in making load balancing decision, I guess these devices are being called 'Application Delivery Controllers". In short they are called ADCs.

First generation of ADCs were trying to solve challenges associated with E-commerce and Session-persisting HTTP based applications. One challenge is to ensure all connections go the server which was chosen for the first connection from the client. This is necessary due to the fact the applications on the server maintain user state in the memory and this state is updated with the user selections until user logs out. This challenge has another twist where these connections are SSL encrypted.

First generation ADCs solve these challenges using HTTP protocol intelligence in ADC devices.

• Persistence using HTTP Cookies : Though IP based persistence works decently, it fails when a given user IP address changes across connections. This can happen if the user router gets new IP address in the life of user session. I believe another possibility is due to different NAT addresses usage by some ISPs. That is, ISP might use one NAT IP address for one connections and another NAT IP address for other connections of same user. Due to these issues IP address persistence does not work in all cases . In addition, IP address persistence occupies some memory in SLB devices. Alternative for HTTP based applications is to use HTTP Cookies. HTTP Cookies allow servers to relate the connections belonging to the same session. Same mechanism is used by ADCs to relate user connections. ADCs define their own cookie for storing the selected server in encrypted/encoded cookie data. When new HTTP connection is received, it checks whether it has its cookie and value. If not, it assumes that this is first connection of session and adds a Cookie in the response going to client with encoded/encrypted selected server information. Any further connections coming from the same user to this site would have this new cookie and this cookie value is used by SLB to map the connection to the server. Since this cookie belongs to the SLB, SLB devices remove the cookie from the request before sending it to the server.
• SSL Termination : E-Commerce web sites require SSL/TLS Security and also session persistence. To facilitate this, SLBs provide SSL termination capability. This capability in addition also offloads CPU intensive SSL processing to ADCs and thereby saving CPU cycles on server for application processing.
  

Second generation ADCs extend above functionality with HTTP Optimization, L7 protocol data based server selection, threat security features and multiple virtual instances.

HTTP Optimization features:

• HTTP Connection pooling : HTTP protocol allows multiple transactions in one TCP connection. This feature in ADCs multiplex multiple client connections into few connections to the server. Without this feature, there are as many TCP connections to the Server from SLB as number of TCP connections it is terminating. With this feature, there are only few TCP connections towards server, there by saving some CPU cycles and memory on the servers. Though the amount of saving one can have with this feature is debatable, this is one feature many ADCs support.
  
• HTTP Compression: HTTP compression using gzip and other algorithms can save bandwidth by compressing the response data. ADC devices offload this capability from the Servers there by making server CPU cycles available for application processing.
  
• HTTP Caching & Delta Encoding: HTTP Caching feature between clients and servers avoids duplicate file download if content was not changed on the server. But there are many instances where file/data content change but not significantly. This feature allows only difference (delta) being sent to the clients rather than complete content there by saving bandwidth usage. Since delta encoding requires significant CPU cycles, ADCs offload this functionality from HTTP Servers.
  

Programmatic L7 Protocol content based Server Selection:

SLBs traditionally don't look at the application protocol data to make server selection. ADCs augment the criteria for selecting the server with protocol data such as HTTP data. Some examples where protocol intelligence is required:

• Multiple server farms with each each farm having different content.
  
• Different server farms for Mobile users and Normal users.
• Different response on errors based on browser type.
• Prioritization/throttling/shaping of requests to specific URL resources upon congestion.
• Addition/Modification of request/responses based on the type of applications running on the server farm.
• Combination of above.

Since each deployment has its own requirements, one configuration solution for all is not sufficient. Hence ADCs provide flexible mechanism using 'rules' similar to IPS devices which provide rules for detecting vulnerabilities. Rules are created by administrators using the application protocol (eg. HTTP) keywords exposed by ADCs. Typically each protocol element is exposed as keyword. Multiple rules can be created with each rule having keywords and associated values. ADCs internally match the connection data with the rules. First match of rule with respect to values of keywords is used to take actions as per the rule. Actions can be 'Selection of Server farm', traffic policing etc..

Threat Security:

Since ADCs are becoming a central location for all requests going to the data center servers, this is place to enforce threat security and thereby offloading threat security functionality from the servers. Some of the threat security functions ADCs support are:

• Traffic Policing based on protocol intelligence: ADCs with protocol intelligence can do better job of DDOS prevention than SLBs. Baseline traffic can be much beyond typical 5 tuples. It can include URL pattern, User-Agent, Cookie values and beyond.
  
• Web Application Firewall: There are many attacks that are being exploited by attackers on web server applications including SQL injections, XSS (Cross Site Scripting), LFI (Local File include) and RFI (Remote File Include). ADCs are increasingly offloading this functionality from the servers. It not only improves server performance, but also administrators can configure signatures at one place i.e ADC.

Virtual Instances:

This feature is not common in ADCs today. But I believe this is going to become critical feature moving forward to ensure that multiple server farms belonging to multiple domains corresponding to different customers are supported by one or few ADCs. One ADC device should be able to support multiple customer server farms in public data centers. There are two types of virtual instances possible - In first type. One executable image supports multiple instances. Here configuration data and run time data are stored separatetly and even provide UI/CLI with role based access to configuration belonging to virtual instances. Second type of Virtual instance is to have multiple images with each image for one virtual instance. In this case, even if there is any issue with one virtual instance, other instances are not affected, there by providing good isolation. First type can provide large number of instances where second type is limited to few tens. I personally think that Linux container approach is better for second type as this is lean and uses common Operating system and TCP/IP stack image. Only user processes are instantiated multiple times as number of containers.

Third generation ADCs:

What are the features one would expect from third generation ADCs. I can think of few based on where data center market is heading.

• Cloud Computing : Traditionally ADCs are configured with all servers in a given server farm. It works well when manual intervention is required to setup/take-off servers from server farm. Administrators are used to configure the ADCs accordingly. In Cloud computing, servers are no longer physical. They are virtual. They can be added and removed dynamically and programmatically. They also can be disabled and enabled dynamically without manual intervention. Some use cases - virtual instances are brought up and down based on the load on the servers or based on time of day, holidays etc.. Since they are coming up and going down dynamically, it is expected that ADCs also would get this information dynamically.
• Beyond HTTP Servers (SIP, CIFS, SMTP, IMAP etc..): HTTP had been and is dominant in data centers today. It is changing with cloud computing. Many organizations are expected to host their internal servers too in the cloud. They include mail servers, File Servers etc.. Third generation ADCs are expected to balance the traffic across Email Servers, File Servers etc.. In addition it is expected to have optimization features to save network bandwidth. File Servers today dominantly support CIFS. ADCs are expected to have CIFS proxy that can save bandwidth by doing delta encoding, caching and de-duplication etc..

RSYNC - Applicability for WAN Deduplication

The very old and good rsync can be key to WAN deduplication.

WAN Deduplication purpose is to reduce the amount of traffic on the WAN links by removing duplicate date. It is also called as DRE (Data Redundancy Elimination).

'rsync' is utility provided in Unix variants for a long time. It is mainly used to mirror the data across multiple servers and also is used to for backing up the data. It recursively goes through all files and directories and updates the data in the backup or mirror server. One feature that is interesting to WAN deduplication is its ability to do 'delta encoding'. 'rsync' has feature to send only 'differences' to the destination machine which inturn creates a new file from the existing file and the 'delta' information it gets from origin server. 'rsync' also can compress the data using 'zlib' and thereby saving even more WAN bandwidth.

To understand the algorithms used by 'rsync' for delta encoding, check this technical report. it uses mechanism called 'rolling checksum' (Alder32 algorithm) to figure out the differences between the file the mirror machine has and the file the origin sever has. Why is this 'rolling checksum' required? Note that when file gets updated by anybody in the origin server, the changes could be anywhere in the file. It can be in the beginning of the file, middle of the file or at the end of the file. 'Delta' generation should work and not duplicate any common data irrespective of placement of changes the original file has undergone. Mirror server breaks down the file it has into multiple chunks of some size (typically S = 1K), calculates both rolling checksum and MD5 checksum on the chunks. Then it sends them to the origin server. Origin Server does rolling checksum for chunks of size S. Since the file might have undergone changes, origin server creates rolling checksum for each byte offset. Note that mirror server generates rolling checksum for non-overlapping chunks. Since origin server generates large number checksums, rolling checksum algorithm needs to be very fast. Alder32 algorithm has property of generating checksum without going through the all bytes of the chunk and hence it is very fast. It can generate checksum incrementally. Then the received checksums are compared with the checksums generated locally by origin server. If any checksum matches, then it verifies by comparing with MD5 checksum. If MD5 checksum also matches, then origin server assumes that mirror server has this chunk. Once it finds out the all duplicate data, it only sends the matching block information and any new or modified data for non-matching chunks. For detailed information on this algorithm, please check this link.

Other utilities which are useful for WAN deduplication : 'rdiff'. 'rdiff' uses the rsync algorithm to generate delta file with the difference from old file and new file. Then this can be applied to old file at other machine to get the new file.

Hardware timer block in Multicore processors for network infrastructure devices

Some use case scenarios of timers for different functions of network infrastructure devices is given here.  

One of the main challenges with software timers is to ensure that jitter and latency of the packets don't go up during the period when some timer block related operations occur.  Latency of the packets or even packet drop happens when CPU takes too long a time to process some timer block related functions.  Any timer block functions that go through the timers in a tight loop would have affect on packet processing if the number of timer elements checked or acted on in the tight loop are more.  The threshold of number of elements that are checked in the tight loop that causes packet latency disruption depends on the frequency of CPU. Based on the software timer block implementation,  traversal of some timers happen for different operations.  Let us see some of the challenges/problems with software timer modules.

• Software timers depend on hardware timer interrupt. In Linux, timer interrupt occurs fore very jiffy( typically 1msec or 2msec).  Due to this any software timer can  have error up to jiffy.  If applications requires  smaller error, say in terms of, micro seconds,  then only method I can think of is to have timer interrupt to occur in terms of microseconds.  This may not work in all processors.  There is too much of interrupt processing overhead in cores and reduces the performance of the system. Fortunately many Applications tolerate millisecond error in firing the timers, but some applications such as QoS scheduling on multi-gig links running general purpose operating systems such as Linux require finer granular and accurate timers.
• Many networking applications require large number of software timers as described in earlier post.  This will lead to traversing many timers on per jiffy basis.  For example, if an application creates 500K timers/sec, then there would be 500 timers on per jiffy basis. For every 1 millisecond,  it needs to traverse 500 timers and may have to fire all 500 of them. This can take significant amount of time based on the amount of time the application timer callback takes. If takes good amount of time to process, you have packet drop or increased packet latency or both the issues. Some software implementations maintain the timers on per core basis. If there are 8 cores,  each core may be processing 62 or 63 timers every millisecond.  This is ideal case, but what if the traffic workload is causing only few cores starting the timers. Only few cores would be loaded to process the expired timers.  Basically the load may not get balanced across the cores.
• To reduce the number of timers to traverse for every hardware timer interrupt,  cascaded timers wheels are normally used by software implementations. This implementation does have different timer wheels for different timer granularity and when the timers are started, they go to appropriate wheel and bucket. Due to this any bucket of timer wheel contains the timers that will get expired. Though it reduces the number of timers to traverse for every timer interrupt, but it may involve movement of large number of timers from one timer wheel to another as described in the earlier post. This movement of timers may take significant amount of time and again could be the cause for packet drop and increased latency.
•  If there are periodic timers or need to be restarted based on activity software timer implementation spend good amount of time in restaring them.

Do hardware timer blocks in Multi-core processors help?  

In my view hardware timer block can help when your applications demand large number of timers, periodic timers or very accurate timers.  If your application requires 'Zero Loss Throughput', then hardware block is going to help certainly as it takes away the CPU cycles used to traverse the timer list or movement of timers in software implementations.

What are the features expected by network infrastructure applications from hardware timer block in Multi-core processors?

• Large number of timers are expected to be supported, ranging in Millions. 
• Decent number of (say 1K) timer groups are expected to be supported.  There are multiple applications running in cores that require timers.  Applications that are being shutdown or that are being terminated due to some error conditions should be able to clear all the timers that it had started.
• Accessibility of timer groups by applications running in different execution contexts. There should be good isolation among timer groups. There should be some provision to program the number of timers that can be added to a timer group.  There should be provision to read the number of timers that are in the timer group.
• Applications running in Linux user space 
• Applications running in Kernel space.
• Applications running in virtual machines. 
• Application should be able to do following operations. All operations are expected to be completed synchronously.
• Start a new timer:  Application should be able to provide
• Timer identification : Timer Group & Unique timer identification within in the group.
• Timeout value (Microsecond granularity)
• One shot or periodic timer or inactivity timer
• Priority of timeout event (upon expiry) : This would help in prioritizing the timer events with respect to other events such as packets.
• If there are multiple ways or queues to provide the timer event upon expiry to the cores,  then application should be able to give its choice of way/queue as part of starting the timer. This would help in steering the timer event to specific core or distribute the timer events across cores.
• Stop existing timer :  Stopping the timer should free the timer as soon as possible.  One existing hardware implementation of timer block in Multi-core processor today has this problem. If the application is starting timers and stopping them in continuous fashion, eventually it runs out of memory and memory will get freed only upon actual timeout value of the timers. If the timeout of these timers are in tens of minutes,  then the memory is not released for minutes together.  Good hardware implementation of timer block should not have this exponent usage of memory in any situation.   Timer stop attributes typically involve
• Timer identification
• Restart the existing timer.
• Timer identification
• New timeout value
• Get hold of remaining time out value at any time, synchronously by giving 'Timer identification'
• Set the actvity on the timer - Should be very fast as applications might use this on per packet basis.

Firewall/NAT/ADC appliances targeting Large and Data center markets would greatly benefit from the Hardware based timer blocks.  All hardware timer blocks are not equally created. Hence check the functionality and efficacy of hardware implementation. 

• Measure the latency, packet drop and jitter of the packets over long time. One scenario that can be tested is given below.
• Without timers,  measure the throughput of 1M sessions by pumping traffic across all sessions using equipment such as IXIA or smartbits. Let us this throughput is B1. 
• Create 1M sessions, hence 1M timers with 10 minutes timeout value.
• Pump the traffic from IXIA or smartbits for 30 minutes.
• Check whether the throughput is almost same as B1 across all 30 minutes.  Also ensure that there is no packet drop or increase in latency of packets, specifically at 10, 20, 30 minute interval.
• Measure the memory usage:
• Do connection rate test with each connection inactivity timeout value 10 minutes.
• Ensure that upon TCP Reset or TCP FIN sequence the session is removed and hence timer is stopped.
• Continue this for 10 or more minutes.
• Ensure that the memory usage did not go up beyond reason. 
• Ensure that timers could be started successfully during the test.