IP Fragmentation versus TCP segmentation

Ethernet Controllers are increasingly becoming more intelligent with every generation of NICs.  Intel and Broadcom have added many features in Ethernet NIC chips in recent past.  Multicore SoC vendors are adding large number of features into Ethernet IO hardware blocks.

TCP GRO (Generic Receive Offload - It used to be called Large Receive offload too) and GSO  (Generic Segmentation Offload and it is used to be called Transport Segmentation Offload)  are two new features (in addition to FCoE offloads) one can see from Intel NICs and many Multicore SoCs.  These two features are  good for any TCP termination applications on the host processors/cores.  These two features reduces the number of packets traversing the host TCP/IP stack. 

TCP GRO works across multiple TCP flows where it aggregates multiple consecutive TCP segments (based on TCP sequence number) of a flow into one or few TCP packets in the hardware itself, there by sending very few packets to the host processor.  Due to this,  TCP/IP stack sees  fewer inbound packets.  Since the packet overhead is significant in TCP/IP stacks, lesser packets uses lesser number of CPU cycles, thereby leaving more CPU cycles for applications, essentially increasing the performance of overall system.

TCP GSO intention is similar to TCP GRO,but for outbound packets.  TCP layer typically segments the packets based on  MSS value. The MSS value is typically determined from PMTU (Path MTU) value.  Since TCP and IP headers take 40 bytes of data,  MSS is typically ( PMTU -  40 ) bytes.  If PMTU is 1500 bytes, then the result MSS value is 1460. When the application tries to send large amount of data,  then the data is segmented into multiple TCP packets where each TCP payload carries up to 1460 bytes.  TCP GSO feature in the hardware eliminates the need for TCP layer to do the segmentation and thereby reduces the number of packets that traverse between TCP layer and to the hardware NIC.  TCP GSO feature in the hardware typically expect the MSS value along with the packet and it does everything necessary internally to segment and send the segments out.

Ethernet Controllers are increasingly providing support for IP level fragmentation and reassembly.  Main reason is being  increasing popularity of tunnels.

With increasing usage of tunnels (IPsec, GRE, IP-in-IP,  Mobile IP, GTP-U and futuristic VXLAN and LISP), the packet size is going up.  Though these tunnel protocol specifications provides guidelines to avoid fragmentation using DF bit and PMTU discovery,  it does not happen in reality.  There are very few deployments where DF (Don't Fragment bit) , which is required for PMTU discovery, is used.   As far as I know,  almost all IPv4 deployments fragment the packets during tunneling.  Some deployments configure network devices to do red-side fragmentation (fragmentation before tunneling so that the tunneled packets appear whole IP packet) and some deployments go for black-side fragmentation (fragmentation after tunneling is done).   On receive direction, reassembly happens either before detunneling or after detunneling. 

It used to be the case where fragmented packets are given lesser priority by service providers during network congestion.  With high throughput connectivity and increasing customer base for networks, service providers are competing for the business by providing very good reliability and high throughput connectivity. Due to popularity of tunnels,  service providers are also realizing that dropping fragmented packets may result in bad experience to their customers.  It appears that service providers are not treating the fragmented packets in a step-motherly fashion anymore.

IP fragmentation and TCP segmentation offload methods can be used to reduce the number of packets traversing the TCP/IP stack in the host.  Next question that comes to mind is how to tune the TCP/IP stack to use these features and how to divide the work  between these two HW features. 

First thing to tune in the TCP/IP stack is to remove the MSS dependency on PMTU.  As described above, today MSS is calculated based on PMTU value. Due to this, IP fragmentation is not used by TCP stack for outbound TCP traffic. 

TCP Segmentation adds the both TCP and IP header to each segment.  That is, for every 1460 bytes, there would be overhead of 20 bytes of IP header and 20 bytes of TCP header.  In case of IP fragmentation,  each fragment would have its own IP header (20 bytes of overhead).  Since TCP segmentation has more overheads,  one can say IP fragmentation is better.  Here, MSS can be set to a bigger value such as 16K and let IP layer fragment the packet if the MTU value is less than 16K.   This is certainly a good argument and it works fine in networks where the reliability is good.  Where the reliability is not good,  if one fragment gets dropped, TCP layer needs to send entire 16K bytes in retransmission.  If TCP had done the segmentation, it would only need to send fewer bytes. 

There are advantages and disadvantages with both approaches. 

With increased reliability of networks and with no special treatment on fragmented traffic by service providers,  IP fragmentation is not a bad thing to do.  And ofcourse, one should worry about retransmissions too. 

I hear few tunings based on the deployments.  Warehouse data center deployments where the TCP client and servers in a controlled environment are tuning MSS to 32K and more with 9K (jumbo frame) of MTU.  I think that , for 1500 bytes MTU,  going with 8K of MSS may work good.

Need for Pattern Matching Accelerators in UTM devices

Network security term typically refers to Threat prevention and Security on the wire.

Threat protection is normally achieved with multiple security technologies.  Basic protection is achieved from firewall technology.  IDS/IPS (Intrusion Detection/Prevention System),  Anti-Virus, Web application firewalls are some of the security technologies that are increasingly being used to protect networks (Network devices, Servers and Client machines).  Application Detection is another technology that is increasingly being used along with firewall to stop/allow traffic that can't be identified using ports in TCP/UDP headers, but requiring deep packet inspection.

Other than firewall,  all technologies listed above require deep packet and deep data inspection.  IDS/IPS technology adopts multiple techniques to identify the attack traffic. One of the techniques is to match the traffic data with known attack patterns.  Application detection also relays on pattern matching on the data as one of the techniques to detect the applications.  Anti Virus technology too depends on some pattern matching to detect viruses.

In almost all technologies above,  patterns get added to the deployed systems on continuous basis by device vendors as more attacks are discovered.  For example,  IPS devices, nowadays have around10,000 patterns (signatures) to detect the known attacks.  It keeps increasing every year.  Additionally, Some of these patterns are checked on every packet that goes through IPS.  This adds to number of CPU cycles requires to do IPS protection.

Many software algorithms are used to speed up the pattern matching performance.  Some of the techniques inlcude:

• DFA (Deterministic Finite State Automata)
• Bloom filters - Filters formed from the hashes of patterns can be used on the traffic to determine whether further analysis is required.
• PCRE algorithms to search for patterns of regular expression type.

IPS and other technologies also use techniques to reduce the number of patterns to be matched using protocol level intelligence and classifying the patterns in multiple buckets (protocol basis,  port basis,  even on the basis of application protocol stages such as URL based pattern database,  HTTP Request header,  Response header pattern databases etc..).

Due to above techniques,  some device vendors  think that there is no need for pattern matching hardware accelerators.  There is a reason for that too as some early developments of snort (popular open source IDS/IPS software) did not find much performance improvement with hardware accelerators.  But I believe HW accelerators are required for following reasons.

Performance Determinism:  IPS, Anti Virus,  Web application firewall and application detection technologies depend on the regular signature updates. Hardware deployed in the fields might have X number of signatures a day of purchase and they might go up to 2X or 3X over the years.  Performance determinism is expected by CSOs.  To maintain performance levels,  CPUs should be avoided in doing pattern matching.  Hardware accelerators specialized in pattern matching help in maintaining performance levels even with increasing number of signatures.

Protection from CPU hogging attacks:  With software based pattern matching, it is possible to hog the CPUs by crafting the packets with each data that matches a patterns multiple times.  Consider that there is a signature rule which tries to match a pattern "abc123def" and if there is 1Mbytes of data is sent with all the data having "abc123def" repeated,  then the CPU would take forever as it matches every packet multiple times.  CPU will not only spend time in matching the patterns, but also spends significant number of cycles in doing  further analysis.   Hardware accelerators normally designed such a way that the performance does not go down even if there are multiple matches.

Next question would be the what capabilities of hardware accelerators one should look for to mitigate performance issues  - One associated with explosive growth of attack patterns (signatures) and avoid CPU hogging by deliberate attempts by attackers.  I believe one should look for following capabilities.

• Accelerators should be programmable with decent number of patterns.
• Accelerators should be able to perform well even with large number of patterns.
• Accelerators should be able to perform well even if there are large number of matches.
• Accelerators should be able to perform pattern matches based on context information such as 'relative offset', 'Depth of the data to look' while doing pattern matching.  This will reduce the number of results being returned by the accelerator.  Smaller the number of results to software, lesser the post processing.
• Accelerators should be able to return results only when multiple patterns match on the data.  This also is required to reduce the number of results. 

In summary,  pattern matching hardware accelerators are required to reduce the CPU hogs either due to increase in signatures or intelligently crafted data by attackers. I feel that end customers should buy the UTM/IPS devices that take advantage of these accelerators to ensure that devices can be used at least for few years (future proof).

 

IGMP Filtering - Developer tips

IGMP protocol allows hosts to report their interest in Multicast address membership with adjacent routers.  These adjacent routers in turn propagate the consolidate membership with upstream routers using PIM-SM or using IGMP proxy functionality.

IGMP protocol sits right on top of IP layer, at the same level as ICMP, UDP and TCP.  IGMPv1 and IGMPv2 protocols are older protocols to IGMPv3.  IGMPv1 and IGMPv2 join the multicast membership by providing Multicast address in IGMP report message.  IGMPv3 takes one step further and even ask for membership with respect to source.  That is, there could be  multiple Multicast sources  sending same streams using one Multicast address. Hosts have choice of receiving the Multicast traffic only from some specific sources - It can specifically include the source addresses in the report message or can exclude some specific sources.

Typical firewalls today don't have capability to restrict the hosts on a specific interface from joining specific Multicast stream.  Firewalls today can allow or deny the IGMP packets on a specific interface though. But this is not sufficient.  It should be possible to restrict a given multicast stream on a specific interface.  This is possible only if firewall on the router deny only IGMP reports having restricted Multicast addresses.

IGMP filtering functionality in routers expected to provide following functions.  Some vendors call this 'IGMP filtering for Multicast authentication'.  I never understood why they use term 'Authentication' though.

• Provide facility for admin user on per interface basis 'Allow Only List' or 'Disallow list'.  Each list contains multiple records. 
• Multicast Address,  Source address.
• Source address can be 'ALL'.  

Some implementation considerations:

It is always good to implement this module separate from IGMP Router or IGMP Proxy modules.  In case of IGMPv1 and IGMPv2,  each membership report message only contains one Multicast address.  If this multicast address is to be restricted,  then the complete message can be dropped.  In case of IGMPv3,  one membership report message can contain multiple Group records with each group record having multicast address, source addresses and qualifier to all the source addresses listed in the group record - include or exclude.  IGMP filter module needs to do quite a bit of work to identify the group records and corresponding sources and remove only the affected ones from the message.  Then rest of the message should be allowed to pass through.  If the complete Multicast address is restricted as per configuration, then the complete Group record from the IGMP report message can be removed.  It is not as simple as though.  From the configuration, some times only specific source addresses are to be removed from the group record in the message. That is, IGMP filter is expected to selectively remove not only group records, but also source addresses selectively in the group records.

Clustering of devices with traffic distribution by L2 Switch - One limitation & Mitigation

In my last post on "Data Center/Enterprises Clustering of Devices"  I discussed on how L2 switches are enabling device equipment vendors to provide cluster solution to take up the increasing load on the networks.  Many L2 switches are capable of analyzing multiple different types of  layer 2 headers to get to the inner IP packet and use inner IP packet source and destination IP address fields to distribute the traffic across multiple devices in the cluster using hash distribution.  L2 switches typically understand Ethernet and MPLS related headers such as Ethernet DIX,  LLC/SNAP,  802.1Q VLAN headers and MPLS Label headers.  That is, L2 switch can get hold of IP packets if packets are sent over above mentioned L2 headers. 

In some deployments, IP packets may be encapsulated in multiple L2 headers or L3 tunnels.  Some examples are : PW (Pseudo wire) header,  Ethernet over Ethernet using PW,  GRE/UDP,  GTP/UDP, IPinIP,  Mobile IP and may more.  L2 switches in the market today are not capable of understanding these headers to to get to the inner IP header.   In these cases, distribution based on inner IP header fileds will not be possible.  In these deployments, L2 switches may need to resort to distribution based on L2 header fields such as source and destination MAC or tunnel IP header fields.  Unfortunately,  distribution based on these fields may not be good at all.  If you take an example of this cluster being places between two routers,  MAC addressees of every packet traversing the cluster will be same.  Hence any distribution based on the MAC addresses would go to only one device in the cluster.  Similar would be the case, if distribution depends on the tunnel IP header fields. 

Switches have one capability though.  They can generate the hash based on calculated CRC on some part of the packet or CRC of the Ethernet packet.   CRC of the Ethernet packet can be assumed to provide good distribution as CRC on Ethernet packet is based on the complete Frame payload, that is, including the inner IP packet payload. Switches have capability to take few bits of CRC to distribute the packets across multiple cluster devices.  But the issue with this is that packets belonging to same connection would go to different cluster devices.  Base on my earlier post, cluster devices assume that packets belonging to a connection would always land on the same device. This assumption is no longer true if the cluster solution is being deployed in above mentioned environments.  These types of environments are not common at all in Data Center and Enterprises environments.  So, this problem may not be there in many instances. But service provider environments, multiple L2 and tunnel header situations are not uncommon.

How do cluster solutions work in these environments?

In these environments, if CRC based distribution is used,  switches are really doing packet sprinkling across multiple devices.  In these cluster devices should have additional intelligence.  

• Cluster devices should be able to get past the headers to get to the inner IP header.
• Cluster devices among themselves should have understanding of session distribution. One simple method is to do what switches were doing.  That is, they can generate the hash on the IP header fields (source and destination IP) and figure out the device which got the packet is the one which needs to server based on the hash value.  If it is,  it should continue processing the packet. If it is not, then it should give the packet to the device that owns the hash value.  

There could be good amount of traffic among cluster devices.  They can use the switch as their back plane to send and receive the traffic among them.  To avoid other device doing same thing, that is getting hold of the inner IP header and hashing on the inner IP header fields again,  the sending device can send this information along with the packet and receiving device can avoid doing same operations again.

As indicated above,  only few deployments where L2 switch does not do inner IP header based distribution.  If same cluster solution being provided for all kinds of deployments, then it is good for network equipment vendors of cluster solution to provide configuration on the cluster whether the packets which are being distributed by L2 switch is packet sprinkler or intelligent IP flow based sprinkling.  If it is packet based sprinkling, then additional logic in devices can kick in to figure out the real destination device.

Data Center/Enterprises - Clustering of Network devices

Throughput requirements of Data center/Enterprise network equipment are going up with increased traffic in data centers and Enterprises.  In addition,  computational requirements of network equipment are also going up.  Some examples of why more computation power is required.

• Intrusion Detection/Prevention now requires almost 3 - 4 times the  computation power on per Mbps of traffic than what was required few years back.  I guess it is mainly due to sophisticated nature of attacks and evasion techniques adopted by attackers.  Javascript analysis itself takes 10 times computational power  than the typical DPI based pattern matching.  Javascript analysis requires proxy based functionality to get hold of the javascript and script analysis for attack detection.  These two tasks require lot more CPU cycles than typical pattern matching.
• Traditional Server Load Balancers (SLB) used to select the internal server based on the IP, UDP/TCP header values.  Next generation server load balancers (SLB) called ADCs do deep packet inspection, such as HTTP, SIP URL,  HTTP Request headers,  to select the internal server to send the load.  DPI requires more CPU cycles.
• Application Firewalls such as Web Application Firewalls and SIP firewalls not only  do the deep packet inspection, but also deep data inspection and that  requires more horse power from CPUs.
• DDOS prevention requires real time analysis of not only packet-by-packet analysis, but also sessions and application protocol level analysis across sessions to identify the attacks.  Many DDOS attacks on per session basis look exactly same as the normal traffic.  Hence the analysis across sessions is required for detecting the anomaly.  This capability requires not only lot of memory, but also good amount of computational power.

Multicore processors are helping some extent in solving performance issues.  Clustering of multiple Multicore SoCs are becoming necessary to solve above performance issues in Data Center and Large Enterprise markets.  Typically,  multiple blades, each using Multicore SoCs, running the same application are clustered to take up the load. L2 switches are increasingly used to front end the cluster.  L2 switches now can be configured to balance the load across multiple devices of cluster.  One might see the cluster and L2 switch in one enclosure giving a feeling that it is one big box providing tens of gigabits of performance.

What features of L2 switch are enabling clustering?

• Distribution of sessions across multiple devices in cluster:  Majority of L2 switches have capability to distribute the  traffic coming from incoming ports (Data Ports) across multiple ports (Device ports).  By connecting devices in the cluster to these ports, then each device gets the traffic that was redirected to that port. But many of the network devices expect that all packets of any given session go to the same device.  For example, all packets belonging to one HTTP connection should go to one device.  If packets of the sessions are distributed across multiple devices,  they will not be able to do their operation of analysis, proxy etc..    A given connection traffic involves both Client to Server and Server to Client traffic.  Though L2 switches don't have session intelligence, due to the hash based distribution mechanism they adopt,  same hash value gets generated for session traffic whether it is C-S or S-C traffic of a connection.   Some cautions:
• L2 switches don't do IP reassembly.  Due to this, hash generated for first fragment of a packet can be different from the non-initial fragments if the hash generation block is configured with L4 fields (TCP source and destination ports).  So, it is advisable to configure the hash block with IP addresses and IP protocol.   This may give rise to unequal distribution. But with large number of sessions in DC, this may not be a big limitation.  
• Some application sessions require multiple connections. Example:  SIP (Session Initiation Protocol).   SIP voice call typically involves three connections - SIP control connection,  RTP for voice/video data and RTCP for control frames.  Many devices expect that all three connections land on the same device.  If all three connections have same source, destination and protocol fields, then all packets of SIP application session would be sent to the same device by the switch.  But, RTP and RTCP IP addresses may be different from the IP addresses of SIP control connection.  If your device needs to support this,  then it is responsibility of cluster devices.  Cluster devices need to have intelligence of ownership of these kinds of  application sessions. If a device receives packets belonging to application that is owned by some other device, it needs to redirect the traffic to that  device that owns the SIP session.
• As indicated implicitly above,  L2 switch port are divided into - Network ports (Data ports) that connect to the DC/Enterprise networks and Device ports where the cluster of devices are connected.  With large density of ports in current generation of switches, some ports even can be dedicated to inter-device communication, there by avoiding any other back plane such as infiniband or some other L2 switch fabric.  L2 switches and devices providing ETS (802.1qaz)  and 10G ports'  support can use the same port for both inter-cluster communication as well as for network traffic. 

New Generation Configuration Framework

Even though there are multiple devices in cluster,  it is required that admin user configures the cluster only once.  Admin users should not be expected to configure each device in the cluster.   Fortunately new generation of configuration framework are designed to handle cluster configuration. 

New generation configuration frameworks support the mechanism to ensure that configuration is same across the devices in the cluster.  Increasingly,  configuration architecture supports central management system which takes care synchronization of configuration across devices on per operation basis.

Network devices maintain several statistics. With multiple devices in the cluster, each device maintains its own set of statistics. Admin user typically expects to see the consolidated list of statistic counter values across all devices in the cluster. Again, new configuration frameworks reads the statistics from each device, consolidates them and show the consolidated output. 

On image upgrade : When new image version is available,   new configuration frameworks allow admin users to upgrade the image only once for the cluster. All devices in the cluster would get the image from the central configuration framework.

With these advancements in L2 switches and configuration frameworks,  clustering is again back in the networks.

Fragmentation before Ipsec Encapsulation - Redside fragmentation and more use cases

I am finding more and more benefits of doing 'red side' fragmentation in Ipsec worl.

One use case is given here:  With red side fragmentation,  any  switches/routers in between security gateways of tunnels don't see  fragmented packets.  Due to this the cases, where some service providers' routers give less priority to the fragmented packets, don't arise. 

Second use case is given here :  When majority of  the traffic goes on IPsec tunnels, LAG can't distribute the traffic across ports since the result traffic  has same 5-tuple information. As described in the post,  multiple IPsec tunnels normally get created with forceful NAT-T.   All packets that are coming out of Ipsec Engine are expected to have 5 tuple information. If fragmentation is done after Encap, then the LAG would see some packets without 5-tuples.  This results to uneven distribution.  Hence redside fragmentation is done to ensure that LAG sees 5-tuples for all the packets.

Third use case:  Avoid mis-ordering of the packets:

There could be packets which are big and small in the traffic.  Big packets may get fragmented after Ipsec encapsulation if the result size exceeds the MTU of outgoing interface.  Small packets may not get fragmented even after encapsulation.

Gateway receiving the Ipsec packets is expected to process them in order. Due to fragmented packets, this may not happen.  Let us say that,  gateway received 1st fragment of 1st packet,  2nd full packet and 2nd and also final fragment of 1st packet in that order.  It is expected that the gateway processes them  in the same order.  But since 1st packet waits for 2nd fragment,  2nd full packet would be processed. Gateways don't stop the full packets getting processed as it may not know whether or not second  fragment of the 1st packet is going to come in and also when it is going to come in. 

So, this leads to packet mis-order.

This can be avoided if there are no fragments. Solution : Red side fragmentation.

Comments?

Link Aggregation and Ipsec

Link Aggregation is also called Ethernet Trunking and Bonding.  This feature is described in 802.3ad. In 2008, this was rolled into 802.1AX group.

What is LAG:

LAG combines multiple Ethernet Ports and exposes it as one link to the upper layers in the system.
It is Layer 2 concept.  Only trunk port would be assigned with IP addresses.  Links in the trunk don't have any Layer 3 information.  Only one MAC address would be used for the trunk.  Individual MAC addresses of the links don't appear in any communication other than control protocol (Marker protocol).

How does it work?

LAG contains two components - Distributor and Collector.
Distributor distributes the outgoing traffic across the links that constitute the trunk.  Collector collects the data in inbound direction coming from different links and tunnels through the trunk port to rest of the system.
LAG assumes that all links in the trunk are full duplex and point to point.
Simplest distribution is to distribute the packet by packet across the links based on weight configured on the links.  But there could be packet mis-ordering issues.


What are some critical items to be taken care by the Distributor:

Packet mis-ordering is one of the issues distributor would face if it distributes the traffic blindly on per packet basis.   To avoid mis-ordering,  distributors are expected to send all the packets of given flow (conversation) sent on the same link.  First generation distributors used to apply hash on source and destination IP and select the link based on hash value.  Though this ensures that the traffic belonging to one conversation goes on the same link, but the distribution may not be symmetric for some workloads.  Second generation distributors go one step beyond and apply the hash on TCP/UDP ports too.  This would give better distribution, but it may have some mis-ordering problem if the outbound packets are fragments.  Only first fragment would have transport header and other fragments don't have transport header. In those cases, there is a chance that non-initial fragments go on some other link and lead to mis-ordering.  Since fragments are not very common, some deployment accepts some level of mis-ordering to get the better utilization of the links.

Collector and Packet mis-ordering:  

To ensure that packets are delivered in order, collector should ensure to send the packets up in order it receives on any given link. There is no order to be maintained on packets coming in across links.  Collector also should ensure that it does not starve any link while receiving the packets.

Ipsec and LAG:

In some deployments, traffic is always encrypted via Ipsec and sent to the remote office.  If one tunnel  is used to send the traffic, all the traffic going from the local network to remote gateway contains same source, destination IP addresses.  In case UDP traversal is applied, it would have same source and destination ports.  Even if there are multiple links in LAG, distributor hash will fall onto only one link and other links would not be used. 

Same is true with Reverse traffic.  Also note that Links in Aggregation group are with local ISP.  Remote gateway under same admin control will not know about local Link aggregation.  That is, incoming traffic balancing across the links is in the hands of service provider.  It is okay to assume that most of 802.3ad distributors are configured with to use IP addresses and in some cases even ports.

Since distributors only know the IP addresses and ports of the packet, links would be utilized well in both directions if there are large number of flows with different IP addresses and Ports. 


Solutions

There are two solutions I can think of.

Solution 1:  Using Multiple IP address on the trunk link. 

Create as many tunnels as number of IP addresses on the trunk link with remote gateway.  As described in the link here,  some software should distribute the flows across these IPsec tunnels.  Since each tunnel now has different source IP address in the outer IP header, LAG distribution hash may fall into different links and thereby utilizing the bandwidth well in outbound direction.  One should ensure that, there are many local IP addresses to ensure that all links are used and also all links are used evenly.

Reverse traffic also would be balanced fine as service provider switch also would see different IP addresses (Destination IP).

Getting or assigning  multiple public IP addresses to the trunk may not be possible.  In which case, second solution can be used. But second solution would have some packet overheads.

Solution 2 :   Usage forceful NAT-T

Even though NAT is not detected,  there are ways to force UDP traversal. That is, ESP packets are sent with in UDP payload.  Create as many tunnels as necessary for good distribution at the LAG level.  Each tunnel would have different UDP source port.  Some software in the device is expected to balance the traffic across these tunnels. LAG would distribute the tunnels across multiple links of the LAG.  Reverse traffic also would be balanced on different links due to different destination port values of tunnels.  Since it is expected that LAG distributor look at the transport header for distribution, it is necessary that there are no fragments.  So, it is mandatory that tunnels are configured with redside fragmentation. This will ensure that fragmentation is done before Ipsec encapsulation.


In both the solutions,  both remote and local gateways should have some logic to

• know that multiple tunnels are created for same selectors for distributing the flows.
• know how to distribute different conversations to different tunnels.

It requires more tunnel capacity in devices.  This should not be a problem as modern devices has good horse power and enough memory to create some more tunnels with peer gateway.


Comments?

Web Application firewalls, IPS & Network Anti Virus - Fixing the performance issues

Security professionals know that the intrusion and malware detection is now beyond looking at stream of packets.  Detection require

• SSL Decryption - Many client side attacks are increasingly hidden in HTTPS connections.( Check this out )
• Extracting data from the packets (Example:  HTML, Javascript,  Different types of files to detect attacks embedded in the data)  ( See this )
• Decoding the data (Such as UTF-8, UTF-16, De-compression etc..)
• Emulation of data if the data is script (such as Javascript) to counter evasion techniques used by attackers.
• Comparing with known signatures or codelets OR doing some kind of heuristics

This kind of analysis is not possible with stream based firewalls, IPS and AV.  These require collection of data.  They all require proxies. If some IPS/AV vendor says that they do detection without reassembling data and collecting the data, then as end user you will not be wrong to say that they either miss lot of intrusions or give too many false positives.

Computational power to do above is very high.  It is not surprising to see just less than 10Mbps of IPS, AV combined performance in devices which give 1Gbps of firewall, Ipsec throughput.  I hear stories of customer disappointments when they turn on IPS and/or AV functionality in security devices.

Network security analysts advising companies to enable full functionality even for traffic originated from trusted networks.  It should not be surprising anybody as trusted network boundary is reducing  due to mobility of machines in trusted network. That is, machines are moving from trusted to untrusted and vice versa. Examples :  laptops, ipads etc..   These  machines may get infected when they are in untrusted network and may get infect other machines in trusted network when they are brought into corporate networks. That is the reason, now full protection is being enabled on the security devices. 

HTTP is singlemost protocol that occupies majority of network bandwidth in many organizations.  HTTP is also interactive protocol. Any performance issue also  impact the user experience.   Solving HTTP performance problem not only improves user experience, but also would increase the performance of overall system.

Techniques that can be used to improve the performance of HTTP Anti-malware and IPS analysis are given below. End users might look for following features.

• Avoid doing duplicate IPS and Anti-Malware checks :  It is very common tha same resource is requested by same/multiple users in the orgnaization via HTTP.  Nework device once AV and IPS check is done on the resource should avoid doing the check again.  This requires caching of AV and IPS analysis and using it when the same resource is requested at later time.   Ofcourse, it should have life time so that it checks for AV/IPS if the content of the resource is changed.  Life time can be equal to the Expiry time of the resource which comes along with the HTTP response headers.  If possible,  this system also can do caching of the response which avoids even going to origin server, there by saving the WAN bandwidth too.  I believe that AV/IPS devices would have HTTP Caching moving forward.
• Auto blacklisting of URIs :  Malware may be served with dynamic content. In which case, above mechanism of caching does not work.  More often, Malware is served using the same URI.  If the data downloaded from a URI contains the malware, that URI can be blacklisted if malware is detected multiple times.   If the request comes to the same URI at later time,  request can be denied without even senidng the request to the origin server.  Always make sure that the newer blacklisted entries are honored by the device.
• TCP and SSL offload:  Proxies can benefit greatly if some other entity such as intelligent PCI-e takes care of TCP/IP stack and SSL offload.   
• Implement proxies as per my earlier post.
• Usage of Multicore processors and distributing the load across multiple cores.  Selection of Multicore processor depends on several factors such as cost, number of cores (performance), acceleration features etc..  But here I am only covering the features.  Features that would help in processing are :
• Processing power - Higher the processing power, better the performance would be.
• Cache Size matters:  Unlike typical firewall/Ipsec processing, amount of code that gets executed in doing AV/IPS analysis is lot higher. Higher sized L1 and L2/L3 caches would store more instructions and goes to DDR less often.  Cache for storing data is also important.
• Acceleration hardware -
• Compression/Decompression Accelerator:  To take care of decomperssing the compressed files coming in the HTTP response.
• SIMD (Single instruction Multiple Data) based hardware to do acceleration of
• Memory /String operations - Copy, Set
• Checksum, CRC operations
• HTML and URL decoding operations.
• and many more...

Hope it helps..

Performance considerations in Proxy based nework applications

Many details on performance considerations on proxy based networking applications are given here. 


There are some more performance considerations in developing proxy based applications. Here they are:

• Use Hugelbfs system to for running code and for application context memory (for connections):  Please see this for getting understanding of this technique.
• Use User space RCU wherever possible:  See the RCU related information here.
• Use Futexes as part of RCU implementation for add/delete operations.  See about Futexes here. 
• Use posix spinlock kind of Mutexes only for small portion of the code.
• Use UIO based Interrupt indication to the User space processes while dealing with memory mapped hardware accelerators.

SMB Evasions by attackers - Tips to prevent them in IDS/IPS devices

DCE RPC packets can also come with SMB.  This article talked about some of the DCE RPC evasions by the attacker and way to detect the attacks even with these evasion techniques.  Since DCE RPC packets can come on SMB packets, it is important to understand some of evasion techniques used by attackers on the SMB protocol itself. 


DCE RPC messages are predominantly embedded in SMB Messages such as SMB_COM_READ response,  SMB_COM_WRITE and ANDx versions of them. Also DCE RPC messages are also sent with SMB_COM_TRANSACT command and response messages. Note that these evasion techniques are not only useful for detecting attacks in DCE RPC based applications, but also CIFS (SMB) itself. 


Protocol details of SMB are described very well here.  


Many IDS/IPS devices don't have protocol intelligence of SMB and DCE RPC protocols. IDS/IPS systems that depend on generic pattern matching can be bypassed by attackers with simple evasion (obfuscation) techniques.  Let us examine some of the evasion techniques.


1. ANDx messages:


As indicated above, there are commands with ANDx version. Any command/response ending with ANDx have following structure in the packet after SMB Header.

SMB_Parameters
  {
  UCHAR  WordCount;
  Words
    {
    UCHAR  AndXCommand;
    UCHAR  AndXReserved;
    USHORT AndXOffset;
    USHORT FID;
    ULONG  Offset;

    USHORT MaxCountOfBytesToReturn;
    USHORT MinCountOfBytesToReturn;
    ULONG  Timeout;
    USHORT Remaining;
    ULONG OffsetHigh (optional);
    }
  }
SMB_Data
  {
  USHORT ByteCount;
  }

Variable size of SMB Parameters : Many IDS/IPS devices assume that the 'wordcount' is constant for a given ANDx message.  For example,  for SMB_COM_READ_ANDx message, the 'word count' is assumed to be 10 words. But it can be 12 words in case of 64 bit offset (OffsetHigh).  IDS/IPS devices assuming 10 words and interpreting the data would have the detection wrong.  Attacker deliberately set the wordCount to 12 words even though OffsetHigh is 0.  IDS/IPS devices must interpret the 'word Count' to move to the data section. 


Multiple ANDx messages under one SMB message (with one SMB Header): 


Many IDS/IPS devices assume that there is only one command (or response) in the SMB message. But SMB protocol allows multiple ANDx commands (or responses) in one single message.  Every command/response would have its own 'SMB Parameter' and 'SMB Data' blocks. Attackers can put the malicious command/response as non-first command/response in the SMB message to bypass detection by security devices. IDS/IPS devices must interpret the 'AndxCommand' to figure out whether any more commands/responses are present in the message.  AndXCommand is normally set to 0xFF if there are no additional commands. 


Filler between ANDx messages :  AndXOffset field indicates the next command in the SMB message. Since there is explicit mention of offset, is is protocol wise legal to send some additional filler data between Andx commands. Attacker can take advantage of this and put some data to confuse security devices. Security devices thinking that all commands are next to each other would fail to detect the attacks.  Security devices must be aware of this and interpret the AndXOffset appropriately as end systems do.


Out-of-Order of ANDx messages : Here AndX commands can refer to the data in the SMB messages before AndX header. Note that AndXOffset indicates the offset from the beginning of SMB header.  Hence it can be set to any place in the SMB message. This is tricky for IDS/IPS devices as they need to store the complete SMB message before analyzing and hence it increases the memory requirements.  But it is necessary to do this to mitigate any evasion techniques used by attackers.


2.  Transaction Messages


Transaction command messages have this structure. Responses also have similar structure but some fields don't exist. So, be careful in analyzing the command and responses. 

SMB_Parameters
  {
  UCHAR  WordCount;
  Words
    {
    USHORT TotalParameterCount;
    USHORT TotalDataCount;
    USHORT MaxParameterCount;
    USHORT MaxDataCount;
    UCHAR  MaxSetupCount;
    UCHAR  Reserved1;
    USHORT Flags;
    ULONG  Timeout;
    USHORT Reserved2;
    USHORT ParameterCount;
    USHORT ParameterOffset;
    USHORT DataCount;
    USHORT DataOffset;
    UCHAR  SetupCount;
    UCHAR  Reserved3;
    USHORT Setup[SetupCount];
    }
  }
SMB_Data
  {
  USHORT ByteCount;
  Bytes
    {
    SMB_STRING Name;
    UCHAR      Pad1[];
    UCHAR      Trans_Parameters[ParameterCount];
    UCHAR      Pad2[];
    UCHAR      Trans_Data[DataCount];
    }
  }

Fragmentation :  If application payload is bigger than the 'MaxBufferSize' negotiated during setup phase,  application payload is divided across multiple SMB messages with first message having SMB_COM_TRANSACTION command/response and further messages are sent with SMB_COM_TRANSACTION_SECONDARY.   Attackers take advantage of this to evade the detection by security devices which are not reassembling the data that is sent across multiple 'TRANSACTION' messages, even if the real application data is less than 'MaxBufferSize'.  Security devices must ensure that all messages have come in to reassemble by checking the 'TotalParameterCount' and 'TotalDataCount'. If all transaction messages (with same PID, MID, TID, UID in the SMB Header) parameter count and data count adds up to TotalParameterCount and TotalDataCount with different ParameterOffset and DataOffset, then security device can assume that all fragments are received. Note that some attackers try to fool security devices by sending duplicate SMB messages.  Security devices blindly adding 'ParameterCount' and 'DataCount' of all matching SMB messages to match 'Total ParameterCount' and 'TotalDataCount' without checking for unique 'ParameterOffset' and 'DataOffset' can be bypassed with attack detection by sending duplicate SMB messages.


Out-of-Order of Transaction fragments :  As seen before 'Parameter Offset' and 'Data Offset' indicate the position of the parameter and data section of the message in the overall application payload. End SMB systems honor these values while reassembling. So, the order in which they come is not important.  Security devices, if they assume that the packets would be in order' can be evaded by attacker by sending these messages in different order.  

Detecting the Malware & Intrusions inside SSL/TLS Connections - Tips

I can't stress any more on the need for detecting and preventing attacks on client applications (Desktop/Laptop/Smart phones).  Network security devices certainly have  ways to detect the attacks on HTTP clear connections, but not when attacks are present in the HTTPS connections. Attackers may relay on this and might host malware on HTTPS and invite innocent users to browse these sites through social networking mechanisms.

To detect the attacks on HTTPS and other SSL/TLS connections, it is required that 'Network Security Devices' act as good Man-in-the-middle and get hold of clear data by decrypting it. Typically servers authenticate themselves to the clients using X.509 certificates and client authenticate to the servers using 'user name'/'password' combination.   Network Security Devices protecting servers can do this easily as administrator can upload server certificate to the network security device which in turn can terminate SSL connection on behalf of servers and analyze the clear data.  But when network security devices protect the client machines, server certificates are not in the control of administrator.  On top of that, there are large number of servers (Servers in Internet) that get accessed by client machines.

Since 'Network Security Devices' need to get the clear data,  only choice to terminate the SSL connections even if they are going towards servers in Internet.  So, one way network security devices (NSD) can do this is by creating certificate dynamically  signed by local CA certificate (self signed certificate).  There are three main considerations these NSDs should ensure.

• User experience at the browser should not suffer on per page/site basis.
• Performance of NSD should not be overly degraded.
• Security of the network should not be compromised.

Let us revisit how these requirements are met after discussing the approach first.

Flow of the Connection:

• NSDs should act as SSL/TLS Proxies.
• NSD terminates TCP Connection made by client.
• NSD makes TCP connection to the Server.
• NSD should also proceed with  the SSL Connection with the Server (Server authenticates with the client via X.509 certificate).
•  NSD must have all known CAs configured at factory default time. Also, it should have facilities to upload new CA certificates, deleted existing CA certificates.
• It is good to understand popular CAs from the browser default repository and ensure that those CA certificates are present in NSD  for authenticating original servers
• If authentication is not successful during authentication phase (There could be multiple reasons : No matching CA certificate,  Certificate was expired,  Certificate was revoked etc.. ), then it should send this information to the client with  with exact reason and all possible information in the certificate extension (vendor) and it can also send this information to the browser (if it is HTTPS). This information can be communicated with the administrator so that he can take any remedy that is possible such as populating with new CA certficate(s).  Whether server authentication is successful or not,  following step of dynamic certificate creation is required.
•  NSD is expected to have pre-configured certificates for its authentication with the clients. NSD is also expected to have self-signed CA certificate.
• NSD changes the pre-configured certificate with information from the received certificate from server in previous step.  Information such as 'Subject name',  'Extensions',  'Serial Number',  'Validity' would need to get copied from server certificate.   If this certificate is being created to send 'Server authentication failure', then  ' Subject name' only is copied from the server certificate as 'Serial number' and 'validity' may be the actual culprits for server side authentication failure.
• Issuer name should be the subject name of 'Self signed CA certificate'.
• NSD signs the certificate with 'Self Signed Certificate's private key.
• Put it in the cache. If the cache is full, remove the currently un-used old entry.
• Note that above operations need not be done, if the cache entry is found matching with the 'Subject name', 'Serial Number', 'Validity', 'Issuer name' of received server certificate.
• Now NSD terminates SSL Connection on the client side.
• Browser may object to this authentication first time as it does not recognize the 'self signed CA' certificate. Administrator of the organization is expected to give instructions to employees of organization to store this self-signed CA certificate in browsers. Once it is accepted,  browser will not complain again.
• Once SSL Connection is established on both server and client side,  clear traffic will be seen by the NSD and it can do every thing it can do to detect and protect client machines the sane way it does with clear connections.

 Now let us revisit on how the main considerations are taken care?

User experience at the browser should not suffer on per page/site basis :  Only thing the end user needs to do in the browser is to accept the 'Self Signed Certificate' provided by the NSD one time.  Since NSD is ensuring to keep the Subject name intact,  browser does not keep alerting the user for every site he/she visits.


Performance of NSD should not be overly degraded :  NSD is not generating the certificate/private key pair for every SSL site that is being visited by users.  It is expected to have some X number of certificates/private key pairs with random subject name,  Serial number.  On per connection basis, these are changed with values from the server certificate received as part of SSL connection  and then the signing processing occurs.  By avoided generation public/private-key pairs,  performance does not go down dramatically.  Also, this new certificate/private-key pair is cached.  Only when there is no space in the cache,  there may be possibility of this pair getting removed.  But immediate HTTPS connections to the same server would not undergo above process.


Security of the network should not be compromised : There could be concerns on security.  But it is not all that bad.  NSD itself is authenticating the server with trusted CA certificates. It also provides the information of CA certificate (Mainly subject name, issuer name, Serial number) in the dynamically generated certificate as part of extension so that clients can look at this information when he/she views the received certificate in browsers.  Clear data is visible within NSD device, but not on the wire. SO there should not be any worry about informational disclosure.

Also, administrators normally would be provided with white list configuration where administrator can configure two kinds of white lists - White list of client IP addresses,  White list of destination IP addresses (administrator can configure domain names).  If the connection is coming from the IP addresses which is in the white list, then NSD is not expected to apply MITM mechanism, that is SSL Connection should not be terminated. Similary, if the the destination IP falls in the white list of destination IP addresses, then also SSL connection should not be terminated.  This facility provides additional tool for administrators to satisfy their end users on doing SSL MITM proxying selectively.

If you are a buyer or already have a security product,  ask your vendor whether the device can protect you from attacks that are hidden in SSL connections.

Network Services deployments in Data Centers

Application Delivery controllers are already part of the many data centers where there are multiple servers to serve the content to end users. ADCs balance the incoming connections to multiple servers for high availability and also to ensure that load is shared.

Let us see where they fit in the data center architectures as listed in here. 

You might have heard the term 'Network Services Layer'.  This particular area contains network security devices, ADCs and WAN optimization devices.  Many traditional data centers don't have this layer, but newer data centers have this layer.  This layer works in conjunction with Core switching layer. 

Core switches are expected to be configured to pass the selective traffic to network services layer.  There is no change expected in access switch layer or SAN switch layer.  One can have security at these layers too for additional security,  but I guess it will be some more time before data center administrators add any additional security at access level.

Before getting into the details of capabilities of core switches, devices in network switch layer,  let us first visit the addressing of typical data centers:

Many time private IP addresses are used for servers.  Though public IP address can be assigned to server(s) if there is only one for a domain name or in case of DNS load balancing, it is my observation that many times only private IP addresses are assigned.  There are several reasons:

•  If there is a need in future to expand to multiple servers due to load or high availability considerations,  no changes would be necessary except for some configuration in ADCs and bring up new server (or virtual machine).
• Many times, servers need to communicate with other internal servers such as database servers, SANs, application servers etc.. .   Private network gives comfort of security as well as reduce the need for number of public IP addresses.
• For a given domain name,  multiple services may need to be exposed.  Also, for each service, different physical server or different virtual machine may be required.  Private IP addressing with ADCs translating to right private IP address on incoming connections help in facilitating this. If public IP address is used, then all services need to be put in same physical server or virtual machine.

Typically private centers have one domain name and multiple services. Public data centers have multiple domain names and multiple services in each domain.  Each domain name + service name combination may contain multiple servers serving the content/users.    You might have heard term called server farm.  A given server farm is identified by  IP address (resolved IP of domain name ) and Service (Ex: Port 80, 25 etc..). Server farm is configured with IP addresses of real servers serving the content.  ADCs do this job of selecting the least loaded server dynamically upon receiving client connection and awards the connection to selected real server.  In recent times, real servers are subdivided into multiple subsets and each server farm provides additional configuration (rules) to select the subset.  Then least loaded server is selected from this subset.  One example deployment may put  all videos, images and static content  in some servers (set1) and other servers serve the dynamic content (set2).   ADCs can be configured with rules to select set1 for URL containing some file extensions and set2 for everything else. I hope server farm concept is clear.

It is important to understand how ADCs are deployed first:

Small data centers don't have two tier architecture of switches. Jut a simple ADC is good enough. This acts as switch as well as Load balancer.  ADCs have L2 switch and one or more Ethernet MACs.  Ethernet MACs are connected to the network that gets connected to the WAN links and servers get connected to the L2 switch. Basically, in simple deployments ADC acts as access layer switch on server side and Core router/switch on the core network side.  Let us call this as 'simple data center'.

In complex data centers or in public data centers, two tier architectures are required. In this case, core switches are configured to pass the traffic coming from core network to the ADCs and traffic from server network to ADCs for load balancing purposes.  Traffic from server network might go to ADCs without any special configuration in core switches if the client to server packets were translated with SNAT. Due to this SNAT,  server to client traffic will have DIP as the ADC ip address and hence packets will go to ADCs without any special configuration in the core switches. Data centers are complex because it needs to handle large amount of traffic and/or has large number of server farms.  Due to this, some times one ADC may not be enough to take the load of all server farms.  Multiple ADC devices are used in those cases with each ADC handling traffic belonging to few server farms.  In these cases, core switches are expected to provide facilities to segment the traffic on server farm basis and redirect the traffic to ADCs for balancing the traffic across servers within server farm.  To give an example:  If a data center has 100 server farms and 10 ADCs, then the core switch should have capability of segmenting the traffic 100 ways and passing 10 sets of traffic to appropriate ADCs.   This is typically achieved via VLANs.  

As discussed above each server farm is identified on incoming traffic (from core network) by public IP address (resolved IP of domain name) and the port.  If there are 100 server farms, then 100 VLAN IDs are required.  Core switches can be configured to generate VLAN ID based on the incoming traffic.  Core switches have this capability called 'rules'.  Rule can be created with selectors (in this case destination IP and destination port) and action as 'redirect' with VLAN ID and Port on which to transmit.  Switches, when the traffic matches with this kind of rule, adds VLAN ID to the packet and transmits the packets on the port indicated in the rule.

Let us take an example of public data center.  Let us assume that there are two domain names - www.example1.com (P1 public IP address) and www.example2.com (P2 public IP address)   www.example1.com has two services - Port 80 and Port 25.   www.example2.com has one one service - Port 80.   Example1 company wanted four servers - P11, P12, P13, P14 to serve Port 80 content and 1 server for Port 25.  Example2 has two servers P21 and P22 to server port 80 content.  It was decided to use ADC1 and ADC2 with ADC1 handling two server farms of www.example1.com and ADC2 handling www.example2.com server farm.   Let us also assume that ADCs are deployed in two-arm mode.  On one arm it expects client-to-traffic to land and another arm it expects server to client traffic land.  Let us assume that VLAN11 and VLAN12 are for client and server traffic of www.example1.com. And VLAN21 and VLAN22 are for client and server traffic of www.example2.com.  Let us also assume that two physical ports PR11 & PR12 and PR21 & PR22 are used to connect to ADC1 and ADC2 respectively. 

In above deployment, core switch cluster is expected to be configured with following rules:

• On ports that are towards the Internet:
• Source IP : ANY  Destination IP: P1  Destination Port : 80 or 25 --------> Add VLAN11 and send on PR11.
• Source IP : ANY Destination IP: P2  Destination Port : ANY --------> Add VLAN21 and send on PR21.
• On ports that are toward the Servers
• Source IP:  P11 Subnet,  Destination IP: ANY, Source Port 80, 25 ----> Add VLAN12 and send on PR12.
• Source IP:  P21 Subnet,  Destination IP: ANY -------> Add VLAN22 and send on PR22.

ADCs/Server load balancing devices would need to have capability to instantiate VLAN interfaces based on VLAN ID.

Network Security Devices and WAN Optimization devices also can be deployed in similar way.  It is good if all functions are available in the same device. This would only require one set of VLAN configuration in the switch. If  these devices are independent to each other, then switch need to configure multiple VLANs to pass traffic from one device to another via switch.  If routing is allowed, then the one device can have route to another device and switch will pass the traffic from one system to another as any switch do. 

In above example, ADC1 is handling two server farms. But it is also possible to divide these two servers across two different ADCs even though they belong to same domain. As long as service is different, it should be possible.  It is also possible to use one ADC for multiple domain names too.  In this case, ADCs or network security devices provide 'virtual instance' to ensure that traffic of two different domain names are independent and isolated. I am too lazy to type in the configuration required on switches for these two configurations, but I think you got the picture :-)

Data Center Switch requirements for new Data Center Architectures

Traditionally data centers have three tiers of switches -  Core switches,  Aggregate switches and Access switches.

• Core Switches :  These switches connect to the network which are connected to the WAN links. This is farthest switch farm with respect to servers.
• Access Switches :  These switches are also called top-of-rack switches.  Servers (Web Servers, Email Servers, Application Servers, Database Servers and others for which data center is built) get connected to the ports of these switches. 
• Aggregation Switches:  Aggregation switches is intermediate switch layer which is sandwiched between Core and Access switch layers.  Aggregation switch aggregates the traffic between core and access layers.  Note that there could be lot of traffic among servers (Specifically among application, web and database servers). This traffic need not be seen by the core switches.  This traffic just need to be among the access layer switches.  Aggregation layer eliminates the traffic being seen by every switch.  Core switches only see the traffic going to/coming from WAN/Corporate network.  Aggregation layer also reduces the traffic among access layer switches.

It was necessary to have three tiers in earlier data center architectures due to

•  Large number of physical machines serving the content requires large number of Ethernet ports.  Due to poor density of the ports on the switches, multiple access layer switches were necessary. Multiple switches means there is lot more traffic across access layer switches. One more hierarchy of switches enable good throughput by eliminating mesh kind of access layer switches for intra switch traffic.

What are some of the changes in Data Centers? One big change is collapse of three tiers to two tiers. Aggregation layer is disappearing.  Let us see what is making this change.

• Virtualizaton technology is reducing the number of physical machines:  This implies that there are less number of ports.
• Traffic on each port is increasing : Virtualization and Mulitcore processor are enabling multiple applications in one physical machine.  It is not uncommon to see the requirement of multi-gig traffic on a single port.
• 10G and in future 40G/100G ports are facilitating the unified fabric for both kinds of traffic - Application traffic and SAN traffic, thus eliminating number of ports and interconnects.

These technologies are reducing the cost by reducing equipment, interconnects, by amount of power required and amount of cooling required. It also reduces the maintenance and hence reduction in cost.

What kind of features one would expect in the switches in new data centers:

• Latency of traffic should be very less:  By eliminating the aggregation layer itself reduces the latency. But that is not good enough for SAN traffic, Video and Voice workloads.  Non-blocking switching or cut-through switching is expected to support real time traffic such as Video, Voice etc..   Traditionally, switches oversubscribe the bandwidth, that is, switches are not capable of receiving and transmitting of traffic of all ports at the same time with full port bandwidth. Hence the packets get blocked.  In non-blocking switches,  they are expected to send and receive traffic equal to number ports * each port bandwidth.  If there are ten 1G ports, switches are expected to receive 10G traffic and send 10G traffic.  
•  802.1qbb (Priority based Flow Control):  When there is a congestion in the receiving node, 802.3x pause frame is generated normally. This makes all the traffic pause for some time. This standard allows pause frame generation on 802.1p priority levels. It lets the high priority traffic flow. Switches are expected to honor and generate theses kinds of frames.
• 802.1qaz (Enhanced Traffic Selection):  This standard allows the bandwidth allocation for different priority levels or group of priority levels.  It lets higher priority bandwidth to be consumed lower priority traffic if there is no higher priority traffic.  SAN traffic would need to be going with higher priority levels. This feature is also expected to be supported by data center switches.
• 802.1qau (Congestion Notification):  This standard allows end nodes to communicate the congestion notification.  It lets the end node receiving the congestion notification to apply rate limiting on the out traffic.  This feature is also expected to be supported by data center switches.
• Port Density should be high.
• Multi-Path support is required - I am not sure whether there are any standard at this time, but spanning tree is not used in these cases as it only provides one path. 
• VEPA Support would be required eventually. Due to VEPA,  it may need to support C-VLAN and P-VLANs.
• Large number of VLANs support is required to work with other network services such as ADCs, WAN Optimization and Network Security (Firewall, IPS, IPSec VPN etc..). 
• Ability to redirect the traffic not only based on L2 and L3 fields, but also L4 fields such as TCP, UDP Source and destination ports. 
• Any switch architecture should work with VM migration from one physical server to new physical server.
• Public Data Center networks require Virtual Instance kind of concept within the switches to reuse VLANs (across different subscribers) due to limited number of VLAN IDs.  

Linux based fast path - Why is this needed?

I found two excellent blog posts on this subject. Please see them here:

http://www.multicorepacketprocessing.com/the-need-a-fast-path-on-multicore-cpu/
http://www.multicorepacketprocessing.com/os-networking-stacks-like-linux-are-not-well-adapted-to-multicore-packet-processing/

I see this statement in one of blog entries "However, the stack of any OS cannot go under ~2000 cycles per packets due to its design.".  It is really true.  Linux and other operating systems are generic and they need to cater to multiple different types of applications. Generality always comes with the price.  Forwarding process of IP packets in  Linux for example go through many layers of software from reception of the packet to the transmit of the packet.  These layers add to cycles.  As excellently listed down in above blogs,  higher layers of the operating systems don't have full control over the hardware (processor and accelerator) features. Since many layers of software comes in picture,  cache utilization is not very good.  On top of it, context switching and locks (for Multcore processors) in these generic layers would use up more core cycles.

Fastpath is nothing new. This has been adopted by networking vendors for a long time.  Fastpath is specific to applications running in the networking devices.  For example,  in networking devices,  fastpath layers are implemented for Firewall, IPsec VPN,  QoS and forwarding.  Fastpath concept is very simple and described below.

• When packet is received, it first checks for the matching context.
•  If no context, packet is given to the normal application modules.
• Normal path based on its policy rules may decide to handle the session by creating its own context in its application. In addition it either decides to create the context in FP, or decides to handle the context itself by not populating the context in FP
• If there is a matching context, packet is handled within fastpath and packet is routed without involving the normal path.

How/Why do fastpath modules get good performance?

• It runs right above hardware and it has access to all hardware features.
• Since FP is specific to given hardware, it can take advantage of all features without worrying about losing generality. Since fastpath module for any given application is expected to be very small in footpriint,  not having generality across all hardware devices is considered okay.
• Fastpath implementations follow run-to-completion model.
• Due to its small footprint,  most of the fastpath code might be in the L1 Cache of processors.
• They can do locking of critical data blocks in processor caches.
• Lock free implementation - Linear performance with cores

As the 'Multicore processing' blogs have indicated it is no surprise to see 3 to 10x performance improvement based on the type of application.

There are different types of fastpath implementations in the industry today.

• ASIC based fast path implementations.
• Network processor based fastpath implementation (Remember NPF. They even have set of API documents for fast path. You can still find there here )
• Control plane and Data Plane with some cores running CP and some cores running FP in DP.
• Linux Ethernet Driver based Fastpath for devices that use Linux SMP.

There are some pros and cons among different fastpath approaches. But the basic idea is same across them.  Do routine jobs of packet processing in fast path and let normal path handle special connections and packets.

I believe many  network device vendors would need to go for some kind of fast path if they need to support high volume of traffic with cost effective processors.

Linux TCP Large Receive Offload optimization to increase performanace

In some network packet processing applications, number of packets being processed determine the performance. TCP is a streaming protocol and hence there is no packet boundary. Hence consecutive packets can be aggregated into few packets when the TCP packets are received at the lowest level.  More the packets that can be aggregated, higher the performance would be.  Applications that can benefit are:

• Any Proxy based applications (Application Delivery controller, WAN optimization,  Network Anti Virus)
• IDS/IPS 
• Firewall ALGs.
• Server Applications

I found one excellent paper describing two techniques to improve the TCP connection throughput performance -  Receive aggregation  and  Acknowledgment offload.  Please find it here. This paper also gives performance improvement with receive aggregation and without these optimization techniques.  Performance was improved from 3.4Gbps to 4.6Gbps, 35% increase.

Receive aggregation technique is already implemented in Linux 2.6 kernel. It is called Large Receive offload feature.  This feature is implemented in net/ipv4/inet_lro.c.

Receive aggregation technique is simple. It is used only when the NAPI functionality is applied on the Ethernet driver.  In NAPI enabled Ethernet drivers,  softirq receives the packets from the descriptors. Typically NAPI reads out all the packets from the receive descriptors (or until some threshold - quota). 

• Ethernet Driver normally sends up the packet to the stack using netif_receive_skb if the NAPI is enabled. In case of LRO,  packet is given to the LRO library using lro_receive_skb function.
• Find the matching flow. If no match, it creates new flow.
• LRO module figures out whether this packet is eligible for aggregation. Packet is non-eligible if any of following conditions apply.
• Non Padded frame (IP total packet length must be received packet length)
• Non-TCP packet.
• IP options are present
• IP ECN CE is set
• TCP segments has no data.
• CWR (Congestion Window Reduced) flag is set
• ECE (ECN Echo) flag is set
• SYN flag is set
• FIN flag is set
• URG flag is set
• PUSH flag is set
• RST flag is set
• ACK flag is not set
• Non TCP Timestamp option is present
• If the next sequence number expected is matches with the sequence number of this packet, packet is added to the existing packet sequence.  If not, packet is not eligible for aggregation.
• When the packet is found to be not eligible for aggregation, it is necessary to send buffered packets first to the stack before sending the current packet. This is done using lro_flush() function.
• If the packet is eligible for aggregation, it associated with existing packets by manipulating the skb.
• When the aggregation stops, it does following before sending the aggregated packet to the stack.
• Changes the ACK to the last packet ACK.
• Keeps the timestamp option of the last packet.
• Recalculates the IP checksum (now the packet became bigger).
• Partial Checksum update of TCP payload.
• When does the aggregation stop:
• When the configured aggregation limit reaches.
• When the total packet size is more than (64K-MTU).
• When the NAPI finishes all the packets in receive descriptors or reaches its quota.
• Ethernet Driver is expected to send all the packets so far buffered at the end of current NAPI instance. It does so by calling lro_flush_all.

PCIe Endpoint developer considerations

See my earlier post on "PCIe End Point Developer Techniques" on some of the considerations developer should  keep in mind.  This post will give some more expectations from Endpoint.

Interrupt Coalescing:

Endpoints will interrupt the host whenever it puts data in the receive descriptor ring(s) . Also, Endpoint interrupts the host whenever it transmits the packet out from the transmit descriptor ring(s).  Interrupt coalescing is normally used to reduce the number of interruptions to the host.  Interrupt coalescing configuration  by host on Endpoint will result in Endpoint interrupting the host only after filling up certain configured number of descriptors or after some configurable amount of time is passed from previous invocation of interrupt.  Time parameter is required to ensure that host gets the interrupt even if configured number of descriptors are not filled up. This parameter should be chosen carefully to reduce the latency of packets.  Interrupt coalescing also can be configured by host to reduce the number of interrupts when descriptors are removed from the descriptor rings.

As an Endpoint developer, always make sure to provide interrupt coalescing parameter configuration to host for every interrupt that gets generated by the Endpoint.

Now the processors are used in Endpoints,  similar functionality is required in reverse direction.  End Point should be able to indicate the interrupt numbers it would assign to the host. In addition, end point can also have interrupt coalescing parameters  in configuration registers for each interrupt.  Host will read them and use this information to reduce the number of interrupts it invokes on the Endpoint. Since these are registers,  host also can put different configuration values.

As an Endpoint developer, make sure to provide this configuration to the host. If you are developing the host driver,  please ensure to use interrupt coalescing functionality.

Multiple Receive Descriptor Rings and Multiple Transmit Descriptor rings:

 In case of intelligent NIC cards, multiple receive descriptor rings are used to pass different kinds of traffic.  For example,  packets related to configuration & management is given highest priority compared to other data traffic. Within data traffic, voice & video traffic would be given higher priority normally.  Different descriptor rings are used to pass different priority traffic.  Endpoint, upon receiving the packets from wire, classifies the packets and puts them in appropriate rings.   Host driver is expected to select the descriptor ring to dequeue the packet.  Basically, scheduling of rings is used to select the ring.  To enable scheduling, rings are given weightage. Some important rings may also be placed in strict priority.   Scheduler in the host is expected to select the strict priority rings if they have packets. If no packets in strict priority rings, then  a ring is selected from weighted ring set for dequeuing the packet.

Similarly,  multiple transmit descriptor rings are used by the host for multiple reasons. One reason is similar to the reason described above for receive descriptor rings. That is,  multiple descriptor rings might be used to send different priority traffic.   Endpoint software is expected to do scheduling across multiple rings to select the ring and then dequeue the packet for transmission on the wire.  This scheduling also could be similar to the one described above.

Set of descriptor rings meant for different kinds of traffic can be termed as 'Descriptor Ring Group".

As Endpoint developer,  one should provide this kind of flexibility for host to process high priority traffic first over low priority traffic.

Multiple Descriptor Ring Groups:


In Multicore processor environments, avoiding locks is very important for achieving the performance. Endpoint is expected to provide facilities such a way that host never need to do any locking while dequeuing the packets from receive descriptor rings or while en-queuing the packets onto the transmit descriptor rings. That is where,  multiple descriptor groups are required.  If the host has 4 cores, then at least 4 receive and transmit groups are needed to avoid  lock by assigning each group to different core.  To also ensure that right core is woken up upon interrupt,  it is also necessary that each group has its own interrupt which is affined to the appropriate core.  That is, when a core is interrupted, it exactly knowns which group of descriptor rings to look at. Since a group is accessed by only one core, there is no lock required.

Now that  Endpoints are also being implemented on processors, it also becomes important to reduce the locks even on the endpoints.   This makes the problem little more complicated.  Let us assume that host has 4 core processor and endpoint has 8 core processor.  As discussed above, since host has 4 cores, 4 groups are good enough.   That is, 8 cores in the endpoint would be updating only 4 groups.  Since more than one core would be manipulating the group, lock would be required in the endpoint.  To avoid lock in both places, it is required that the number of groups need to be at least maximum of number of cores in host and endpoint.  Descriptor groups would be equally divided across multiple cores on each side.  Even though this formula avoids the lock in both places,  but the traffic distribution can be a problem if the number of groups can't be divided across cores equally. This will lead to some cores getting utilized more than other cores.  If you take an example where the host has 4 cores and endpoint has 7 cores,  then to make the distribution of groups across cores equal on each side,  one should have GCF of 4 and 7. That is, 28 groups are required.  On host side, each core would need to be assigned with 7 groups and on endpoint side, each core is assigned with four groups.

If one core has more than one group, then there could be two challenges.  One is interrupt assignment and other is distribution across groups.   Even though PCIe support MSI and MSI-X, there could always be some practical limitations on the number of interrupts that can be used.  Hence each group can't have its own interrupt.  Since interrupts are used to interrupt the core,   one interrupt per core is good enough.

Producer core is expected to distribute the traffic equally among the groups - Round Robin on packet basis would be one acceptable method. Once the group is chosen,  based on the classification criteria as described in the previous section a descriptor ring  in the group gets selected to place the packet. Consumer core is expected to do similar RR mechanism to select the group to dequeue from.  Once the group is selected, another scheduler as described under the previous section would be used to select the descriptor ring to dequeue the packet from.

As an Endpoint developer,  one should ensure to support group concept to avoid locks.

Command Descriptor rings:


So far we have talked about packet transmit and receive descriptor rings and groups.  Command rings are also required to pass commands to the Endpoint and for end point to respond back.  There are many cases where command rings are required.  For example,  Ethernet endpoint might be required to know information from host such as

• MAC addresses
• Multicast Addresses
• Local IP addresses
• PHY configuration.
• Any other offload specific information.

Unlike packet descriptor rings, here response is expected.  Hence the descriptor should have facility for host to provide both command and response buffers.  Endpoint is expected to act on the command and put the response in the response buffer.

As an endpoint developer,  one should provide facility to send commands and receive responses.