Multicore Networking applications - Mitigating the Performance bottlenecks

I had given this talk in 2010 Multicore Expo in San Jose.  It was in presentation document in concise form. I voiced the most of the details during my talk.  Many people requested me to provide details in written form.  I tried to give details here in this post.  I hope this post would give enough details on 'New techniques to improve software performance with increasing number of cores'.

Before going further, I would like to differentiate two kinds of applications - Packet processing applications and Stream processing applications.

Packet processing applications in my definition are the ones which  take  packet by packet, work on the packet and send out the same packet or packet with some minor modifications.  In packet processing applications,  there is one-to-one correspondence between input and output packets except for very small number of exceptions. One example where there is no one-to-one correspondence is when there is IP reassembly or fragmentation. Other example is when the packet is dropped by the application. Example applications in this category are:  IP forwarding, L2 Bridging,  Firewall/NAT,  Ipsec and even some portions of IDS/IPS.

Stream processing applications are the ones which may take packets or stream of data,  work on data and send out the data or  send out different packets. Most of  the TCP socket based proxy applications come under this category. Examples:  HTTP Proxy,  SMTP Proxy,  FTP Proxy etc..

This post tries to aid the programmers debugging the software to find out the performance bottlenecks in Multicore networking applications.

Always Ensure to do  flow/Session Parallelization 

Ensure that only one core is processing the session at any given time.  If multiple packets from the same session are being processed by more than one core at the same time,  then there would be requirement to ensure that Mutual exclusion on the session variables.  That would be very expensive.  Multicore SoCs actually aid you to do flow parallelization in packet processing applications.  Many Multicore SoCs support parsing the fields from the packets,  calculate hash on the software defined fields and distribute the packets across the multiple queues based on the hash value.  And then they provide provision for software threads to dequeue the packets from the queues.  These SoCs also provide provision to stop dequeue of packets from threads until the control of the queue is given up explicitly.  This ensures that a given flow is processed by only software thread at any time.

Many Multicore SoCs also have facility to bind the queues to the software threads and each software thread to the core.  If the number of flows are small, there is a possibility of cache being warmed with contexts due to previous packets. This reduces the data movement from DDR.   Also, many Multicore SoCs provide facility to stash the context as part of dequeue operation which reduces the cache thrashing issue even if  binding of the queues to the cores are not done. 

Flow parallelization not only eliminates the need for Mutexes, it also ensures that there is no packet mis-ordering in the flows.

Many stateful packet processing applications require not only flow parallelization, but also session parallelization.  Session typically consists of two flows - Client to Server traffic and Server to Client traffic.  It is possible that two packets from both the flows may be coming to the device and two separate software threads might be processing these packets. Stateful applications share many state variables across these two flows. Due to this, you may require mutual exclusion operation if both the packets are allowed to be processed at the same time.  Session Parallelization as described here would eliminate the need for mutual exclusion.  Unlike flow parallelization, session parallelization is not available in many Multicore SoCs for cases where the tuple values are different in both the flows and hence needs to be done in software.  Packet tuples are different when NAT is applied. Note that many Multicore SoCs enqueue the packet to the same queue if there is no NAT.  They are intelligent enough to generate the same hash value even though the tuples position get changed, that is, source IP in one flow would be destination IP in reverse flow and same is true for destination IP, Source Port and Destination Port.

Stream processing modules such as proxies would need to ensure that both client side and server side sockets are processed by the same software thread to ensure that there is no  Mutual exclusion operations requirement to  protect the sanctity of state variables.  Stream processing modules typically create many software threads - worker threads.  Master thread terminates the client side connections and handover the connection descriptor to one of the less loaded worker threads.  Worker thread is expected to create new connection to the server and do rest of the application processing.  Worker threads are typically implement FSM for processing multiple sessions. More often,  the number of worker threads would be same as number of cores dedicated for that application.  In cases where the threads need to block for some operations such as waiting for accelerator results, then more threads, in multiples of number of cores, would be created to take advantage of full power of accelerators.

Eliminate the Mutual Exclusion Operation while Searching for Session/Flow Context

This technique is also expected to ensure that there are no mutual exclusion operations in the packet path.  Any networking application do some search operations on the data structures to figure out the operations and other action to be done on the packet/data.  Upon the incoming packet/data,  search is done to get hold of session/flow context and then further packet processing happens based on the state variables in the session.  For example,  IP routing does search on the routing table to figure out the destination port, PMTU and other information for operations such as fragmentation, TTL decrement and packet transmit.  Similarly firewall/IPsec packet processing applications maintain the sessions in a easy to search data structures such as RB trees, hash lists etc..   Since the sessions are created or removed dynamically from these structures, it is necessary to protect the data structure while doing operations such as add/delete/search.  Mutual exclusion operations using spinlock,  futex, up/down are one way to do this.   RCU (Read-Copy-Update) is another method that can be used which eliminates the Mutex operation during search.   RCU operation is described in earlier post.  Please check that here and here.  RCU lock/unlock operations in many operating systems is very simple operation. Note that Mutex operations are still required for add/delete even in RCU based usage.

Eliminate Reference Counting 

One of the other bottlenecks in the Multicore programming is the need to keep the session safe from deletion while it is being used by other software threads. Traditionally this is achieved by doing 'reference counting'. Reference counting is used in two cases - During packet processing operation or  When neighbor module store the reference.

In the first case, reference count of the session context is incremented as part of the session lookup operation.  During packet processing, the session is referred many times to get hold of state variable values and to set the new values in the state variables of the session.  It is expected that if the session is deleted, it should not be freed until the current thread is done with its operation. Otherwise, it would corrupt some other memory if the session memory is freed and allocated to somebody else during packet processing.  To ensure that the session ownership is not given away, the reference count is checked as part of 'delete operation'.  If is is not zero, then the session is marked for deletion, but not freed until the reference count becomes zero.  If the value is zero, it indicates there is no reference to this session and the session gets freed. 
 
Since RCU operation postpones the delete operation until current processing cycles of all other threads,  reference counting becomes redundant.  Elimination of reference count not only helps in improving the performance, but also reduces the maintenance complexity. Note that reference counting operation requires atomic usage of count variable. Atomic operations are not inexpensive.

Second use case of reference count is when the neighbor modules store the reference (pointer) to the sessions in their session contexts. By eliminating the storage of pointer,  reference count usage can be eliminated.  This post helps you understand how this can be done.

Linux user space programs also can take advantage of RCUs. See this post for more details.

Use the Cache Effectively

Once the matching session is found upon incoming data/packet,  processing functionality uses many variables in the session. If these variables are together in a cache line,  any cache fill due to access of one variable result all other variable in the cache line available in the cache.  That is, Access to other variables will not result in going to DDR.  But all variables may not fill in one cache line. In those cases, it is necessary to group the related variables together to reduce going to DDR.

To effectively use instruction cache, always arrange your code with likely/unlikely compiler directives. Compilers will try to arrange the likely() code together. 

Reduce Cache Thrashing due to Statistics variables

Almost all networking applications update statistics variables.  Some variables are global and some of them are session context specific variables.  There are two types of statistics variables -  increment variables and add variables. Increment variables are typically used to maintain the count of packets.  Add variables are used to maintain the byte count.  Updating these variables require getting hold of current values and then add or increment operation.   If these variables are updated by multiple threads (with each thread running on a specific core), then every time an variable is updated,  cache information of this variable is no longer valid in other cores.  When one of other cores needs to do same operations,  it needs to get the current value first from the DDR and apply the operation.  In worst case scenario, where packets are going to round robin fashion to different software threads (hence cores), then the cache thrashing due to statistics variables would be very high and this would reduce the performance dramatically.

Always use 'per core/thread statistics counters'  whenever possible.  Please see this post for more details. 

Some Multicore SoCs provide special feature which also eliminates the need for 'per core' statistics maintenance.  These SoCs provide facility to allocate memory block for statistics.  These SoCs provide facility to fire the operation and forget about it.  Firing the operation involves the operation type (increment, decrement, add X or sub Y etc..) and memory address (32 bit or 64 bit).  SoCs internally do this operation without cache thrashing.  I suggest strongly to use this feature, if it is available in your SoC.

Use LRO/GRO facilities

Many networking applications' performance depends on the number of packets being processed than the number of bytes processed.  Examples: IP Forwarding,  Firewall/NAT and Ipsec with hardware acceleration.  So, reducing the number of packets processed becomes key in improving the performance.

LRO/GRO facilities provided by operating system in Ethernet drivers or by Multicore SoCs reduce the number of TCP packets, if multiple packets from the same TCP flow are pending to be processed.  Since TCP is byte oriented stream protocol, it does not matter whether or not the processing happens on packets.  Please see this post for more information on LRO feature in Linux operating system.  If it is supported by your operating system or Multcore SoC, always make use of it.

Process Multiple Packets together


Each packet processing module does set of operations on the packets/data - such as Search,  Process and  Pkt out.  If the packet is going through multiple modules, there are many C functions get called.  Each invocation of C function has its own overhead such as pushing the variables in the stack, initializing some local variables etc..    By bunching multiple packets of same flow together can reduce search/pkt out overhead and overhead associated with the C functions.


Some Multicore SoCs provide facility to coalesce packets together on per queue basis with coalescing parameters -  Packet threshold and time threshold.  Queue does not let the target thread to dequeue until one of the conditions reached - either number of packets in the queue exceed the packet threshold parameter or if no packet was dequeued for time 'time threshold'.   If this facility is available, ensure that your software dequeues multiple packets together and processes them together. 

Yet times, there is no one-to-one correspondence between queues and sessions.  In that case, one might ask that search overhead can't be reduced as there is no guarantee that the packets in the same queue belong to the same session.  Though it is correct, it might still have some improvements due to cache warming if there are more than one packet belonging to same session in the bunch.

As a software developer,  it would be required to strive for one-to-one correspondence between queues and sessions. This can be done easily among the modules running in software.  Some Multicore SoCs provide queues for not only to access hardware blocks, but also for inter-module communication.  Software can take advantage of this to create one-to-one mapping between queues and destination module's sessions.

It is true that when the packets are being read from the Ethernet controllers, there is no way to ensure that a queue only holds packets of one session as the queue selection happens based on the hash value of packet fields.  Two different sessions may fall into same queue.  In those cases,  as mentioned above you might not see improvement from 'serach' functionality, but you would still see improvements due to less number of invocations of C functions.

Many Multicore SoCs also have functionality to take multiple packets together for acceleration and for sending the packets out. This also will reduce the number of invocation to acceleration functions and for sending packets out.  If this facility is available in your Multicore SoCs,  take advantage of it. 

Eliminate usage of software queues

Some Multicore applications need to send the packets/data/control-data to other modules.  If multiple threads send the data to the queue, then there is a need for mutual exclusion to protect these data structure queues.

Many Multicore SoCs provide queues for software usage.  These queues would eliminate the need for software queues and hence eliminate the mutual exclusion problem, there by improving performance.   Some Multicore SoCs also provide facility to group multiple queues together into a queue group which allows sending and receiving applications to enqueue priority items and dequeue based on priority.  These queues can be used even among different processes or virtual machines as long as shared memory is used for items that get enqueued and dequeued.  Some Multicore SoCs even went a step further to provide 'copy' feature which avoids shared memory and there by providing good isolation. This feature makes a copy of these items from source process to internal managed memory by Multicore SoCs and copy to the destination process memory as part of dequeue operation.

Always use this feature if it is available in your Multicore SoC.

Eliminate the usage of Software Free pools 

Networking applications use free pools of memory blocks for memory management.  These free pools are used to allocate/free session contexts, buffers etc..   Many software threads would require these facilities at different times. Software typically maintains the memory pools on per core basis to avoid mutual exclusion operations on per allocation basis.  Since there is a possibility of asymmetric usage of pools by different threads, yet times there is a possibility of memory allocation failures even though there are free memory blocks in other threads' pools.   To avoid this, software does complex operations during these scenarios of moving memory blocks from one pool to another through global queues.   Many Multicore SoCs provide 'free pool' functionality in hardware.  Allocation and free can be done by any thread at any time without mutual exclusion operations. Use this facility whenever it is available.  It saves some core cycles.  More than that is provides efficient usage of memory blocks.

Use Multicore SoC acceleration features to improve performance

There are many acceleration features that are available in Multicore SoCs.  Try to take advantage of them.  I classify acceleration functions in Multicore SoCs into three buckets -  Ingress In-flow acceleration,  In-flight acceleration and Egress in-flow acceleration.

Ingress In-flow acceleration:  Acceleration functions that are done by Multicore SoCs in the hardware on the packets before they are handed over to software are called Ingress In-flow accelerations.  Some of the features, I am aware, in Multicore SoCs are:

• Parsing of Packet fields :  Some Multicore SoCs parse the headers and make those fields available to the software along with the packet.  Software needing the fields can eliminate the parsing of fields.   These SoCs also provide facility for software to choose the fields to be made available along with the packet.  They also provide facilities for software to create parser to extract fields from proprietary headers or from non pre-defined headers.  Try to take advantage of this feature.
• Distribution of packets across threads:  This is basic feature required in Multicore environments.  Packets needs to be distributed to different software threads.  Many Multicore SoCs also ensure that packets belonging to one flow go to one software thread at any time to ensure that packets will not get mis-ordered within a flow.  As described above,  multiple queues would be used by hardware to place the packets.  Selection of queue is based on hash value calculated on the set of software programmable fields.  As a software developer, take advantage of this feature rather than implementing the distribution in software.
• Packet Integrity checks & Processing offloads:  Many Multicore SoCs do quite a bit of integrity checks on the packet as  listed below.  Ensure that your software don't do them again to save some core cycles.
• IP Checksum verification.
• TCP, UDP checksum verification.
• Ensuring that the headers are there in full.
• Ensure that size of packet is not less than the size indicated in the headers.
• Invalid field values.
• IPsec inbound processing.
• Reassembly of fragments
• LRO/GRO as described above.
• Packet coalescing as described above.
• Many more.
• Policing :  This feature can police the traffic and reduce the amount of traffic that is seen by the software.  If your software requires policing of some particular traffic to stop cores from getting overwhelmed, this feature can be used rather than doing it in the lowest layers of software.
• Congestion Management :  This feature ensures that the number of buffers used up by the hardware won't go up exponentially. Without this feature, cores may not find buffers to send out the packets if all buffers are used up by the receiving hardware. This situation typically happens when the core is doing lot of processing and hence slow in dequeuing while lot more packets are coming in.  Many Multicore SoCs also have facility to generate pause frames in case of congestion. 

Egress In-flow acceleration:   Acceleration functions that are done in the hardware once the packets are handed over to it by software to send the packets out are called Egress in-flow acceleration functions.  Some of the Egress in-flow acceleration functions are given below.  If these are available, take advantage of them in your software as these can reduce significant number of cycles in the core.

• Shaping and Scheduling :  High priority packets are sent out within the shaped bandwidth.  Many Multicore SoCs provide facilities to program the effective bandwidth. These SoCs shape the traffic with this bandwidth. Packets which are queued to it by software would be scheduled based on the priority of the packets.  Some SoCs even provide multiple scheduling algorithms and provide facility for software to choose the algorithm on per physical or logical port.  Some SoCs even provide hierarchical scheduling and shaping.  Take advantage of this in your software if you require shaping and scheduling of the traffic.
• Checksum Generation for IP and TCP/UDP transport packets :  Checksum generation, especially for locally generated TCP and UDP packets is very expensive.   Use the facilities provided by hardware.  
• Ipsec Outbound processing :  Some Multicore SoCs provide this functionality in hardware.  If you require Ipsec processing,  use this facility to save large number of cycles on per packet basis.
• TCP Segmentation and IP Fragmentation :  Some Multicore SoCs provide this functionality.  TCP segmentation performs well for local generated packets. Use this functionality to get best out of your Multicore.

In-flight Acceleration :   Acceleration functions provided by hardware that can be used during packet processing are called In-flight acceleration functions.  Crypto,  Crypto with protocol offload,  Pattern Matching,  XML acceleration are some of the acceleration functions that come in this category.  Here the packet/data for acceleration is handed over to the hardware acceleration functions by software. Software reads the results at later time when the results are ready.  Take advantage of these feature in your software wherever they are available . Some Multicore SoCs differentiate themselves by doing lot more in the acceleration functions.  For example,  some Multicore SoCs do protocol offload along with crypto such as Ipsec ESP,  SSL record layer protocol , SRTP and MACSec offloads which do beyond crypto offload.

I see many times people asking me a question on how to use the acceleration functions.  I had detailed this long time back here. Please see the details there and there.

Software Directed Ingress In-flow accelerations:

As described before, Ingress in-flow acceleration is applied before the packets are given to the software. Packets that are received on integrated Etherent controllers go through this acceleration.  But many times this acceleration is required from software too.  Take the example of Ipsec, SSL or any tunneling protocol.  Once the software processes these packets, that is once it gets hold of inner packets,  software would like ingress in-flow acceleration to be applied on the inner packets for distribution across cores and other acceleration functions.  To facilitate these kinds of scenarios, some Multicore SoCs provide concept of 'offline port' which allows software to reserve the offline ports and send the traffic for ingress in-flow acceleration.  Some software features that can take advantage of this feature are:

• Tunneled traffic as described above to let the inner packets to go through the ingress in-flow acceleration,
• IP reassembled traffic - Once the fragments are reassembled, it would have all 5-tuples which can be used to distribute the traffic through offline port.
• L2 encapsulated packets - Such as IP packet from PPP, FR etc..
• Ethernet controllers on PCI and Traffic from Wireless interfaces :  Here the traffic might need to be read by the software and Ingress in-flow acceleration might not have been implemented for these features. Software after getting hold of packets can be directed to in-flow acceleration functions through offline ports.

Use Multicore core features wherever they are available

Multicore SoCs from different vendors have different core architecture. Some Multicore SoCs are based on power pc, some based on MIPS core and Intel Multicore is based on x86 processors. Multicore SoC vendors provide different features to improve performance of Multicore applications.  Whenever they are available, software should make use of them to get the best performance out of cores.  Some of the features that I am aware of are listed below.

Single Instruction & Multiple Data instructions (SIMD)

Multicore SoCs from Freescale and Intel have this block in their cores.   This feature in the cores allows software do a given operation on the multiple data elements.  This kind of parallelism is called 'Data level parallelism'.  'Add' operation in typical cores is performance either on 32 bit or at the most 64 bit operands.  Current generation of SIMD do this operation on 128 bit operands. They also provide flexibility to do multiple 16 bit, 32 bit add operations on different parts of data simultaneously.  SIMD greatly helps in operations which involve arithmetic, bit, copy, compare operations on large amount of data.  Any operation that is done in a loop can be accelerated using SIMD.   In networking world,  SIMD is helpful in following cases:

• Memory compare, copy,  clear operations.
• String compare, copy, tokenization and other string operations.
• WFQ scheduling of QoS, where multiple queues need to be checked to figure out which queues need to be scheduled based on sequence number property of queues.  If the sequence numbers are arranged in array form, then SIMD can be used very effectively.
• Crypto operations.
• Big Number arithmetic which is useful in RSA, DSA and DH operations.
• XML Parsing and schema validations.
• Search algorithms -  Accelerating compare operation to find matching entry from collision elements in a hash list.
• Check-sum verification and generation:  In some cases Ingress and Egress In-flow accelerations can't be used to verify and generate the checksums.  One example is,  TCP and UDP packets that come in IPsec tunnel.   Since the packets are in encrypted form,  ingress and egress accelerators will not be able to verify and generate checksums in inner packets.  Even packets that get encapsulated in tunnels will not be able to take advantage of Ingress & Egress in-flow accelerations.  Checksum verifications and generations need to be done in software by cores.  SIMD would help in those cases tremendously.
• CRC verification and generation:  These algorithms are not very expensive to have In-flight acceleration and not inexpensive for core to do.  SIMD in these cases help as it does not involve any architecture changes to the software and still get lot better performance over the cores which don't have SIMD.

Normally SIMD based cores give at least 50% more performance improvement for typical workloads.  So, as a software developer, figure out the ones that can be improved using SIMD and modify the code to improve performance of your application.

Speculative Hardware Data Prefetching & Software Directed Prefetching

This feature fetches the next cache line worth of data from the current memory access in the hopes that software would use next memory line.  Many core technologies provide control on enabling and disabling this at run time.  Software can take advantage of this while doing memory copy, set and compare operations.  Any data is arranged in linear fashion in the memory (such as arrays) can get good boost of performance with this feature. Note that, if this feature is not used selectively and carefully, it might even give degradation in performance. Be careful in using this feature.

Many cores also provide special instruction to warm the cache given a memory address. Software developers know the kind of processing (next module) and many times next module session context is also known. In those cases, software can be developed such a way that next module session is prefetched while packet processing happens in current module.  When the next module gets the control of the packet, it already has session context in the cache which avoids getting it DDR in serial fashion.  My experience is that using software directed prefetching gives very good results.  This also ensures that the performance does not go down even with large number of sessions.

Some Multicore SoCs provide support for Cache warming on the incoming packets.  As part of making packets ready for the software, these SoCs warm the cache with some part of packet content,  annotation data containing parsed fields and software issued context data.   When the software dequeues the packet, most of the information required to process the packet of the module that is getting hold of packet is in place in the cache, thereby, avoiding on-demand DDR access.  Software can program its context on per queue basis.  Note that, this feature is useful for the first module that receives the packet.  Actually that is good enough as this module can prefetch the next module context while the packet is being processed in the current module.  As long as each modules does this, there is no performance degradation even with high capacity. 

As described before,  hardware queues may not have one-to-one correspondence with the receiving module session contexts.  A queue might be having packets for multiple session contexts. Many times, software maintains the sessions in the hash table with large number of hash buckets.  All collision sessions are arranged in linked list or RB tree.  Software can ensure that there are as many queues as number of hash buckets and program the first collision element in the queue.  If the matching context is not same as the one that was programmed, then one might not get the full benefit of cache warming by the hardware. But if there are 4 collision elements and the traffic across these four are same, cache warming would come in handy 25% of the time. Some software developers might even store the collision elements in an array and program the array to the queue.

Software directed prefetch works very well as long as there is one-to-one correspondence between current module session context and next module session context.  In this case,  current module session context can cache the reference to the next module session context and use this to do prefetch operation.  This scheme also work fine if next module context is super set of multiple current module contexts.  But it does not work well if the next module context is finer granular.  Example:  Ipsec SA transfer packets from  multiple firewall/NAT sessions.  In this case, 'Software Directed Ingress In-flow acceleration' method can be used to direct the hardware to send the packet to next module.  This method not only provides cache warming, but also distributes the processing to multiple cores.


Hardware Page Table walk:

Some cores provide nested hardware page table walk to find out the physical address given the virtual address.  This is really useful for user space applications in Linux kind of operating systems.  Hardware page table walk feature is expected to be taken care by operating system vendors.  But unfortunately many OS vendors are not taking advantage of this feature.  As a software developer, if your Multicore SoC provide this feature, don't forget to ask your OS vendors to take advantage of this.  This will ensure that your performance does not go down when you move your application from Bare-metal environment (where the TLB are fixed and there is no page walk required) to Linux user space.

I hope it helps.

Fastpath Ipsec implementations - Developer integration tips on Inbound policy check

Basic purpose of Ipsec fast path implementations is to reduce the IPsec processing load on the main processing cores.  Since most of the Ipsec processing is same across different kinds of packets, offloading of this processing to hardware makes sense.

There are companies today who provide fast path implementations - either as software component or as an add-on card such as PCIe card that goes onto PCI slot of main processing unit such as x86 based mother board.

Software based fast path implementations are becoming quite popular in Multicore processing environment.  Fast path is run on some cores and rest of the cores are used for other applications.

Ipsec fast path implementations typically work as follows:

• Fast path typically owns the Ethernet and other L2 ports. That is, all packets come to the fast path plane first.
• If there is enough state information to process the packet,  fast path implementations act on the packets without involving normal path running in main cores. Packet might even get transmitted out after working on the packet.  If the packet requires some other application processing that is not present in the fast path,  then the packet is handed over to normal path processing unit.  In case of Ipsec fast path,  decrypted packets are  given to the normal path in inbound direction. In outbound direction,  it does Ipsec processing before packet is sent out.  

Basic purpose of fast path is to save CPU cycles so that it can do some other processing.
All fast path implementation from different vendors are not created equal.

In this post, I specifically would like to concentrate on 'Inbound Policy Check'.  Some fast path implementation skip this check. Reasons given by vendors of the fast path implementation typically is that this is done for performance reasons.  Some people believe that it can be done without any implications on security too.   Unfortunately, that is not true.


What is inbound policy check?

Inbound policy check ensures that the decapsulated IPsec packets used the SA that was formed for this traffic.   And also it ensures that the inbound policy rules allow this traffic to come through.


What are the issues if the inbound policy check is not done?

I can think of two issues -  DoS attack &  Allowing the traffic that is supposed to be denied (Access control violation).

DoS attack:

Let us assume that  a corporation gateway  has two tunnels to two different partners - Partner1 and Partner2. Without inbound policy check, it is possible for partner1 to interfere with the sessions/traffic between corporate gateway and partner2. That is, partner1 can create denial of service attack on partner2 traffic.  Even though I have taken the example of partners, this kind of attack is possible among IPsec remote users.

Let us assume this scenario:

10.1.10.0/24-----------SGW---------Internet-----------PSGW1-----------10.1.11.0/24
                                                             |
                                                             |-----------------PSGW2------------10.1.12.0/24


SGW:  Security Gateway of a corporation - It is protecting network 10.1.10.0/24
PSGW1:  Partner 1 Security Gateway - Its LAN is 10.1.11.0/24
PSGW2:  Parnter 2 Security Gateway. Its LAN is 10.1.12.0/24

There are two security tunnels from SGW - One to PSGW1 and another to PSGW2. Let us call them Tunnel1 and Tunnel2 respectively.

Tunnel1 is negotiated to secure traffic between 10.1.10/24 to 10.1.11.0/24.  Tunnel2 is negotiated to secure traffic between 10.1.10/24 and 10.1.12.0/24.   Let us also assume that the Tunnel1 SPI at the SGW is  SPI1 and the Tunnel2 SPI at the SGW is SPI2.  

It is expected that any packets coming from PSGW1 are expected to have SPI1 in its ESP header and inner IP packet SIP address  be one of addresses in 10.1.11.0/24 and DIP address  be one of 10.1.10.0/24.  Similarly it is expected that any packet coming from PSGW2 is expected to have SPI2 in its ESP header and inner IP packet SIP address be one of addresses in 10.1.12.0/24 and DIP address  be one of 10.1.10.0/24. 

Now to the attack scenario:

If PSGW1 network sends the inner packets whose IP addresses is other than 10.1.11.0/24 and 10.1.10.0/24 , then the SGW is expected to drop those packets.  SGW can only drop the traffic only if SGW does the inbound policy check.  If PSGW1 is allowed to send any inner packets, then it is possible that PSGW1 and its network can misuse this by sending packets with inner packets with IPs of PSGW2 LAN and SGW LAN.  Since it sends the traffic on the right SA using its own SPI,  SGW IPsec packet processing will happen smoothly.  If no other check is done, this traffic can go to SGW LAN.   Based on type of traffic, different attacks are possible.  Some of the attacks that are possible are:

• If attacker at PSGW1 guesses the TCP ports of some long lived sessions between PSGW2 network and SGW network, it can send the RST packets or ICMP Error messages to terminate the connections.
• Attacker at PSGW1 can send ICMP Echo message to SGW1 LAN network multicast IP address with SIP as PSGW2 LAN machine.  Replies from all machines in the SGW1 LAN go to PSGW2 victim machine and overwhelm the machine.

If SGW checks the inbound policy after the IPsec decapsulation is done using inner IP packet,  then it would have found that the SA used for the matching inbound policy is not same as the SA used to decapsulate the packet.  Whenver there is any mismatch, it is expected to drop the packet.  Due to this, no malicious traffic would have gone to the SGW LAN in above scenario.  Also, by logging these events,  administartor can find out the misbehaving peer security gateway and take appropriate out-of-band action.

Access Control Violation:

This is one more problem that can be faced if inbound policy check is not done.
Many Ipsec normal path implementations provide facility for administartors to add multiple rules with different actions to the secuirty policy database (SPD).   Rules normally have 5-tuple selectors in ranges/subnets/exact IP addresses for source and destination and ranges/exact values for UDP/TCP ports.  Actions can be one of 'Bypass', 'Discard' and 'Apply'.   Rules are arranged in a ordered list.  During packet processing,  rule search is done.  Rule search is stopped upon match.  Action specified on the matching rule is taken.  If the action is 'Bypass', then the packet is forwarded without any Ipsec processing.  'Discard' action indicates the packet is to be dropped.  Apply action indication that Ipsec processing is to be applied.  Normally administartors configure the rules with respect to outbound traffic.  Inbound policy rules are created automatically by the system from outbound policy rules by reversing the selectors - That is SIP becomes DIP and vice versa. Similary SP becomes DP and vice versa.

Now let us look at the possible access policy violation with following example:

Let us take these two policy rules in outbound list:

Rule 1:  SIP:  10.1.10.0/24  DIP 10.1.11.0/24  Protocol UDP   Action :  Discard
Rule 2:  SIP:  10.1.10.0/24  DIP 10.1.11.0/24  All Protocols   Action : Apply.

Inbound policy rule list would look like this:

Rule 1:  SIP:  10.1.11.0/24  DIP:  10.1.10.0/24  Protocol:  UDP   Action : Discard
Rule 2:  SIP:  10.1.11.0/24  DIP: 10.1.10.0/24  Protoco:  All   Action:  Apply.

Administartor creates the rules in above fashion to indicate that to discard any UDP traffic between the networks, but allow everything else by securing the traffic.

Assume that above policy rules are created in SGW1.

10.1.10.0/24----------SGW1----------Internet-----------SGW2-------10.1.11.0/24

When a TCP packet is sent from the SGW1 LAN and SGW2 LAN, then second rule gets matched and SA is created to allow traffic 10.1.10/24 to/from 10.1.11.0/24 for all protocols.  If SGW2 either misconfigured or intentionally sends UDP traffic in the SGW1-SGW2 tunnel,  then SGW1 is expected to drop the packet even if it successfully decrypts and decapsualtes the packet.  This can only happen if the SGW1 does the inbound policy check on inner IP packet.

If SGW1 does not do any inbound policy check, UDP traffic would have been passed to the 10.1.10.0/24 network thereby violating the access rules configured by the administartor.

I hope I could give good reasoning on why inbound policy check is required.  Some fast path implementation don't do this.   So, as a development integration engineer, please ensure that not only your implementation, but also fast path implementation does all the checks that are required.

Comments?

Fragmentation before Ipsec Encapsulation - Redside fragmentation and more use cases

I am finding more and more benefits of doing 'red side' fragmentation in Ipsec worl.

One use case is given here:  With red side fragmentation,  any  switches/routers in between security gateways of tunnels don't see  fragmented packets.  Due to this the cases, where some service providers' routers give less priority to the fragmented packets, don't arise. 

Second use case is given here :  When majority of  the traffic goes on IPsec tunnels, LAG can't distribute the traffic across ports since the result traffic  has same 5-tuple information. As described in the post,  multiple IPsec tunnels normally get created with forceful NAT-T.   All packets that are coming out of Ipsec Engine are expected to have 5 tuple information. If fragmentation is done after Encap, then the LAG would see some packets without 5-tuples.  This results to uneven distribution.  Hence redside fragmentation is done to ensure that LAG sees 5-tuples for all the packets.

Third use case:  Avoid mis-ordering of the packets:

There could be packets which are big and small in the traffic.  Big packets may get fragmented after Ipsec encapsulation if the result size exceeds the MTU of outgoing interface.  Small packets may not get fragmented even after encapsulation.

Gateway receiving the Ipsec packets is expected to process them in order. Due to fragmented packets, this may not happen.  Let us say that,  gateway received 1st fragment of 1st packet,  2nd full packet and 2nd and also final fragment of 1st packet in that order.  It is expected that the gateway processes them  in the same order.  But since 1st packet waits for 2nd fragment,  2nd full packet would be processed. Gateways don't stop the full packets getting processed as it may not know whether or not second  fragment of the 1st packet is going to come in and also when it is going to come in. 

So, this leads to packet mis-order.

This can be avoided if there are no fragments. Solution : Red side fragmentation.

Comments?

Look-aside acceleration & Application Usage scenarios

Performance and flexibility are two different factors that play role on how applications use look-aside accelerators.  As described in  post,  applications use accelerators in synchronous or asynchronous fashion.  In this post, I would give my view of different types of applications and their usage of look-aside accelerators.

I would assume that all applications are running in Linux user space.  I also would assume in this post that all applications are using HW accelerators by memory mapping the registers in the user space.  Based on these assumption, I could categorize applications into these types:

• Per_packet processing applications with Dedicated core to the User Process and HW Polling Mode :  In this type, application runs in the user process. A core or set of cores are dedicated to the process, that is, these cores are not used for anything else other than executing this process.  Since core is dedicated, it can wait for the events on some HW interface until some event is ready to be processed.  In this mode,  it is expected that Multicore hardware provides single interface to wait for the events.  Application wait in a loop forever for the events. When the event is ready, it takes action based on the type of event and then come back to wait for new events.  This type of application is more suitable for per-packet processing applications such as IP forwarding, Firewall/NAT,  IPsec, MACSec,  SRTP etc..   
• Per-packet processing applications would use look-aside accelerators in asynchronous fashion. Incoming packets from Ethernet or other L2 interfaces and the results from the look-aside accelerators are given through the common HW interface.   
• Typical flow would be some thing like - When the incoming packet is ready on Ethernet port,  polling function returns with 'New packet' event.   New packet is processed by the user space and at one time decides that it needs to be sent to the HW accelerator, sends it to HW accelerator and then come back to poll again.   HW accelerator at some time returns the result through same HW interface.  When polling function returns with 'Acceleration result' event, user process processes the result and may send the packet out onto some other Ethernet port.   It is possible that more packets would have been processed by the user process before the acceleration result is returned for previous packets.  Due to this asynchronous nature, cores are utilized well and system throughput would be very good.
• IPsec, MACSec, SRTP uses Crypto algorithms in asynchronous fashion.
• PPP and IPsec IPCOMP use compression/decompression accelerators in asynchronous fashion.
• Some portion of DPI use Pattern Matching acceleration in asynchronous fashion.
• Per-Packet processing application with Non-Dedicated core to the user process & SW polling mode:  This is similar to above type 'Dedicated core to the user process and HW polling mode'.   In this type,  core(s) are not dedicated to the user process.  Hence HW polling is not used as this would make core not relinquish the control as often for doing other operations.  SW polling is used, typically using ePoll() call.   In this mode, interrupts are using UIO facilities provided by Linux.  When the interrupt is raised whenever the packet is ready or accelerator result is ready. UIO wakes up the epoll() call in the user space.  When the ePoll() returns, it reads the event from HW interface and it executes different function based on event type.  
• All per-packet processing applications such as IPsec, SRTP, MACSec, Firewall/NAT can also work in this fashion.
• IPsec, MACSec, SRTP uses Crypto algorithms in asynchronous fashion.
• PPP and IPsec IPCOMP use compression/decompression accelerators in asynchronous fashion.
• Some portion of DPI use Pattern Matching acceleration in asynchronous fashion.
• Stream Based applications :  Stream based applications are normally work at high level away from packet reception and transmission.  For example,  Proxies/Servers work on BSD sockets - The data which they receive is the TCP data, not the individual packets.  Crypto file system is another kind of stream application, where it works on the data, not on the packet.  These applications collect data from several packets. Some times this data gets transformed such as packet data gets decoded into some other form.   HW accelerators would be used on top of this data.  In almost all cases the HW accelerators are used in synchronous fashion.  In this type of applications ,  synchronous mode is used in two ways -  Waiting for the result in a tight loop without relinquishing the control and waiting for the result in a loop by yielding to Operating system.   First sub-mode (tight loop mode) is used when the HW acceleration function takes very less time and second mode (yield mode) is used when the acceleration function takes long. 
• Public Key acceleration such as RSA sign/verify, RSA encrypt/decrypt, DH operations and DSA sign/verify work in yield mode as these operations take significant number of cycles.  Applications that require this acceleration are:  IKEv1/v2,  SSL/TLS based applications,  EAP Server etc..
• Symmetric Cryptography such as AES & different modes,  Hashing algorithms, PRF Algorithms would be used in tight loop submode as these operations take less cycles.  Note that  Yielding might take anywhere between 20000 cycles to 200,000 cycles based on number of other ready processes and that is not acceptable latency for these operations.  Applications based on SSL/TLS,  IKEv1/v2,  EAP Server etc..
• I would put compression/decompression HW accelerator usage in slightly different sub-mode.  Compression/Decompression works in this fashion for each context.
• Software thread issues the operation.
• Immediately reads if there is anything pending result (based on previous operations). Note that the thread is not waiting for the result.
• Works on the result if available
• And above steps happen in a loop until there is no input data.
• At the end,  it waits (in yield mode) until the all the result is returned by the accelerator.
• Application that can use compression accelerators:  HTTP Proxy, HTTP Server,  Crypto FS, WAN optimization etc..

 Any comments?

LAG, Load Rebalancing & QoS Shaping

LAG feature exposes only one L2 interface to the IP stack for each LAG instance.  It hides all the links in the LAG instance underneath it.  It sounds good in the sense that IP stack & other applications are completely transparent with respect to number of links that are being added and removed.

Though many applications and IP stack don't care about the LAG, links and its properities,  one application QoS would need to worry about the link properities - specifically its bandwidth (shaping bandwidth).  In ideal world,  even QoS does not need to worry about links and its properties. As we all know, to ensure that there is mis-ordering of the packets in a given conversation,  distributor function of the LAG module distributes the conversations across the links, not the packets. If there are large number of conversations compared to the links, there is always possibility of equal distribution of the traffic across the links. But when there are small number of conversations, which by the way not so uncommon, then there is a possibility of unequal distribution with respect to the traffic.  That is, there could be more traffic in some conversations compared to others. If high traffic conversations go to few links, then there is unequal distribution.  Let me cover QoS and changes required in QoS to work with LAG.

Load Rebalancing:

LAG distributor normally implements the concept of 'Load Rebalancing'.  Load rebalncing happens in three cases.

•  When LAG observes that there is unequal distribution.
•  When new link is added to the LAG instance.
•  When existing link is removed, disabled or broken.

Though new link and removal of existing link to/from the LAG instance is not the focus of this article, let me just give a gist of  issues that need to be taken care.  Packet mis-order issue must be taken care well.  When the new link is added,  if hash distribution is changed immediately, some of the existing conversations might be balanced to other links. If it is done arbitrarily, then there is a possibility of packets being received by collector in out-of-order for brief amount of time.  To make sure that new link is used effectively,  there are two methods can be used. Both can be used togehter though.

•  New conversations would use new hash distribution.
•  Current conversations can be put onto other links only if the conversation is idle for X milliseconds - Time at which we know that packets would have been collected by the collector.

When link is no longer active,  then packet mis-ordering is no longer a big issue.  The traffic has to flow and new distribution can take effective immediately and distirbute the conversations that belong to the old port to existing ports immediately.

Now on to redistribution due to unequal utilitization of links:

Redistribution can be done in two ways - Changing the hash algorithm or fields to be used in hash algorithm.  Second is to some how increase the number of conversations.  Second method of increasing the conversations would work only in cases where tunnels (such as Ipsec) are conversations.  By increasing the number of tunnels,  there is a good possibility of increasing the distribution. Actual 5-tuple flows are sent on multiple tunnels. See this link here on how LAG & IPsec work together.

Changing the hash algorithm or adding/removing fields to the hash algorithm would have mis-order issues.  In some deployments mis-order once in a while is okay.  In those cases, this methoed can be used. To use this method, rebalancing should not happen very frequently.  Typically following mehtod is used - If a link utilization is more than X% (Typically 5 to 10% - configurable parameter) away from the average usage of the trunk, then it is candidate for redistribution.  Stop doing redistribution for configurable amount of seconds to ensure that there are no frequent redistributions.

QoS:

Typically QoS shaping & scheduling function runs on top of L2 interfaces.  Trunk link would be given the shaping bandwidth. Shaping is typically implemented using token bucket algorithm.  Whenever there are tokens available,  scheduling function is invoked.  Scheduling function selects the next packet and sends the packet out.

LAG instance which is actiing as L2 interface has the shaping bandwidth which is sum of all the links. If the scheudling decision is taken purely based on the LAG trunk bandwidth, there is a possibility that scheduled packet would get dropped if the packet goes on link which is already completely utilized. This happens when there is uneven traffic in the convesations.  Rebalancing helps, but it takes some time rebalance the traffic. Hence QoS shaping and scheduling function should not only consider the LAG instance bandwidth, but also the individual link bandwidth while making scheduling decision. By considering both,  at least the paket from the high traffic conversation is not scheduled and resides still in the queue, there by avoiding packet drop.
At the same time, it is not good to under utilize other links. Scheduling, in this case, can move to other traffic that fall in other under-utilized links.

LAG is important feature, but it has its own challenges.  IPsec and Qos implementations need to work with LAG properly to utilize LAG effectively.

Comments?

eNodeB and IPsec

eNodeB secures the traffic over the IPsec tunnels to the Serving Gateway (SGW) over backhaul network.  Also, eNB creates many tunnels to peer eNBs for X2 and handover traffic.  Though all features related to Ipsec are valid in eNB scenarios too,  some features are worth mentioning in eNB context.

LAG and IPsec:

Please this link here to understand the issues and solutions related to LAG and Ipsec in general. This scenario is very much valid for eNB to SGW communication. Note that traffic from all GTP tunnels in non-handoff scenario go between eNB and SGW on one or few (when DSCP based tunnels) Ipsec tunnels.  When LAG is used between eNB and LAG,  similar issue of not utilizing more than one link would arise.  Both the solutions suggested in earlier article are valid in this scenario too.  In cases where it is difficult to get multiple public IP addresses to the LAG link, then scenario 2 - forceful NAT is only option I can think of.


Capabilities expected in eNB and SGW:

Using LAG effectively requires many tunnels.  It is good to have 1 + (number of links - 1 ) * 32 Ipsec tunnels for good distribution across links.  User traffic, in this case GTP traffic should be balanced across these IPsec tunnels.

Typically there are two GTP traffic tunnels for each cell phone user - One is typically created for Data traffic and another for voice traffic.  Without LAG, normally two Ipsec tunnels are created - One for data traffic coming/going  from/to  all the cell users and another for voice traffic for all voice traffic coming/going to cell users.  GTP traffic is distributed across these two Ipsec tunnels based on DSCP value.

Now, we have lot more Ipsec tunnels.  There should be additional logic in eNB and SGW which distributes the GTP traffic across these multiple Ipsec tunnels.  This logic should distribute the traffic from a given conversation to one Ipsec tunnel.  Each GTP tunnel traffic can be viewed as one conversation.   That is, GTP tunnels are distributed across the Ipsec tunnels.   One way is to look at the TEID (Tunnel Endpoint ID) and use hash on TEID to distribute the traffic across Ipsec tunnels.

Ipsec implementation on eNB and SGW should have capability to create multiple tunnels - number of tunnels to be created should be configurable.  eNB and SGW implementations also should have capability to bring up the tunnels on demand basis too. That is,  they should ensure that these number of tunnels are UP and running as long as there is traffic.  Note that all ipsec tunnel negotiation would have same selectors and ipsec implementations should not be having intelligence to remove old tunnels with same selectors.

If persistent feature is selected on SPD rules, then the implementations should ensure that all ipsec tunnels are UP and running all the time.

As described in the earlier article, it is necessary that Ipsec implementation have capability of doing 'Red side fragmentation' so that the LAG always sees UDP header in every packet which is required for its distribution.

DSCP based Ipsec tunnels:

LTE uses packet based network for  voice, streaming,  interactive and non-interactive data.  Hence it is necessary that Ipsec tunnel honor this priority to ensure that voice and other real-time traffic is given priority.  If both data and voice is sent on the same tunnel, there is a possibility of traffic getting dropped due to sequence number checks as part of anti-replay checks in the receiver.  Even though packets are marked with increasing sequence number in both data and voice traffic and encapsulated in the Ipsec tunnel,  due to local QoS and QoS in intermediate devices, voice traffic may be sent before the data traffic - that is traffic is reordered.  As you know, receiver window right edge moves with newer sequence number.  Due to this, some data packets which have lower sequence number get dropped if they are less than the lower edge.  To avoid unnecessary drops, there are two methods used - Increase the Anti-replay window size  or use different SAs (tunnels) for different kinds of traffic. Second method is normally used.

Due to this feature and above LAG feature, number of tunnels that need to be created in eNB and SGW can go up significantly.  Hence both eNB and SGW should have enough memory and computation power to handle multiple tunnels.

Persistent tunnels

To reduce the latency of initial traffic, it is necessary to have this feature. Tunnels are UP and running all the time even when there is no traffic.  This feature is good if the links are always-on and if there is no cost based on the traffic amount.


DSCP and ECN Copy settings:


Ipsec implementations expected to copy DSCP and ECN bits from inner header to outer header.  Inner header DSCP value is set by applications and this should be continued even when the traffic is tunneled.  This will ensure that the nodes between the eNB and SGW will also give QoS treatment.  Hence it is necessary to copy the DSCP bits from inner header to outer header.

ECN bits indicate the congestion to the peer so that peer entity can inform the source entity to apply the congestion.  TCP protocol has a way to inform the source entity when the receiver gets the IP packets with CE (congestion experienced) bit on in ECN bits of IP header.  Intermediate nodes, including eNB and SGW should honor this by copying from inner header to outer header while encapsulating and copy from outer header to inner header while decapsulation.

Peer IP address adoption 

eNodeB gets the IP address from backhaul provider dynamically. It is possible that IP address might be changed by the provider while traffic is going on.  Ipsec tunnels are expected to be UP and running even if the IP address changes on the gateways.   This internet draft discusses the mechanism to adopt the peer gateway address change.  This feature is expected to be present to ensure that voice traffic does not observe too much of jitter and latency.  Note that tunnel establishment takes hundreds of milliseconds as it involves IKE negotiation of the keys.  This can introduce jitter and latency when the voice traffic is going on at that time. Implementation must implement this draft to eliminate jitter and latency issues when the IP address changes on the remote gateway.

IP Fragmentation and Reassembly

Normally many vendors of eNB give performance with respect to UDP traffic without involving IP fragmentation and reassembly. Even though this gives one data point, this may be misleading to customers. Most of the traffic in Internet today is TCP and same is true in LTE world too.  TCP MSS is chosen such as way that TCP data packet with IP and TCP header would be MTU size.  When the traffic undergoes IPsec encapsulation, it is almost certain that packets would need to be fragmented as it exceeds the MTU of the link.  Though DF bit facility is available for end points to know the Path MTU,  this feature is not used in Ipv4 end points today.  Since packets are fragmented,  reassembly is required on other side.

This feature is implemented in Ipsec implementations, but I am afraid that many implementations, though have very good IPsec performance on non-fragmented packets, they are not optimized when fragmentation and reassembly is required.  Customers need to watch out for this as significant amount of traffic would be fragmented and reassembled.

IPv6 Support

Ipv6 is fast becoming choice of service providers and LTE core network.  Hence Ipv6 support is expected in ipsec implementations.  eNB and SGW must support both Ipv4 and IPv6 tunnels.  Also, they should be able to send IPv4 and Ipv6 traffic on IPv4/Ipv6 tunnels.

TFC

Traffic Flow Confidentiality feature is normally given less importance.  I was told that LTE networks require this feature be implemented in Ipsec tunnels so that the anybody who gets hold of backhaul network traffic will not be able to guess the type of traffic that is going on in the tunnels based on traffic characteristics such as - frequency of traffic,  size of packets,  distribution of packets  etc.. 


AES-GCM

AES-GCM combines both encryption and integrity in one algorithm. Hence it is called combined algorithm.  This algorithms is supposed to be 2x faster than AES-CBC algorithm. Also it is supposed to have half of latency of AES-CBC.  Hence it is good for both performance and also for latency which is required for voice traffic.  Hence it is becoming popular algorithm in eNB and SGW.

Validation Engineers and customers I believe should look for above features in eNB and SGW.

Comments?

Link Aggregation and Ipsec

Link Aggregation is also called Ethernet Trunking and Bonding.  This feature is described in 802.3ad. In 2008, this was rolled into 802.1AX group.

What is LAG:

LAG combines multiple Ethernet Ports and exposes it as one link to the upper layers in the system.
It is Layer 2 concept.  Only trunk port would be assigned with IP addresses.  Links in the trunk don't have any Layer 3 information.  Only one MAC address would be used for the trunk.  Individual MAC addresses of the links don't appear in any communication other than control protocol (Marker protocol).

How does it work?

LAG contains two components - Distributor and Collector.
Distributor distributes the outgoing traffic across the links that constitute the trunk.  Collector collects the data in inbound direction coming from different links and tunnels through the trunk port to rest of the system.
LAG assumes that all links in the trunk are full duplex and point to point.
Simplest distribution is to distribute the packet by packet across the links based on weight configured on the links.  But there could be packet mis-ordering issues.


What are some critical items to be taken care by the Distributor:

Packet mis-ordering is one of the issues distributor would face if it distributes the traffic blindly on per packet basis.   To avoid mis-ordering,  distributors are expected to send all the packets of given flow (conversation) sent on the same link.  First generation distributors used to apply hash on source and destination IP and select the link based on hash value.  Though this ensures that the traffic belonging to one conversation goes on the same link, but the distribution may not be symmetric for some workloads.  Second generation distributors go one step beyond and apply the hash on TCP/UDP ports too.  This would give better distribution, but it may have some mis-ordering problem if the outbound packets are fragments.  Only first fragment would have transport header and other fragments don't have transport header. In those cases, there is a chance that non-initial fragments go on some other link and lead to mis-ordering.  Since fragments are not very common, some deployment accepts some level of mis-ordering to get the better utilization of the links.

Collector and Packet mis-ordering:  

To ensure that packets are delivered in order, collector should ensure to send the packets up in order it receives on any given link. There is no order to be maintained on packets coming in across links.  Collector also should ensure that it does not starve any link while receiving the packets.

Ipsec and LAG:

In some deployments, traffic is always encrypted via Ipsec and sent to the remote office.  If one tunnel  is used to send the traffic, all the traffic going from the local network to remote gateway contains same source, destination IP addresses.  In case UDP traversal is applied, it would have same source and destination ports.  Even if there are multiple links in LAG, distributor hash will fall onto only one link and other links would not be used. 

Same is true with Reverse traffic.  Also note that Links in Aggregation group are with local ISP.  Remote gateway under same admin control will not know about local Link aggregation.  That is, incoming traffic balancing across the links is in the hands of service provider.  It is okay to assume that most of 802.3ad distributors are configured with to use IP addresses and in some cases even ports.

Since distributors only know the IP addresses and ports of the packet, links would be utilized well in both directions if there are large number of flows with different IP addresses and Ports. 


Solutions

There are two solutions I can think of.

Solution 1:  Using Multiple IP address on the trunk link. 

Create as many tunnels as number of IP addresses on the trunk link with remote gateway.  As described in the link here,  some software should distribute the flows across these IPsec tunnels.  Since each tunnel now has different source IP address in the outer IP header, LAG distribution hash may fall into different links and thereby utilizing the bandwidth well in outbound direction.  One should ensure that, there are many local IP addresses to ensure that all links are used and also all links are used evenly.

Reverse traffic also would be balanced fine as service provider switch also would see different IP addresses (Destination IP).

Getting or assigning  multiple public IP addresses to the trunk may not be possible.  In which case, second solution can be used. But second solution would have some packet overheads.

Solution 2 :   Usage forceful NAT-T

Even though NAT is not detected,  there are ways to force UDP traversal. That is, ESP packets are sent with in UDP payload.  Create as many tunnels as necessary for good distribution at the LAG level.  Each tunnel would have different UDP source port.  Some software in the device is expected to balance the traffic across these tunnels. LAG would distribute the tunnels across multiple links of the LAG.  Reverse traffic also would be balanced on different links due to different destination port values of tunnels.  Since it is expected that LAG distributor look at the transport header for distribution, it is necessary that there are no fragments.  So, it is mandatory that tunnels are configured with redside fragmentation. This will ensure that fragmentation is done before Ipsec encapsulation.


In both the solutions,  both remote and local gateways should have some logic to

• know that multiple tunnels are created for same selectors for distributing the flows.
• know how to distribute different conversations to different tunnels.

It requires more tunnel capacity in devices.  This should not be a problem as modern devices has good horse power and enough memory to create some more tunnels with peer gateway.


Comments?

Web Application firewalls, IPS & Network Anti Virus - Fixing the performance issues

Security professionals know that the intrusion and malware detection is now beyond looking at stream of packets.  Detection require

• SSL Decryption - Many client side attacks are increasingly hidden in HTTPS connections.( Check this out )
• Extracting data from the packets (Example:  HTML, Javascript,  Different types of files to detect attacks embedded in the data)  ( See this )
• Decoding the data (Such as UTF-8, UTF-16, De-compression etc..)
• Emulation of data if the data is script (such as Javascript) to counter evasion techniques used by attackers.
• Comparing with known signatures or codelets OR doing some kind of heuristics

This kind of analysis is not possible with stream based firewalls, IPS and AV.  These require collection of data.  They all require proxies. If some IPS/AV vendor says that they do detection without reassembling data and collecting the data, then as end user you will not be wrong to say that they either miss lot of intrusions or give too many false positives.

Computational power to do above is very high.  It is not surprising to see just less than 10Mbps of IPS, AV combined performance in devices which give 1Gbps of firewall, Ipsec throughput.  I hear stories of customer disappointments when they turn on IPS and/or AV functionality in security devices.

Network security analysts advising companies to enable full functionality even for traffic originated from trusted networks.  It should not be surprising anybody as trusted network boundary is reducing  due to mobility of machines in trusted network. That is, machines are moving from trusted to untrusted and vice versa. Examples :  laptops, ipads etc..   These  machines may get infected when they are in untrusted network and may get infect other machines in trusted network when they are brought into corporate networks. That is the reason, now full protection is being enabled on the security devices. 

HTTP is singlemost protocol that occupies majority of network bandwidth in many organizations.  HTTP is also interactive protocol. Any performance issue also  impact the user experience.   Solving HTTP performance problem not only improves user experience, but also would increase the performance of overall system.

Techniques that can be used to improve the performance of HTTP Anti-malware and IPS analysis are given below. End users might look for following features.

• Avoid doing duplicate IPS and Anti-Malware checks :  It is very common tha same resource is requested by same/multiple users in the orgnaization via HTTP.  Nework device once AV and IPS check is done on the resource should avoid doing the check again.  This requires caching of AV and IPS analysis and using it when the same resource is requested at later time.   Ofcourse, it should have life time so that it checks for AV/IPS if the content of the resource is changed.  Life time can be equal to the Expiry time of the resource which comes along with the HTTP response headers.  If possible,  this system also can do caching of the response which avoids even going to origin server, there by saving the WAN bandwidth too.  I believe that AV/IPS devices would have HTTP Caching moving forward.
• Auto blacklisting of URIs :  Malware may be served with dynamic content. In which case, above mechanism of caching does not work.  More often, Malware is served using the same URI.  If the data downloaded from a URI contains the malware, that URI can be blacklisted if malware is detected multiple times.   If the request comes to the same URI at later time,  request can be denied without even senidng the request to the origin server.  Always make sure that the newer blacklisted entries are honored by the device.
• TCP and SSL offload:  Proxies can benefit greatly if some other entity such as intelligent PCI-e takes care of TCP/IP stack and SSL offload.   
• Implement proxies as per my earlier post.
• Usage of Multicore processors and distributing the load across multiple cores.  Selection of Multicore processor depends on several factors such as cost, number of cores (performance), acceleration features etc..  But here I am only covering the features.  Features that would help in processing are :
• Processing power - Higher the processing power, better the performance would be.
• Cache Size matters:  Unlike typical firewall/Ipsec processing, amount of code that gets executed in doing AV/IPS analysis is lot higher. Higher sized L1 and L2/L3 caches would store more instructions and goes to DDR less often.  Cache for storing data is also important.
• Acceleration hardware -
• Compression/Decompression Accelerator:  To take care of decomperssing the compressed files coming in the HTTP response.
• SIMD (Single instruction Multiple Data) based hardware to do acceleration of
• Memory /String operations - Copy, Set
• Checksum, CRC operations
• HTML and URL decoding operations.
• and many more...

Hope it helps..

LSN A+P Q&A

In A+P mode, what is the need for CPE to send NATTed packets to the provider box (PRR or LSN)?
This is mainly for security reasons.  You can't assume that CPE are all good citizens. Here, one public IP address is assigned to multple CPE devices with different port range for source port NAT.  If the rogue CPE or misbehaving CPE uses ports for source port NAT beyond assigned values, it can distrub the traffic of some other CPE.  To ensure that this does not happen, all packets are sent to the centralized box in provider network.  Provider network validates the source ports of outgoing connections of the CPE and transmits out onto the Internet only if source port is one of the ports assigned to the CPE box. Provider box (PRR) maitains the table with IPv6 address (which is used as source IP of tunnel by the CPE ) and the allocated ports.  This table would be referred by the PRR to validate the source ports of the connections.

Can rogue CPE mount DOS attack on other CPE if the rogue CPE knows the IPv6 address and the port allocations of the victim CPE?

In theory, it is possible.  Rogue CPE may not be able to get hold of the traffic of victim CPE, but it can mount DOS attack.  Rogue CPE can disable some portions of victim CPE communication by using all ports.   I believe it is necessary that each CPE authenticates itself to the PRR before sending the traffic over the IPv6 tunnel.  It is not there today, but it is natural expect in my view.

One proposal I saw can make this DoS attack difficult.  If the PRR assigns the random ports to the CPE rather than fixed range, then it makes it difficult for rogue CPE to determine the  exact pots used by other CPEs.

Havind said that, a rogue CPE can, it it wants, mount DOS attack such a way that it can use up the ports of different CPE devices if it knows the IPv6 addresses.  So, it is good to have authenticaiton support bulit into creating the IPv6 tunnel.

I personally belive that IPsec IPv6 tunnels with IKEv2 would be the right fit. It increases the processing requirements, but it is secure.  IPsec allows transport of IPv4 packets over IPv6 tunnel in addition to IPv6 packets in IPv6 tunnel.  It provides not only authentication of the CPE device, but also secures the traffic between CPE device and the PRR.  It also can retain the QoS characterstics of the differnet packets between CPE and LSN.  If data security is not required, ESP with Authentication can be used which is less expensive from computation processing is concerned on the LSN device.

In 3GPP,  Femto Access Points (CPE devices) already have IPsec tunnel with IKEv2 to SeGW (ePDG).  Same tunnel can be used to transport IPv4 packets to the ePDG, if ePDG is equipped with the PRR & LSN functionality.

Are there any CPE devices or Smart phones supporting LSN & A+P functionality?

I don't have this information.  I saw some internet postings from Android based phone vendors asking about LSN.  So, I beleive it is in vendors radar, but not sure whether anybody has solutions in the market. As I indicated in my last post, Cisco has LSN functionality on service provider side  in their portfolio.  A10networks also has some service provider boxes supporting LSN. 

IPv6 - finally? Is LSN becoming transition technology?

Some articles in recent past is indicating that IPv6 is being adopted by vendors and service providers.

This article here indicates that Cisco has LSN solution for service providers.  As described here, LSN provides transition from IPv4 to IPv6 gradually.  I like the phrase used by Cisco in the article on IPv6 transition -  preserve, prepare and prosper.  Preserve the Ipv4 infrastructure while preparing for IPv4/IPv6 transition (LSN) and then move to Prosper phase with complete Ipv6.

This article here says that comcast is going to test LSN in Q3 2010.  This is very good news that IPv6 is being adopted by service providers.

As I indicated in my previous article LSN itself is not good enough for some subscribers. A+P addition in my view is very much required.

I am not seeing much activity in CPE device market supporting A+P functionality in it.  I guess it is natural that it would happen soon.  I also see some internet drafts which add some additional options in DHCPv6 and PPPv6 to assign PRR address.  I am still yet to see any internet drafts on allocate/de-allocate public IPv4+port pairs dynamically.  If anybody has come across any specifications on this, please comment.

Intel McAfee Deal - My two cents

Many analysts and Intel is saying that more security would be baked into the hardware for growing number of internet-connected devices.

It is certainly true that security would become concern for users using different kinds of gadgets and other internet-connected personal and Enterprise devices.  Some of the devices run on battery.  As we all know, kind of attacks are becoming sophisticated day by day.  Sophistication of attacks translates to more computing power to detect these attacks and stop them.  Would this kind of computing power available on gadgets such as smartphones and other mobile devices?  Even if it is available, how much battery power it takes.  I guess it would be very high.

I believe that computational intensive attack detection and prevention of many of these mobile internet-connected devices would provided in the cloud or in Enterprise networks.  There would be some kind of agent and root-of-trust which is required in mobile devices, but majority of the traffic analysis for attack detection and protection of these devices would happen in the cloud. By the way, Intel vPro already has root-of-trust technology built into it and can be used in mobile devices.

My guess is that Intel is going to target the cloud/Enterprise market for doing traffic analysis for attack detection and secure the mobile devices. This would be growing market for sure in coming years.  Intel might add some features to the server chips to scale the security computation.  If you really see, McAfee is more popular in doing heavy weight Anit-Malware and Intrusion analysis for laptops. Similar technology is needed for mobiles without running them on mobiles.  In recent past,  McAfee bought some Mobile related companies such as Trust Digital,  WaveSecure etc..  This gives me an impression that Intel along with McAfee might come out with Enterprise device product which does not only 'Mobile device provisioning', 'Lost Mobile Security', but also secure the traffic between mobiles and Enterprise networks.  That is,  a device which is complete mobile portal to Enterprise from provisioning, reporting and security.  McAfee technologies IPS, Anti Malware, Anti-phishing, Anti-spam can be used in these Mobile Portal devices to ensure that the traffic going to/from the Mobile device is secure and sanitized. TrustDigital technology would be used to provision the mobile devices and for generation of reports on usage of mobile devices.  This combination can be powerful.

What are the features Intel might add to the hardware?  Majority of attack detection now requires proxies.  Almost all the Anti-malware companies implement proxies to terminate the connection,  decrypt the traffic if it is encrypted, do de-archive and de-compression if required,  analyze the traffic for attacks, sanitize the traffic,  create new connection to the other end point and send the traffic.  There are heavy computational intensive low hanging items that can be taken care in the hardware.  Many of the computational intensive items are already taken care either by Intel or by the third party hardware cards such as compression/decompression,  Encryption/Authentication,  XML/Xpath analysis etc..  Majority of proxy handling is still done in the cores. I believe this part of the proxy management and TCP/SSL Offload would be required to get the maximum power of the hardware.  That is, software running in the cores would only see the application protocol (HTTP, SMTP etc..) data, not the individual packets.

Let us see how things turn out in future...

Large Scale NAT with DS-Lite & A+P

Dual Stack Lite, Address plus Port assignment to CPE devices by ISPs are two most important mechanisms being adopted by ISPs to provide connectivity to IPv4 internet to smart phones, CPE devices in Residential markets and Femto CPEs.

Dual Stack Lite and A+P mechanisms are being done to ensure that IPv6 transition is smooth as Internet becomes Ipv6 addressable over time.

ISPs are facing shortage of public IPv4 addresses. Demand was increased with popularity of Internet in general and in particular the explosion growth of smart phones. Many ISPs are no longer in position to provide the dynamic public IP address to the CPE devices and smart phones. Note that CPE devices and smart phones have become always-on. So dynamic IP address is really becoming static.

Until world moves to the Ipv6,  only way is sharing of IPv4 address across multiple subscribers.

Many mobile service providers are only giving the private IP address to the smart phones and ISPs  This trend is continuing even for CPE devices with its explosive growth.   ISPs maintain mega NAT boxes which translate the traffic from CPE and smart phones with certain number of public IP addresses.  CPE devices already do their own NAT between IP addresses it assigns to local machines in the LAN with the IP address provided by the ISP.  Due to this, there is double NAT.  Though it works in majority of cases, there are some limitations which could be problematic for end customers, hence to the ISP business.

•  Connectivity could be lost if dynamic  private IP address is assigned by the ISP is part of the private subnet the CPE is configured to assign to the local machines.
• Though not a big concern immediately,  Bigger ISPs might have customer more than the private IP address space. If one goes with 10.x.x.x network, then ISPs might provide address to 2^24 subscribers.  With smart phones in the range of 120M in 2012,  it is a possibility that ISPs might even don't have many unique private IP addresses to assign.
• Applications requiring special ALG will not work if both NAT devices (CPE as well as Carrier NAT) don't support the ALGs.   ISP Carrier NAT box may not entertain proprietary ALGs or may not have many ALGs.
• Two internal machines that need to communicate among themselves (peer-to-peer) applications may not work in double NAT scenarios (hair pin scenarios).
• Many peer-to-peer applications expect same IP and Port to be used for SNAT even though destination machines are different.  If not supported by Carrier NAT, many peer-to-peer applications may not work.

Large Scale NAT  (LSN) solves some of above limitations by doing NAT at only one place. In this model,  CPE is given the IPv6 address on its WAN interface.  If IPv6 machines are communicating with IPv6 destinations, then there is no IPv4 involved and it works fine.  If IPv4 machine in private network is communicating with public Ipv4 network in the Internet, then these packets are tunneled to the LSN box sitting in the provider network.  Ipv6 is used to tunnel IPv4 packets between CPE and LSN.  LNSN box in Provide network does the NAPT.   LSN eliminates double NAT.  LSN also takes care of overlapping private IP addresses among multiple subscribers by keeping the IPv6 tunnel end point address as one of the identification parameter to map the NAT entry.  It still has problems related to ALGs.  That is, if the LSN does not support ALGs for some applications, then these applications never work.  Also, LSN need to be high performing box with respect to throughput,  latency and jitter.  Though this can be solved by multiple LSN boxes, but ALG problems are too big for adoption of this technology. Also, if any application needs to be hosted, this becomes tough as port forwarding is controlled at Carrier NAT rather than the CPE gateway as we all normally accustomed to.

A+P (Address Plus Port) specifications provides the flexibility of doing NAT with CPE.  In this case, multiple CPEs are given with same public IP address, but with different ports.  CPE NAT is only expected to use assigned ports for source port NAT.  CPE can decide not to do NAT for some connections and in which case LSN in provider network would do the NAT.  Based on different people experience, I believe only few ports are necessary by the CPE due to feature called 'Dense NAT'.  That is, same source port can be used across multiple connections as long as 5-tuple is different across the connections on the external realm. Some web sites using AJAX may make multiple connections at the same time. I believe there are some sites which make almost 60+ connections at a time.  128 port range is good enough for many cases.  What it means is that, even without LSN, same IP address can be used across 128 subscribers assuming each subscriber requires 128 ports.  With A+P alone, the ISP can increase his customer base 128 times with the public IPv4 addresses the ISP has.  Since NAT is done at the CPE, all the facilities as in current CPE boxes are possible. It can have port forwarding feature,  each CPE can have its own ALGs and each one can have port triggering feature.  Having said that, it has its own limitations - IPsec without UDP NAT traversal does not work.  ICMP Echo Request/Reply would need to be taken care little bit more carefully as it does not have port concept.

I believe Comcast is in advanced stages of deploying LSN. Many ISPs would be requiring to install some kind of solution very soon.  In my view, the solution would be combination of LSN and A+P.  ISPs will differentiate subscribers using following subscriptions.

• Subscribers requiring static public IP address.
• Subscribers hosting any servers on standard ports which are expected to be reached via their own Domain name.
• Subscribers requiring dynamic public IP address.
• Subscribers hosting servers on standard ports with DDNS.
• Subscribers with shared public IP address and dedicated ports for SNAT.
• Subscribers hosting games or hosting servers on non-standard ports.
• Subscribers with shared public IP address and shared ports (LSN).
• Subscribers just requiring outbound access.

CPE devices and Smart phones require some support if they intend to take advantage of LSN and A+P.  When I was going through the specifications,  I had one doubt on why the CPE NATted packets need to go through the IPv6 tunnel to the PRR (Port Range Router).  I think the reason why they need to go is to ensure that the CPE device really did NAT with the IP address and Ports that were allocated to it by the PRR. It is required to ensure that CPE devices are behaving well and does not damage the connectivity of other CPE devices.

Let us see the kind of changes required in CPE devices.

Features expected in CPE:

• CPE must support IPv6 addressing on its WAN Interfaces. 
• Learning of LSN IPv6 address using DHCP extensions, PPP extensions Or CPE should have facility to provide static configuration.
• In case CPE supports A+P, it should also learn the public IPv4 address and Port range(s)  from DHCP, PPP or via local configuration.  In some cases, more port ranges also can be requested dynamically when the ports are getting exhausted.  If it does not require any port ranges, it should be able to free them back to the PRR.
• CPE must be able to provide IPv6 addresses to the IPv6 capable hosts in its internal network.
• CPE must be able to provide IPv4 private addressing to the local hosts in its internal network.
• CPE must be able to tunnel packets from private IP hosts to the LSN in provider network.
• In case of A+P, it should have intelligence to figure out which connecting to be NATTed at the CPE and which one are allowed to be done at the LSN.
• CPE optionally also can support providing the A+P to the local hosts if they are A+P aware. In which case, CPE also acts as local PRR.

Features expected in PRR:

• It should be able to do Address+Port Management via signaling protocols (DHCP, PPP or web based management).
• It should be figure out the packets that needs to go through LSN and if so, send those packets to LSN.
• It should ensure that the packets are NATted by CPE with its delegated addresses and ports. If not, it should discard the packets.
• It should provide facilities WCCP protocol for security checks (AV, AS,  IPS etc..).
• It should terminate IPv6 tunnels and should be prepared to make Ipv6 tunnels to the LSN.
• It should be scalable:  Good algorithms to 
• Search tunnel for incoming packets from the CPE.
• Search Address+Port based entries for packets coming from Internet to identify the CPE device and hence the tunnel.

Features expected in LSN:

• It should be able to terminate large number of tunnels.
• It should be able to maintain large number of NAT entries.
• It should be able to work with CPE devices having same private IP address space.
• It should be able to stateful failover if device fails.
• It should support popular application ALGs such as:
• FTP, RTSP, SIP, H.323. MGCP,  PPTP, L2TP and more..

I hope it helps.