Fastpath Ipsec implementations - Developer integration tips on Inbound policy check

Basic purpose of Ipsec fast path implementations is to reduce the IPsec processing load on the main processing cores.  Since most of the Ipsec processing is same across different kinds of packets, offloading of this processing to hardware makes sense.

There are companies today who provide fast path implementations - either as software component or as an add-on card such as PCIe card that goes onto PCI slot of main processing unit such as x86 based mother board.

Software based fast path implementations are becoming quite popular in Multicore processing environment.  Fast path is run on some cores and rest of the cores are used for other applications.

Ipsec fast path implementations typically work as follows:

• Fast path typically owns the Ethernet and other L2 ports. That is, all packets come to the fast path plane first.
• If there is enough state information to process the packet,  fast path implementations act on the packets without involving normal path running in main cores. Packet might even get transmitted out after working on the packet.  If the packet requires some other application processing that is not present in the fast path,  then the packet is handed over to normal path processing unit.  In case of Ipsec fast path,  decrypted packets are  given to the normal path in inbound direction. In outbound direction,  it does Ipsec processing before packet is sent out.  

Basic purpose of fast path is to save CPU cycles so that it can do some other processing.
All fast path implementation from different vendors are not created equal.

In this post, I specifically would like to concentrate on 'Inbound Policy Check'.  Some fast path implementation skip this check. Reasons given by vendors of the fast path implementation typically is that this is done for performance reasons.  Some people believe that it can be done without any implications on security too.   Unfortunately, that is not true.


What is inbound policy check?

Inbound policy check ensures that the decapsulated IPsec packets used the SA that was formed for this traffic.   And also it ensures that the inbound policy rules allow this traffic to come through.


What are the issues if the inbound policy check is not done?

I can think of two issues -  DoS attack &  Allowing the traffic that is supposed to be denied (Access control violation).

DoS attack:

Let us assume that  a corporation gateway  has two tunnels to two different partners - Partner1 and Partner2. Without inbound policy check, it is possible for partner1 to interfere with the sessions/traffic between corporate gateway and partner2. That is, partner1 can create denial of service attack on partner2 traffic.  Even though I have taken the example of partners, this kind of attack is possible among IPsec remote users.

Let us assume this scenario:

10.1.10.0/24-----------SGW---------Internet-----------PSGW1-----------10.1.11.0/24
                                                             |
                                                             |-----------------PSGW2------------10.1.12.0/24


SGW:  Security Gateway of a corporation - It is protecting network 10.1.10.0/24
PSGW1:  Partner 1 Security Gateway - Its LAN is 10.1.11.0/24
PSGW2:  Parnter 2 Security Gateway. Its LAN is 10.1.12.0/24

There are two security tunnels from SGW - One to PSGW1 and another to PSGW2. Let us call them Tunnel1 and Tunnel2 respectively.

Tunnel1 is negotiated to secure traffic between 10.1.10/24 to 10.1.11.0/24.  Tunnel2 is negotiated to secure traffic between 10.1.10/24 and 10.1.12.0/24.   Let us also assume that the Tunnel1 SPI at the SGW is  SPI1 and the Tunnel2 SPI at the SGW is SPI2.  

It is expected that any packets coming from PSGW1 are expected to have SPI1 in its ESP header and inner IP packet SIP address  be one of addresses in 10.1.11.0/24 and DIP address  be one of 10.1.10.0/24.  Similarly it is expected that any packet coming from PSGW2 is expected to have SPI2 in its ESP header and inner IP packet SIP address be one of addresses in 10.1.12.0/24 and DIP address  be one of 10.1.10.0/24. 

Now to the attack scenario:

If PSGW1 network sends the inner packets whose IP addresses is other than 10.1.11.0/24 and 10.1.10.0/24 , then the SGW is expected to drop those packets.  SGW can only drop the traffic only if SGW does the inbound policy check.  If PSGW1 is allowed to send any inner packets, then it is possible that PSGW1 and its network can misuse this by sending packets with inner packets with IPs of PSGW2 LAN and SGW LAN.  Since it sends the traffic on the right SA using its own SPI,  SGW IPsec packet processing will happen smoothly.  If no other check is done, this traffic can go to SGW LAN.   Based on type of traffic, different attacks are possible.  Some of the attacks that are possible are:

• If attacker at PSGW1 guesses the TCP ports of some long lived sessions between PSGW2 network and SGW network, it can send the RST packets or ICMP Error messages to terminate the connections.
• Attacker at PSGW1 can send ICMP Echo message to SGW1 LAN network multicast IP address with SIP as PSGW2 LAN machine.  Replies from all machines in the SGW1 LAN go to PSGW2 victim machine and overwhelm the machine.

If SGW checks the inbound policy after the IPsec decapsulation is done using inner IP packet,  then it would have found that the SA used for the matching inbound policy is not same as the SA used to decapsulate the packet.  Whenver there is any mismatch, it is expected to drop the packet.  Due to this, no malicious traffic would have gone to the SGW LAN in above scenario.  Also, by logging these events,  administartor can find out the misbehaving peer security gateway and take appropriate out-of-band action.

Access Control Violation:

This is one more problem that can be faced if inbound policy check is not done.
Many Ipsec normal path implementations provide facility for administartors to add multiple rules with different actions to the secuirty policy database (SPD).   Rules normally have 5-tuple selectors in ranges/subnets/exact IP addresses for source and destination and ranges/exact values for UDP/TCP ports.  Actions can be one of 'Bypass', 'Discard' and 'Apply'.   Rules are arranged in a ordered list.  During packet processing,  rule search is done.  Rule search is stopped upon match.  Action specified on the matching rule is taken.  If the action is 'Bypass', then the packet is forwarded without any Ipsec processing.  'Discard' action indicates the packet is to be dropped.  Apply action indication that Ipsec processing is to be applied.  Normally administartors configure the rules with respect to outbound traffic.  Inbound policy rules are created automatically by the system from outbound policy rules by reversing the selectors - That is SIP becomes DIP and vice versa. Similary SP becomes DP and vice versa.

Now let us look at the possible access policy violation with following example:

Let us take these two policy rules in outbound list:

Rule 1:  SIP:  10.1.10.0/24  DIP 10.1.11.0/24  Protocol UDP   Action :  Discard
Rule 2:  SIP:  10.1.10.0/24  DIP 10.1.11.0/24  All Protocols   Action : Apply.

Inbound policy rule list would look like this:

Rule 1:  SIP:  10.1.11.0/24  DIP:  10.1.10.0/24  Protocol:  UDP   Action : Discard
Rule 2:  SIP:  10.1.11.0/24  DIP: 10.1.10.0/24  Protoco:  All   Action:  Apply.

Administartor creates the rules in above fashion to indicate that to discard any UDP traffic between the networks, but allow everything else by securing the traffic.

Assume that above policy rules are created in SGW1.

10.1.10.0/24----------SGW1----------Internet-----------SGW2-------10.1.11.0/24

When a TCP packet is sent from the SGW1 LAN and SGW2 LAN, then second rule gets matched and SA is created to allow traffic 10.1.10/24 to/from 10.1.11.0/24 for all protocols.  If SGW2 either misconfigured or intentionally sends UDP traffic in the SGW1-SGW2 tunnel,  then SGW1 is expected to drop the packet even if it successfully decrypts and decapsualtes the packet.  This can only happen if the SGW1 does the inbound policy check on inner IP packet.

If SGW1 does not do any inbound policy check, UDP traffic would have been passed to the 10.1.10.0/24 network thereby violating the access rules configured by the administartor.

I hope I could give good reasoning on why inbound policy check is required.  Some fast path implementation don't do this.   So, as a development integration engineer, please ensure that not only your implementation, but also fast path implementation does all the checks that are required.

Comments?

eNodeB and IPsec

eNodeB secures the traffic over the IPsec tunnels to the Serving Gateway (SGW) over backhaul network.  Also, eNB creates many tunnels to peer eNBs for X2 and handover traffic.  Though all features related to Ipsec are valid in eNB scenarios too,  some features are worth mentioning in eNB context.

LAG and IPsec:

Please this link here to understand the issues and solutions related to LAG and Ipsec in general. This scenario is very much valid for eNB to SGW communication. Note that traffic from all GTP tunnels in non-handoff scenario go between eNB and SGW on one or few (when DSCP based tunnels) Ipsec tunnels.  When LAG is used between eNB and LAG,  similar issue of not utilizing more than one link would arise.  Both the solutions suggested in earlier article are valid in this scenario too.  In cases where it is difficult to get multiple public IP addresses to the LAG link, then scenario 2 - forceful NAT is only option I can think of.


Capabilities expected in eNB and SGW:

Using LAG effectively requires many tunnels.  It is good to have 1 + (number of links - 1 ) * 32 Ipsec tunnels for good distribution across links.  User traffic, in this case GTP traffic should be balanced across these IPsec tunnels.

Typically there are two GTP traffic tunnels for each cell phone user - One is typically created for Data traffic and another for voice traffic.  Without LAG, normally two Ipsec tunnels are created - One for data traffic coming/going  from/to  all the cell users and another for voice traffic for all voice traffic coming/going to cell users.  GTP traffic is distributed across these two Ipsec tunnels based on DSCP value.

Now, we have lot more Ipsec tunnels.  There should be additional logic in eNB and SGW which distributes the GTP traffic across these multiple Ipsec tunnels.  This logic should distribute the traffic from a given conversation to one Ipsec tunnel.  Each GTP tunnel traffic can be viewed as one conversation.   That is, GTP tunnels are distributed across the Ipsec tunnels.   One way is to look at the TEID (Tunnel Endpoint ID) and use hash on TEID to distribute the traffic across Ipsec tunnels.

Ipsec implementation on eNB and SGW should have capability to create multiple tunnels - number of tunnels to be created should be configurable.  eNB and SGW implementations also should have capability to bring up the tunnels on demand basis too. That is,  they should ensure that these number of tunnels are UP and running as long as there is traffic.  Note that all ipsec tunnel negotiation would have same selectors and ipsec implementations should not be having intelligence to remove old tunnels with same selectors.

If persistent feature is selected on SPD rules, then the implementations should ensure that all ipsec tunnels are UP and running all the time.

As described in the earlier article, it is necessary that Ipsec implementation have capability of doing 'Red side fragmentation' so that the LAG always sees UDP header in every packet which is required for its distribution.

DSCP based Ipsec tunnels:

LTE uses packet based network for  voice, streaming,  interactive and non-interactive data.  Hence it is necessary that Ipsec tunnel honor this priority to ensure that voice and other real-time traffic is given priority.  If both data and voice is sent on the same tunnel, there is a possibility of traffic getting dropped due to sequence number checks as part of anti-replay checks in the receiver.  Even though packets are marked with increasing sequence number in both data and voice traffic and encapsulated in the Ipsec tunnel,  due to local QoS and QoS in intermediate devices, voice traffic may be sent before the data traffic - that is traffic is reordered.  As you know, receiver window right edge moves with newer sequence number.  Due to this, some data packets which have lower sequence number get dropped if they are less than the lower edge.  To avoid unnecessary drops, there are two methods used - Increase the Anti-replay window size  or use different SAs (tunnels) for different kinds of traffic. Second method is normally used.

Due to this feature and above LAG feature, number of tunnels that need to be created in eNB and SGW can go up significantly.  Hence both eNB and SGW should have enough memory and computation power to handle multiple tunnels.

Persistent tunnels

To reduce the latency of initial traffic, it is necessary to have this feature. Tunnels are UP and running all the time even when there is no traffic.  This feature is good if the links are always-on and if there is no cost based on the traffic amount.


DSCP and ECN Copy settings:


Ipsec implementations expected to copy DSCP and ECN bits from inner header to outer header.  Inner header DSCP value is set by applications and this should be continued even when the traffic is tunneled.  This will ensure that the nodes between the eNB and SGW will also give QoS treatment.  Hence it is necessary to copy the DSCP bits from inner header to outer header.

ECN bits indicate the congestion to the peer so that peer entity can inform the source entity to apply the congestion.  TCP protocol has a way to inform the source entity when the receiver gets the IP packets with CE (congestion experienced) bit on in ECN bits of IP header.  Intermediate nodes, including eNB and SGW should honor this by copying from inner header to outer header while encapsulating and copy from outer header to inner header while decapsulation.

Peer IP address adoption 

eNodeB gets the IP address from backhaul provider dynamically. It is possible that IP address might be changed by the provider while traffic is going on.  Ipsec tunnels are expected to be UP and running even if the IP address changes on the gateways.   This internet draft discusses the mechanism to adopt the peer gateway address change.  This feature is expected to be present to ensure that voice traffic does not observe too much of jitter and latency.  Note that tunnel establishment takes hundreds of milliseconds as it involves IKE negotiation of the keys.  This can introduce jitter and latency when the voice traffic is going on at that time. Implementation must implement this draft to eliminate jitter and latency issues when the IP address changes on the remote gateway.

IP Fragmentation and Reassembly

Normally many vendors of eNB give performance with respect to UDP traffic without involving IP fragmentation and reassembly. Even though this gives one data point, this may be misleading to customers. Most of the traffic in Internet today is TCP and same is true in LTE world too.  TCP MSS is chosen such as way that TCP data packet with IP and TCP header would be MTU size.  When the traffic undergoes IPsec encapsulation, it is almost certain that packets would need to be fragmented as it exceeds the MTU of the link.  Though DF bit facility is available for end points to know the Path MTU,  this feature is not used in Ipv4 end points today.  Since packets are fragmented,  reassembly is required on other side.

This feature is implemented in Ipsec implementations, but I am afraid that many implementations, though have very good IPsec performance on non-fragmented packets, they are not optimized when fragmentation and reassembly is required.  Customers need to watch out for this as significant amount of traffic would be fragmented and reassembled.

IPv6 Support

Ipv6 is fast becoming choice of service providers and LTE core network.  Hence Ipv6 support is expected in ipsec implementations.  eNB and SGW must support both Ipv4 and IPv6 tunnels.  Also, they should be able to send IPv4 and Ipv6 traffic on IPv4/Ipv6 tunnels.

TFC

Traffic Flow Confidentiality feature is normally given less importance.  I was told that LTE networks require this feature be implemented in Ipsec tunnels so that the anybody who gets hold of backhaul network traffic will not be able to guess the type of traffic that is going on in the tunnels based on traffic characteristics such as - frequency of traffic,  size of packets,  distribution of packets  etc.. 


AES-GCM

AES-GCM combines both encryption and integrity in one algorithm. Hence it is called combined algorithm.  This algorithms is supposed to be 2x faster than AES-CBC algorithm. Also it is supposed to have half of latency of AES-CBC.  Hence it is good for both performance and also for latency which is required for voice traffic.  Hence it is becoming popular algorithm in eNB and SGW.

Validation Engineers and customers I believe should look for above features in eNB and SGW.

Comments?

Network infrastructure application validation Engineers - Things to ask yourself

One of the challenges many Intelligent network infrastructure device vendors have is to ensure that their product withstands attacks on protocol processing logic of the device and also ensure that system withstands (robustness) even if the device is used beyond its marketed capabilities. Validation phase (Quality Assurance phase) is one of the phases in software engineering which must ensure  device robustness.  QA Engineers play one important role in product development.  Test Engineers, in my view, contribute to success of product more than development Engineers. Test Engineers need to be more agile, adaptive, understand product features, deployment scenarios and aware of all product features.  Only then  Test Engineers would be able to contribute more to validate the product. It is very easy for Test Engineers to get lost of above qualities and become dump black box testers running predefined functional tests again and again without adding value by creating new test cases with respect to attacks, scale and robustness.

Don't get me wrong.  Running the predefined tests on all releases is good to ensure that there is no regression.  But test engineers need to go beyond that be creative and think like attackers and think of multiple kinds of deployments. Creating test cases is continuous process.

I have written one blog entry on "advice to network equipment testers".  I hope that was useful.

What are the test Engineers need to know from their counter parts in development and product management. What questions you need to ask yourself as test Engineer.

1. Do I know the purpose of the feature which I am testing. 
2. What are deployment scenarios where this feature is applicable.
3. What are configuration parameters exposed by this feature to the end user.
4. How do I map these configuration parameters to the deployment scenarios.
5. Is this feature associated with any protocol processing?
6. If there is any protocol processing, do I know the protocol.
1. Go through the standard document if it standard protocol.
2. Go through your internal protocol documentation in case if it is proprietary protocol.
3. Go through the messages and fields AND possible values of protocols.
4. Find out if there are any tools in the public domain related to this feature.
   
5. Think like an attacker and note down the messages and fields for possible negative test cases.
   
7. What are the hardcoded parameters for this feature
1. How many configuration records can it take.
2. How many run time blocks can it create.
3. Think of deployments where the traffic can't be predicted.  These will help in creating test cases to test the robustness of the feature.
8. If there is any standard based protocol processing,  do I need to ensure that it is inter-operating with other products in the market.  
1. What are different ways this feature is used and interoperability testing should cover those.
9. What is the expected performance of this feature in low load and in high load conditions?
10. What is other traffic that is typically run when this feature is enabled. 
1. Find out all kinds of traffic that is possible in deployment scenarios. Ensure that this feature works along with other features.
11. Can the test cases automated?  
1. If so, automate them. So that next time you don't have to spend time on doing manual testing.
   

Above questions are not some thing new for you all. But if you don't know the answers for above questions, you are not doing good job for yourself and your organization.

WAN Optimization Devices - Features that should be expected by buyers

Many WAN optimization solutions are available today. What are the features one should look for?  This list is my view of features the box should provide.  Some features might or might not be used in some deployments though.

WAN Optimization is all about utilizing the WAN resources effectively. At very high level main features one should look for are:

• Usability and  Experience
• Application Detection and providing differential QoS based on applications.
• Load balancing and failover of the connections across multiple WAN links 
• De-Duplication of the data among offices 
• Compression
• Caching 
• Security
• Reliability

WAN optimization has been and will continue for at least a year or two to be a special device in Enterprise networks.  As the value of WAN optimization is realized,  more and more routing, switching, ADC and network security vendors are adding this  as an additional feature in their offerings as a blade or as a software component running on some cores of Multicore processors.  Buyers should not be just looking at the tick mark on WAN optimization, but should check the details.

Usability features: 

• No configuration changes to the client and server machines and their applications should be expected as part of WAN optimization device installation in the network:  Client machines (Desktops, Laptops,  Mobiles etc..) and Server machines (Running HTTP Server, Email Server, CIFS Server) should not even know that WAN optimization device is installed in the network.  No changes to be expected to be made to the machines or applications running on those machines when these devices are added or removed.
• No changes to the existing network infrastructure devices should be expected when WAN optimization devices are installed in the network, except for the cases where there is asymmetric routing.  It is understandable that,  In case of asymmetric routing,  routers need to be configured to redirect the packets to WAN optimization devices using WCCPv2 protocol.
• WAN optimization device should not be appearing as a L3 hop:  WAN optimization devices are expected to provide Layer 2 transparency.  WAN Optimization Device is expected to intercept the packets at the Layer 2 level by acting as a L2 bridge.
• IPv4 and IPv6 Support:  During this migration times to IPv6 from IPv4,  both types of networks, clients and servers are possible.  WAN Optimization device is expected to support IPv4, IPv6 networks, clients and servers.
• When WAN Optimization device is inline of traffic, it is expected that the device has high availability feature using Active-Backup method at the minimum :  If one device fails, other device should be able to take over the WAN optimization functionality. UDP connections must work fine even after backup device takes over the WAN optimization functionality. Existing TCP connections might break, but it must ensure that new connections go successfully through the backup device.
• Device is expected to provide GUI for configuration. 
•  WAN optimization devices should have facility to learn other devices and their reachability information dynamically. WAN devices are also expected to provide configuration facility to add/remove reachability information of other WAN devices statically.
• Any configuration made on a device should be propagated to other devices if some incarnation of this configuration is required on other peer devices.
• It is normally expected that any configuration change done gets reflected immediately. that is, no restart should be necessary for the configuration to be effective.
• Configuration through secure mechanism is expected:  SSH for CLI access,  HTTPS for GUI.
• Configuration consistency across device restarts is expected.
• For Common Critiria and other certifications,  Configuration facility are expected to have role based management with multiple roles with multiple users belonging to the roles.  It is also expected that audit trail is created upon configuration changes.  Audit logs are expected to have all the configuration information changed for the changed records.
• Any configuration update on the Active device should reflect in the backup device without any additional effort by administrator.
• Centralized Management System to configure multiple WAN optimization devices from a single console is normally expected when number of WAN optimization devices are more than few (example: more than 4).
•  Device is expected to provide multiple kinds of reports and statistics to the admin.
• Reports related to amount of WAN bandwidth savings that occurred over specific time period.
• Due to de-duplication,  Due to compression, Due to Caching etc..
• On different protocols (HTTP, NFS, CIFS etc..)
• Reports related to Integrity of dedup repositories.
• Reports related to possible savings if more memory/hard drive capability is added.
• Reports related to traffic belonging to different applications over specific period of time.
• Reports related to amount of WAN utilization and under-utilization.
• Multiple different types of statistics collected over significant amount of time and represented in specific time periods such as hours, weeks, months etc..
• Debug statistics which aid in field debugging.
• Tracing facilities in field with different levels of traces.

Application Detection and providing differential QoS based on applications:


One of the features to utilize the WAN links effectively is to identify the applications and apply traffic management facilities such scheduling, marking and bandwidth control.  Lower priority application such as P2P and non-interactive/non-realtime applications can be ensured to use lesser bandwidth when higher priority application data is pending to be sent on the WAN links.

Many applications can be detected by based on the Destination Port of TCP or UDP protocols.  Application detection is expected to be provided by the WAN Optimization device to detect applications that do port hopping. Examples:  P2P and IM applications.  Application detection is also required to detect HTTP connections being used for social networking,  DDL (Direct Download Links) etc..  Application detection identifies the application ID for each connection. QoS policies would need to have application ID as one of the criteria elements to choose the policy so that the policy rule specific actions such as bandwidth control and prioritization of traffic on to the WAN link can be applied.

Load balancing and fail-over of the connections across multiple WAN links 


It appears that many deployments go with more number of WAN links to satisfy their bandwidth requirements than going for a bigger pipe.  I believe it is less expensive. Also it provides organization to scale the bandwidth as the organization grows.  Having multiple WAN links rather than single big pipe also avoid network discontinuity if one link fails.  These WAN links normally are also taken from different ISPs so as to avoid discontinuity in case of one ISP failure.

WAN optimization functionality is expected to provide capability to support multiple links going towards WAN.  These devices are expected to balance the traffic (based on hash result of IP packet header fields such as source IP, Destination IP etc.) across multiple WAN links. Also these devices are expected to transfer the existing connections on a failed WAN link to new links.  These devices are also expected to maintain order of packets in a flow and hence the balancing criteria configurations selection should be provided to administrators.

De-Duplication of the data among offices

This is one important feature to reduce the amount of traffic exchanged (on WAN links) among offices of an organization.  Basic purpose of de-duplication is to ensure duplicate data is not seen on the wire. Peer WAN optimization devices are expected to hide these details from clients and servers which are exchanging the data. Handling and processing of deduplication happens among WAN optimization devices. Different vendors may have implemented this in different ways. It is important to check the de-dup efficiency.   Some of the features to look for are:

• Block-level Deduplication with configurable block size is to be expected by the administrators. 
• De duplication must be across the protocols.  That is, if the data is downloaded by a client from a server using HTTP protocol first time and same data is downloaded by the client from the same server, but using CIFS, it is expected that actual data is not seen on the WAN link. That is deduplication must happen across protocols.  
• Dedup feature efficiency when the data is not changed on the server, but being downloaded by the client again.  In this case, it is expected that no data, but only the blocks identifiers would be sent on the WAN link, that is, 100% dedup efficiency expected.
• Minimal changes to the data on the server should also lead to near 100% dedup efficiency.  For example, if the additional data of few bytes is added in the beginning of the file in the server, only the changed data or atmost one or two additional blocks of data is expected to be seen on the WAN link when client downloads the changed file.  This change in the data should not lead to transfer of complete file content. In further attempts of same file download should have again 100% dedup efficiency assuming that the file is not changed on the server.
• Blocks that are stored by the WAN optimization should be persistent. This data should be available after any device restarts.  Example scenario:  A file is downloaded by the client machine from server machine. WAN optimization devices cache the blocks of data. Restart the device and download the same file from the server. There should be 100% dedup efficiency.  It is understandable that devices take some time to recreate the internal serach lists from the disk when the device restarts.  During this time the any download of the file will not be able to achieve 100% dedup. But when the system is ready with internal lists, it should lead to 100% dedup efficiency.
• Look for amount of disk space and memory the device has.  Dedup efficiency is directly proportional to this.  Some devices don't support the disk drives for storing the dedup data. It would have multiple problems: 
• Dedup efficiency will not be good as DDR space is limited to store both search lists and data blocks.
• When device restarts, there is no data in the DDR which leads to learning the data afresh.
• Dedup efficiency, when tools like fragroute are used, is as good as the cases when it is not used.   It appears that some WAN optimization devices don't work well in these scenarios.  One might argue that fragroute is a lab tool, but it is necessary to remember that fragroute is simulating some real network conditions. For example, it is normal practice to break the TCP segments to smaller segments to reduce head-of-line blocking created by large TCP packets to allow VOIP RTP traffic.  'Frag Route' tool can be used to do multiple things such as:
• Breaking the TCP segments to smaller segments.
• Breaking IP datagrams to multiple IP fragments.
• Reordering of TCP segments and IP datagrams/fragments.
• Whatever disk size the WAN optimization devices have, it may not be sufficient when compared with amount of new data that is flowing on the WAN links. It is expected that WAN optimization devices throw the blocks which were not used for a long time to make space for new data. New data should be given higher priority all the time.  One way to test to ensure this is to fill the disk by sending unique blocks of data.  Then let the client download a big file from the server and ensure that there si 100% dedup efficiency when the client downloads the same file again.
• Support for Protocol adapters :  Expect protocol adapters for different protocols for following reasons.
• Some protocols such as CIFS are chatty. To reduce the chattiness, some intelligence of protocol is required to do operations such as 'read-ahead'. 
• Knowing data boundary would make deduplication efficient.  WAN Optimization can wait for data with this boundary intelligence and then do dedup processing.  There is higher chance of finding the dedup blocks.
• Doing ALG functionality to figure out data connections such as RTP to apply special processing to reduce any latency and jitter.
• Decoding and Decompression of data before dedup processing occurs.
• SSL Support :  For SSL termination and SSL connection establishment.
• Support for dedup feature for real time and/or  streaming traffic:  When reliability channels are used between WOC devices, then real time traffic quality can suffer.  Hence it is expected from WAN optimization devices to use non-reliable channel for real time traffic.

Compression of data before sending it on the WAN links

WAN optimization devices are expected to provide compression feature to reduce the data on the WAN link. Dedup functionality reduces the data by not sending duplicate data.  Compression reduces the data that is being sent. Hence both functions are expected in the WAN optimization products. 

• It is expected that compression is beyond packet level compression. It should be across the reliable channel (connection) normally established among WAN optimization devices. Compression by maintaining its history can do better job of reducing the data over time.

Caching

Caching feature completely eliminates any data including dedup block identifier data going on the wire if the file is not changed across downloads.  It is only possible to do this in HTTP protocol. 

• WAN Optimization device should act as HTTP Proxy supporting HTTP/1.0 and HTTP/1.1
• SSL Termination to ensure that it does both Caching and Deduplication across WAN Optimization devices.  Peer WAN Optimization device can make SSL connection to the Server.

Security
  
Security on data at Rest:  WAN optimization device stores the dedup data in the hard drive.  It could be confidential data.  Expect WAN optimization to support secure storage of the data.

• Crypto file system versus normal file system:  Devices must ensure that confidential data is stored in crypto file sysetm.
• Encryption key used by crypto file system must not be stored in the same device.  Expect it to provide KMIP or equivalent functionality to get the keys from Key Management Server. It ensures that when device is stolen,  thieves don't get hands on the clear data.

Security on data on wire:  WAN Optimization devices are expected to provide secure transfer of data among them.  Note that these devices are terminating SSL connection at one device and make new SSL connection from remote device to the ultimate server.  Hence it is necessary that communication among these devices is secured.  IPsec is one popular technology to secure the data on the wire.  Expect IPsec kind of functionality in WAN Optimization devices.

Reliability & Data Consistency:

As part of dedup and caching, data is stored in the disks.  Expect some functionality to ensure that the data written to the disk is same as data being read.  RAID is one method by which it can ensure that kind of integrity upon any disk related errors.

As always, it is always good to get the devices and evaluate them in your network for significant of time before buying them. 

Advice to Network Device testers - Simulate Capacity/Stress related faults

Capacity in network devices such as UTM is specified with respect to simultaneous connections in case of firewall, ALGs, intrusion Prevention functionality, tunnels in case of IPsec VPN, number of sessions in case of Anti Virus and Spam functions and many more related to different smaller functions. All the functions are not normally used at the same time. Even if all functions are used at the same time, all sessions may not be going through all functions. Due to this, network device vendors typically oversubscribe memory. That is, the memory needed for all functions for the specif ed capacity would be lot more than the memory available in the devices.

This could pose interesting problem in the field. If there is a deployment where multiple functions are used by large number of connections, there could be memory shortage and other resources shortage. This leads to error being returned when the resource is being allocated. If error detection, propagation and recovery is not taken care well by the software, this could lead to instability, leaks, crashes and lockups. It is tester job to ensure that these kinds of problems do not happen in the field. Typically testers simulate different conditions and ensure that system is stable. Yet times, it is not possible for testers to test all different combinations or simulate different conditions.

I believe testers should be able to simulate all possible combinations by simulating all kinds of error conditions. As part of it, testers should ask development team to provide facilities to inject the faults. In particular, testers should ask for facilities to inject faults for following.

• Memory allocation failures: Almost all functions in software would allocate memory either at the time of connection establishment or on packet basis or to queue the packets and control data etc... Testers should have ammunition to inject the memory fault for specific functions.
  
• Socket/File open failures
• Semaphore creation failures
• Thread/Tasklet creation failures
• Fault simulation of any other OS resource that gets allocated after software is completely initialized.

Testers should go at testing in methodical way:

• Keep list of all functions and OS resource allocations they do.
• For each one of them, create a test case.
• Before running the test case, configure to inject fault.
• Run the test and ensure that system works as expected.
• Run the test without fault and ensure that system is stable.
  

I believe that this kind of testing should happen for every release - feature or maintenance releases. If these tests are done manually, it takes very long time. My suggestion is to automate them.

Firewall and NAT ALG Testing Recommendations

Overview:

Stateful inspection firewalls open temporary holes to allow data connections based on information it reads from the control connection. Some protocols such as FTP, SIP, RSTP, H.323, MGCP open a connection and exchange IP address, port information to peer end point for data transfer. Ports that are exchanged in control connections are not well known ports and they are ephemeral. Due to this, administrators can't configure firewall rules to allow these connections without allowing everything. Application Level Gateways (ALGs) are software modules within firewall interpret the protocol packets by extracting the ephemeral port information and open temporary holes to allow data connections to pass through the firewall between protocol end points. Since each protocol is different, multiple ALG modules are required - one for each protocol.

ALGs also do the address and port translation in the protocol data if firewall supports NAT functionality. If the IP address or ports are specified in ASCII form, there is a big possibility where the data length of the packet changes after translation. In case of TCP based ALG, this results into sequence number modifications in TCP header. Firewalls typically take care of maintaining the delta sequence numbers and modify further packets with this delta in both "Sequence number" and "Ack number" fields to ensure the integrity is maintained with client and sever end points of the connection. It is also important to note that ALGs modify different packets during the life of session and firewall software is expected to keep updating the delta sequence numbers appropriately. It is also to be noted that firewalls need to keep the history of delta numbers with respect to original sequence numbers to apply appropriate delta in case of retransmitted packets which are older.

To apply translation on the data, it is required that the ALG has complete PDU. In some protocols such as H.323 and SIP, this can be large. If there is congestion in the network, the end point does not send the PDU in one TCP packet and requires acknowledgment to send rest of PDU. Due to this, newer generation of firewalls send the acknowledgment to make the end point send rest of protocol data.

Many routers change the TCP MSS value of SYN and SYN+ACK packets of transit traffic to lower value whenever there is multimedia traffic to ensure that VOIP packets do not get stuck. As we all know, Voice traffic is delay sensitive and it should be transmitted as soon as possible. If routers has slow link then it takes significant time to transmit 1500 byte packets. If link bandwidth is 256kbps, it takes around 45msec to transmit 1500 byte packet. If WAN controller of the router chooses 1500 byte packet and if VOIP packet comes right after that, then VOIP packet may need to wait upto 45msec there by increasing the delay of real time traffic. By lowering the TCP MSS value, the size of TCP packets generated by end points can be controlled. Broadband routers setting the value of MSS value of transit TCP packets to 256 bytes are quite common. In these cases, the protocol data of complex protocols requiring ALG comes in many packets. Firewall and ALGs ensure to extract the relevant data for opening holes and translation even protocol data is coming in multiple TCP packets.

As discussed before, ALGs open temporary holes - pin holes. If ALGs are not implemented well, attackers can make control connection and send PDUs with data such a way that pin holes are created to access internal critical services. Also attacker can DoS the firewall by sending large number of PDUs which creates large number of pin holes there by causing service disruption to genuine users/connections.

ALG implementation can become very complex. Vulnerabilities increase with complexity. Buffer overflows, boundary conditions are typical problems associated with complex protocol implementation. That is one place validation should concentrate on.

Many protocols specifications (standards) don't specify maximum length of protocol messages - especially text based protocols such as SIP, HTTP etc.. Protocol implementations (end points and ALGs) typically assume the typical sizes while allocating buffers to buffer the data and don't allow the traffic if it exceeds this limit. Since these sizes are not universally adopted by different implementations, this could pose interoperability problems if ALG implementation assumption of size is different from end point implementations. This is one area validation should concentrate on. In my view ALG implementations should not assume any size restrictions for the PDUs which are not interpreted for its operation. For PDUs that are needed to be buffered, this size restriction should be as maximum as it can be. Some times the protocol messages is prepended with size information. In those cases, ALG implementations must allocate the buffer based on this size information. Validation should concentrate to ensure that ALG does not impose any problems in functionality.

As said before, ALGs also do the translations in the protocol data. ALG implementation should ensure that the translations happen for all three cases - Source NAT, Destination NAT and Source & Destination NAT. Many times validation Engineers concentrate on testing using one session. Many times problems related to NAT can't be found if only one session is used. Validation Engineers should test the ALG based firewall implementations with multiple sessions.


Recommendations:

As you can see, validation testing of ALGs is not as simple as running standard applications on both ends of firewall device. For example, running standrad FTP client and Server applications on two sides of firewall device and ensuring the file transfer succeeds is necessary, but not enough validation of FTP ALG. I recommend validation engineers to consider following for each ALG before certifying.

Functional testing:

• Test with standard applications: Make a list of popular applications. Ensure that different combination of applications as client and server succeed in following cases. Configure firewall to allow control connections (initial connections) only.
  
• Without NAT
  
• Source NAT : With clients behind internal network and servers in external network.
  
• with NAT IP address whose length in dotted decimal form is more than that of source IP address.
• with NAT IP address whose length in dotted decimal form is equal to length of source IP address.
• With NAT IP address whose length in dotted decimal form is less than that of source IP address.
• Destination NAT: With servers in internal network and clients in external network.
• DNAT IP address in dotted decimal is equal to length of Destination IP address in dotted decimal form.
• DNAT IP address is more in length than that of destination IP address in dotted decimal form.
• DNAT IP address is less in length than that of destination IP adderss in dotted decimal form.
• Source NAT and destination NAT together: With servers in internal network and clients in external network.
• Explore all options of applications and ensure that all options work with above NAT combination.
• Ensure that Private IP address (client IP address in case of SNAT, Destination IP address in case of DNAT) does not appear on the packets after translation. This requires capturing the packets and searching for IP address in both binary and dotted decimal form.
  
• Understand the protocol and get familiar with messages and fields and their lengths. If size is not mentioned in the protocol specification, get familiar with realistic maximum length of messages and fields. Test to ensure that ALG does not drop messages when messages with maximum sized fields are sent.
  
• Ensure that ALGs perform well even when there is temporary packet loss. FragRoute tool can be used to drop the packets. Ensure to test with all NAT combinations.
• Ensure that ALGs perform well when the TCP packet sizes are smaller than protocol messages. FragRoute tool can be used to change the TCP packet sizes. Ensure to test with all NAT combinations.
• Ensure that ALGs perform well when TCP packets with smaller size and reordered. FragRoute tool can be used to reorder the TCP segments. Ensure to test with all NAT combinations.

Negative Testing: This testing is required to ensure that systems don't crash when invalid packets are sent.

• Ensure that ALGs don't misbehave (crash or lockup) when protocol messages and fields of different lengths and values are sent. Make a note of all messages in the protocol specifications. Ensure to send messages of different length by writing your own client and server protocol or instrument the existing client and server implementation. I prefer later to reduce the effort required to do this kind of testing. It is relatively simple to generate messages and fields with different lengths and values for the first message of protocol. But for messages that are deep down the protocol require successful initial protocol phase. Hence I prefer going with instrumenting the existing open source client/server protocol implementations.
• Go through some of the common vulnerabilities found in client and server implementations by searching through the CERT repository. Since ALG is also interpreting the protocol messages, ensure that these kinds of vulnerabilities are not present in the ALG.
  

Stress testing: This is one important step to ensure that system under test can cope up with capacity specified. Also, it is important to ensure that system performs as per performance requirements.

• Use IXIA/SmartBits kind of tools to simulate large number of client and servers to ensure that system works as specified with respect to capacity.
• Use IXIA/SmartBits to test the connection rate and ensure it satisfies the specifications of the box.
• Use IXIA/SmartBits to test the throughput requirements.
• Use IXIA/SmartBits to test throughput and connection rate combination requirements.
• Repeat above test cases for 12 hours to ensure that the system is stable.