Security in Hybrid Cloud sans Private Cloud

Check this link :  http://blogs.barrons.com/techtraderdaily/2015/10/21/vmware-plunges-16-business-eaten-alive-by-the-public-cloud-says-street/?mod=yahoobarrons&ru=yahoo

Few predictions made before are becoming reality.  Pure private clouds are disappearing slowly. Enterprises are increasingly using public clouds for may workloads and going for very small private clouds for critical workloads.  Combination of public cloud hosting with private cloud is called hybrid cloud.

I believe that hybrid cloud market as defined today (Private + Public combination) would decline over time and would become niche market.  But another other kind of hybrid cloud market, where Enterprises use multiple public clouds, would increase in future.

Security considerations :  In my view,  Enterprises need to  embed security in their workloads and not depend on generic security solutions provided by cloud operators.  Few reasons on why this is required.

• Enterprises may need to host their services in various countries, where there may not be stringent laws on data protection,  data security.   
• Enterprises may not like to depend on the integrity of administrators of Cloud operators.
• Enterprises may not like Cloud operators to share the data & security keys to governments without their consent 

What it means is that :

• Enterprises would need to consider hypervisor domain as insecure, at least for data.

What is it Enterprises would do in future :

• Security will be built within the workloads (VMs)
• Threat Security such as firewall, IPS, WAF.
• Transport level data security such as SSL/TLS.
• Network Level Security such as Ipsec, OpenVPN
• Visibility would be built into the virtual machines for 
• Performance visibility
• Traffic visibility
• Flow visibility

Essentially, virtual machines would have all necessary security and visibility agents built into them. Centralized management systems, controlled by Enterprises,  will now configure these agents from a central location to make the configuration & management simpler.

There is a concern that if security is built into the VMs, then attacker exploiting the applications in the VMs may be able to disable the built-in security functions, falsify the data or send wrong information to analytic engines. 

That is a valid concern.  I believe that containers would help in mitigating those concerns.

• Run all security functions in the root container of  the VM.
• Run applications in non-root containers within the VM

Isolation provided by containers can mitigate the challenges associated with combining security with the applications.

Service Chaining :  Traditionally, multiple security services are applied by middle virtual appliances. If the traffic is encrypted end-to-end,  these middle virtual appliances will not be able to do good job. Yes, that is true.  This can be solved by Cloud-SFC (SFFs within the virtual machines) where VM SFF itself steer the traffic to various middle appliances or container services within the VM.  More later on this...

I believe that with increasing popularity, flexibility, scale-out, performance provided by CSPs,  it is just matter of time where private clouds would disappear or decline dramatically.  Cloud users (Enterprises) would go for inbuilt security within VMs to host them in public clouds and security/visibility companies would have to address this trend. In  my view only those security/visibility companies would survive.  May be dramatic?  What do you think?

Dual Stack Support in LTE eNodeB - Technical bit

This technical bit summarizes the type of functionality expected out of Dual stack in user plane of eNodeB.

Introduction

eNodeB connects to UE on Air interface side  and connect to multiple types of devices in the core network over backhaul network.  It communicates with MME,  S1 Gateways via Security Gateways.

GTP-U layer is the relay module which transfers the packets to/from UE to wireless core network.  GTP-U tunnel is normally terminated on S1 Gateway for normal traffic.  GTP-U tunnels are also terminated with other eNodeB in handover cases.

GTP-U packets can be optionally secured using IPsec.  Though transport mode is good enough, but tunnel model is commonly used.  IPSec tunnels are terminated at the Security Gateway in the core network.  Security Gateway are typically placed at the edge of core network and its placement is between eNodeB and S1 and other gateways in core network.

eNodeB Dual Stack Requirements:


eNodeB must ensure that it works with its peers.

UEs:

• IPv4 only UEs :  These UEs generate and consume Ipv4 packets only.
• IPv6 only UEs:  These UEs generate and consume Ipv6 packets only.
• Dual Stack UEs:  These UEs can generate and consume both IPv4 and IPv6 packets at the same time.
• IPv4 Radio Bearers
• IPv6 Radio Bearers
• IPv4 and IPv6 Radio Bearers where both Ipv4 and IPv6 packets can be seen on the same RB.

S1 Gateways:

• IPv4 Only S1 Gateways  :  GTP-U tunnels are IPv4 tunnels 
• IPv6 only S1  Gateways -  GTP-U tunnels are IPv6 tunnels.
• Dual Stack S1 Gateways  -  GTP-U tunnels to these gateways could be either IPv4 or IPv6.
• In a given deployment, there is a possibility of having any of above types of S1 Gateways.

IPsec VPN Gateways:

eNB typically contains IPsec part of the eNB itself.  On the core network though,  Ipsec gateway is normally not combined with S1 Gateway.  It is a separate device/blade that sits in the core network.  Ipsec tunnels from eNB are terminated at this gateway whereas GTP tunnels are terminated at the S1 Gateway.

With this above background,  following functionality is expected typically in eNB:

GTP-U Layer:

• Since there are different types of S1 gateways (IPv6 only,   IPv4 only and Dual stack),  MME can decide to put different UEs on various types of S1 gateways.  Hence GTP-U layer must be able to support IPv4 tunnels and IPv6 tunnels. 
• GTP-U must be able to transport both Ipv4 and Ipv6 packets between UE and Core network on IPv4 based GTP tunnels and IPv6 based GTP tunnels.
• GTP-U layer must be able to set DSCP values on outer IP header (IPv4 or IPv6) and should be able to copy DSCP values from inner IP packet (Could be IPv4 or IPv6).  Hence GTP layer should be able to copy DSCP value from  (For Uplink packets) 
• IPv4 to IPv6
• IPv6 to IPv6
• IPv4 to IPv4
• IPv6 to IPv4

IPSec Layer:

As noted above,  one security gateway might be front ending multiple S1 Gateways with respect to base stations.  Since there could be multiple GTP-U tunnels on one IPsec tunnel,  IPsec layer must support following:

• Must be able to work with Security Gateway on core network whether Security Gateway in core network supports Ipv4 or IPv6 tunnel.
• Must be able to transport both IPv4 and IPv6 packets (GTP-U tunnel packets) on one tunnel.
• Must be able to do DSCP copy from GTP-U header to Outer IP header (could be IPv4 or IPv6). 

QoS Layer:

Since both Ipv4 and IPv6 packets traverse on the same Ethernet Port or VLAN port, it is necessary that the shaping and scheduling does not require two different types of configuration. It should be possible to create

• QoS ACL  to have both Ipv4 and IPv6 rules.
• A rule with both Ipv4 and Ipv6 address tuples.
• Multiple rules pointing to the same Queues.

Fastpath Ipsec implementations - Developer integration tips on Inbound policy check

Basic purpose of Ipsec fast path implementations is to reduce the IPsec processing load on the main processing cores.  Since most of the Ipsec processing is same across different kinds of packets, offloading of this processing to hardware makes sense.

There are companies today who provide fast path implementations - either as software component or as an add-on card such as PCIe card that goes onto PCI slot of main processing unit such as x86 based mother board.

Software based fast path implementations are becoming quite popular in Multicore processing environment.  Fast path is run on some cores and rest of the cores are used for other applications.

Ipsec fast path implementations typically work as follows:

• Fast path typically owns the Ethernet and other L2 ports. That is, all packets come to the fast path plane first.
• If there is enough state information to process the packet,  fast path implementations act on the packets without involving normal path running in main cores. Packet might even get transmitted out after working on the packet.  If the packet requires some other application processing that is not present in the fast path,  then the packet is handed over to normal path processing unit.  In case of Ipsec fast path,  decrypted packets are  given to the normal path in inbound direction. In outbound direction,  it does Ipsec processing before packet is sent out.  

Basic purpose of fast path is to save CPU cycles so that it can do some other processing.
All fast path implementation from different vendors are not created equal.

In this post, I specifically would like to concentrate on 'Inbound Policy Check'.  Some fast path implementation skip this check. Reasons given by vendors of the fast path implementation typically is that this is done for performance reasons.  Some people believe that it can be done without any implications on security too.   Unfortunately, that is not true.


What is inbound policy check?

Inbound policy check ensures that the decapsulated IPsec packets used the SA that was formed for this traffic.   And also it ensures that the inbound policy rules allow this traffic to come through.


What are the issues if the inbound policy check is not done?

I can think of two issues -  DoS attack &  Allowing the traffic that is supposed to be denied (Access control violation).

DoS attack:

Let us assume that  a corporation gateway  has two tunnels to two different partners - Partner1 and Partner2. Without inbound policy check, it is possible for partner1 to interfere with the sessions/traffic between corporate gateway and partner2. That is, partner1 can create denial of service attack on partner2 traffic.  Even though I have taken the example of partners, this kind of attack is possible among IPsec remote users.

Let us assume this scenario:

10.1.10.0/24-----------SGW---------Internet-----------PSGW1-----------10.1.11.0/24
                                                             |
                                                             |-----------------PSGW2------------10.1.12.0/24


SGW:  Security Gateway of a corporation - It is protecting network 10.1.10.0/24
PSGW1:  Partner 1 Security Gateway - Its LAN is 10.1.11.0/24
PSGW2:  Parnter 2 Security Gateway. Its LAN is 10.1.12.0/24

There are two security tunnels from SGW - One to PSGW1 and another to PSGW2. Let us call them Tunnel1 and Tunnel2 respectively.

Tunnel1 is negotiated to secure traffic between 10.1.10/24 to 10.1.11.0/24.  Tunnel2 is negotiated to secure traffic between 10.1.10/24 and 10.1.12.0/24.   Let us also assume that the Tunnel1 SPI at the SGW is  SPI1 and the Tunnel2 SPI at the SGW is SPI2.  

It is expected that any packets coming from PSGW1 are expected to have SPI1 in its ESP header and inner IP packet SIP address  be one of addresses in 10.1.11.0/24 and DIP address  be one of 10.1.10.0/24.  Similarly it is expected that any packet coming from PSGW2 is expected to have SPI2 in its ESP header and inner IP packet SIP address be one of addresses in 10.1.12.0/24 and DIP address  be one of 10.1.10.0/24. 

Now to the attack scenario:

If PSGW1 network sends the inner packets whose IP addresses is other than 10.1.11.0/24 and 10.1.10.0/24 , then the SGW is expected to drop those packets.  SGW can only drop the traffic only if SGW does the inbound policy check.  If PSGW1 is allowed to send any inner packets, then it is possible that PSGW1 and its network can misuse this by sending packets with inner packets with IPs of PSGW2 LAN and SGW LAN.  Since it sends the traffic on the right SA using its own SPI,  SGW IPsec packet processing will happen smoothly.  If no other check is done, this traffic can go to SGW LAN.   Based on type of traffic, different attacks are possible.  Some of the attacks that are possible are:

• If attacker at PSGW1 guesses the TCP ports of some long lived sessions between PSGW2 network and SGW network, it can send the RST packets or ICMP Error messages to terminate the connections.
• Attacker at PSGW1 can send ICMP Echo message to SGW1 LAN network multicast IP address with SIP as PSGW2 LAN machine.  Replies from all machines in the SGW1 LAN go to PSGW2 victim machine and overwhelm the machine.

If SGW checks the inbound policy after the IPsec decapsulation is done using inner IP packet,  then it would have found that the SA used for the matching inbound policy is not same as the SA used to decapsulate the packet.  Whenver there is any mismatch, it is expected to drop the packet.  Due to this, no malicious traffic would have gone to the SGW LAN in above scenario.  Also, by logging these events,  administartor can find out the misbehaving peer security gateway and take appropriate out-of-band action.

Access Control Violation:

This is one more problem that can be faced if inbound policy check is not done.
Many Ipsec normal path implementations provide facility for administartors to add multiple rules with different actions to the secuirty policy database (SPD).   Rules normally have 5-tuple selectors in ranges/subnets/exact IP addresses for source and destination and ranges/exact values for UDP/TCP ports.  Actions can be one of 'Bypass', 'Discard' and 'Apply'.   Rules are arranged in a ordered list.  During packet processing,  rule search is done.  Rule search is stopped upon match.  Action specified on the matching rule is taken.  If the action is 'Bypass', then the packet is forwarded without any Ipsec processing.  'Discard' action indicates the packet is to be dropped.  Apply action indication that Ipsec processing is to be applied.  Normally administartors configure the rules with respect to outbound traffic.  Inbound policy rules are created automatically by the system from outbound policy rules by reversing the selectors - That is SIP becomes DIP and vice versa. Similary SP becomes DP and vice versa.

Now let us look at the possible access policy violation with following example:

Let us take these two policy rules in outbound list:

Rule 1:  SIP:  10.1.10.0/24  DIP 10.1.11.0/24  Protocol UDP   Action :  Discard
Rule 2:  SIP:  10.1.10.0/24  DIP 10.1.11.0/24  All Protocols   Action : Apply.

Inbound policy rule list would look like this:

Rule 1:  SIP:  10.1.11.0/24  DIP:  10.1.10.0/24  Protocol:  UDP   Action : Discard
Rule 2:  SIP:  10.1.11.0/24  DIP: 10.1.10.0/24  Protoco:  All   Action:  Apply.

Administartor creates the rules in above fashion to indicate that to discard any UDP traffic between the networks, but allow everything else by securing the traffic.

Assume that above policy rules are created in SGW1.

10.1.10.0/24----------SGW1----------Internet-----------SGW2-------10.1.11.0/24

When a TCP packet is sent from the SGW1 LAN and SGW2 LAN, then second rule gets matched and SA is created to allow traffic 10.1.10/24 to/from 10.1.11.0/24 for all protocols.  If SGW2 either misconfigured or intentionally sends UDP traffic in the SGW1-SGW2 tunnel,  then SGW1 is expected to drop the packet even if it successfully decrypts and decapsualtes the packet.  This can only happen if the SGW1 does the inbound policy check on inner IP packet.

If SGW1 does not do any inbound policy check, UDP traffic would have been passed to the 10.1.10.0/24 network thereby violating the access rules configured by the administartor.

I hope I could give good reasoning on why inbound policy check is required.  Some fast path implementation don't do this.   So, as a development integration engineer, please ensure that not only your implementation, but also fast path implementation does all the checks that are required.

Comments?

Fragmentation before Ipsec Encapsulation - Redside fragmentation and more use cases

I am finding more and more benefits of doing 'red side' fragmentation in Ipsec worl.

One use case is given here:  With red side fragmentation,  any  switches/routers in between security gateways of tunnels don't see  fragmented packets.  Due to this the cases, where some service providers' routers give less priority to the fragmented packets, don't arise. 

Second use case is given here :  When majority of  the traffic goes on IPsec tunnels, LAG can't distribute the traffic across ports since the result traffic  has same 5-tuple information. As described in the post,  multiple IPsec tunnels normally get created with forceful NAT-T.   All packets that are coming out of Ipsec Engine are expected to have 5 tuple information. If fragmentation is done after Encap, then the LAG would see some packets without 5-tuples.  This results to uneven distribution.  Hence redside fragmentation is done to ensure that LAG sees 5-tuples for all the packets.

Third use case:  Avoid mis-ordering of the packets:

There could be packets which are big and small in the traffic.  Big packets may get fragmented after Ipsec encapsulation if the result size exceeds the MTU of outgoing interface.  Small packets may not get fragmented even after encapsulation.

Gateway receiving the Ipsec packets is expected to process them in order. Due to fragmented packets, this may not happen.  Let us say that,  gateway received 1st fragment of 1st packet,  2nd full packet and 2nd and also final fragment of 1st packet in that order.  It is expected that the gateway processes them  in the same order.  But since 1st packet waits for 2nd fragment,  2nd full packet would be processed. Gateways don't stop the full packets getting processed as it may not know whether or not second  fragment of the 1st packet is going to come in and also when it is going to come in. 

So, this leads to packet mis-order.

This can be avoided if there are no fragments. Solution : Red side fragmentation.

Comments?

eNodeB and IPsec

eNodeB secures the traffic over the IPsec tunnels to the Serving Gateway (SGW) over backhaul network.  Also, eNB creates many tunnels to peer eNBs for X2 and handover traffic.  Though all features related to Ipsec are valid in eNB scenarios too,  some features are worth mentioning in eNB context.

LAG and IPsec:

Please this link here to understand the issues and solutions related to LAG and Ipsec in general. This scenario is very much valid for eNB to SGW communication. Note that traffic from all GTP tunnels in non-handoff scenario go between eNB and SGW on one or few (when DSCP based tunnels) Ipsec tunnels.  When LAG is used between eNB and LAG,  similar issue of not utilizing more than one link would arise.  Both the solutions suggested in earlier article are valid in this scenario too.  In cases where it is difficult to get multiple public IP addresses to the LAG link, then scenario 2 - forceful NAT is only option I can think of.


Capabilities expected in eNB and SGW:

Using LAG effectively requires many tunnels.  It is good to have 1 + (number of links - 1 ) * 32 Ipsec tunnels for good distribution across links.  User traffic, in this case GTP traffic should be balanced across these IPsec tunnels.

Typically there are two GTP traffic tunnels for each cell phone user - One is typically created for Data traffic and another for voice traffic.  Without LAG, normally two Ipsec tunnels are created - One for data traffic coming/going  from/to  all the cell users and another for voice traffic for all voice traffic coming/going to cell users.  GTP traffic is distributed across these two Ipsec tunnels based on DSCP value.

Now, we have lot more Ipsec tunnels.  There should be additional logic in eNB and SGW which distributes the GTP traffic across these multiple Ipsec tunnels.  This logic should distribute the traffic from a given conversation to one Ipsec tunnel.  Each GTP tunnel traffic can be viewed as one conversation.   That is, GTP tunnels are distributed across the Ipsec tunnels.   One way is to look at the TEID (Tunnel Endpoint ID) and use hash on TEID to distribute the traffic across Ipsec tunnels.

Ipsec implementation on eNB and SGW should have capability to create multiple tunnels - number of tunnels to be created should be configurable.  eNB and SGW implementations also should have capability to bring up the tunnels on demand basis too. That is,  they should ensure that these number of tunnels are UP and running as long as there is traffic.  Note that all ipsec tunnel negotiation would have same selectors and ipsec implementations should not be having intelligence to remove old tunnels with same selectors.

If persistent feature is selected on SPD rules, then the implementations should ensure that all ipsec tunnels are UP and running all the time.

As described in the earlier article, it is necessary that Ipsec implementation have capability of doing 'Red side fragmentation' so that the LAG always sees UDP header in every packet which is required for its distribution.

DSCP based Ipsec tunnels:

LTE uses packet based network for  voice, streaming,  interactive and non-interactive data.  Hence it is necessary that Ipsec tunnel honor this priority to ensure that voice and other real-time traffic is given priority.  If both data and voice is sent on the same tunnel, there is a possibility of traffic getting dropped due to sequence number checks as part of anti-replay checks in the receiver.  Even though packets are marked with increasing sequence number in both data and voice traffic and encapsulated in the Ipsec tunnel,  due to local QoS and QoS in intermediate devices, voice traffic may be sent before the data traffic - that is traffic is reordered.  As you know, receiver window right edge moves with newer sequence number.  Due to this, some data packets which have lower sequence number get dropped if they are less than the lower edge.  To avoid unnecessary drops, there are two methods used - Increase the Anti-replay window size  or use different SAs (tunnels) for different kinds of traffic. Second method is normally used.

Due to this feature and above LAG feature, number of tunnels that need to be created in eNB and SGW can go up significantly.  Hence both eNB and SGW should have enough memory and computation power to handle multiple tunnels.

Persistent tunnels

To reduce the latency of initial traffic, it is necessary to have this feature. Tunnels are UP and running all the time even when there is no traffic.  This feature is good if the links are always-on and if there is no cost based on the traffic amount.


DSCP and ECN Copy settings:


Ipsec implementations expected to copy DSCP and ECN bits from inner header to outer header.  Inner header DSCP value is set by applications and this should be continued even when the traffic is tunneled.  This will ensure that the nodes between the eNB and SGW will also give QoS treatment.  Hence it is necessary to copy the DSCP bits from inner header to outer header.

ECN bits indicate the congestion to the peer so that peer entity can inform the source entity to apply the congestion.  TCP protocol has a way to inform the source entity when the receiver gets the IP packets with CE (congestion experienced) bit on in ECN bits of IP header.  Intermediate nodes, including eNB and SGW should honor this by copying from inner header to outer header while encapsulating and copy from outer header to inner header while decapsulation.

Peer IP address adoption 

eNodeB gets the IP address from backhaul provider dynamically. It is possible that IP address might be changed by the provider while traffic is going on.  Ipsec tunnels are expected to be UP and running even if the IP address changes on the gateways.   This internet draft discusses the mechanism to adopt the peer gateway address change.  This feature is expected to be present to ensure that voice traffic does not observe too much of jitter and latency.  Note that tunnel establishment takes hundreds of milliseconds as it involves IKE negotiation of the keys.  This can introduce jitter and latency when the voice traffic is going on at that time. Implementation must implement this draft to eliminate jitter and latency issues when the IP address changes on the remote gateway.

IP Fragmentation and Reassembly

Normally many vendors of eNB give performance with respect to UDP traffic without involving IP fragmentation and reassembly. Even though this gives one data point, this may be misleading to customers. Most of the traffic in Internet today is TCP and same is true in LTE world too.  TCP MSS is chosen such as way that TCP data packet with IP and TCP header would be MTU size.  When the traffic undergoes IPsec encapsulation, it is almost certain that packets would need to be fragmented as it exceeds the MTU of the link.  Though DF bit facility is available for end points to know the Path MTU,  this feature is not used in Ipv4 end points today.  Since packets are fragmented,  reassembly is required on other side.

This feature is implemented in Ipsec implementations, but I am afraid that many implementations, though have very good IPsec performance on non-fragmented packets, they are not optimized when fragmentation and reassembly is required.  Customers need to watch out for this as significant amount of traffic would be fragmented and reassembled.

IPv6 Support

Ipv6 is fast becoming choice of service providers and LTE core network.  Hence Ipv6 support is expected in ipsec implementations.  eNB and SGW must support both Ipv4 and IPv6 tunnels.  Also, they should be able to send IPv4 and Ipv6 traffic on IPv4/Ipv6 tunnels.

TFC

Traffic Flow Confidentiality feature is normally given less importance.  I was told that LTE networks require this feature be implemented in Ipsec tunnels so that the anybody who gets hold of backhaul network traffic will not be able to guess the type of traffic that is going on in the tunnels based on traffic characteristics such as - frequency of traffic,  size of packets,  distribution of packets  etc.. 


AES-GCM

AES-GCM combines both encryption and integrity in one algorithm. Hence it is called combined algorithm.  This algorithms is supposed to be 2x faster than AES-CBC algorithm. Also it is supposed to have half of latency of AES-CBC.  Hence it is good for both performance and also for latency which is required for voice traffic.  Hence it is becoming popular algorithm in eNB and SGW.

Validation Engineers and customers I believe should look for above features in eNB and SGW.

Comments?

Link Aggregation and Ipsec

Link Aggregation is also called Ethernet Trunking and Bonding.  This feature is described in 802.3ad. In 2008, this was rolled into 802.1AX group.

What is LAG:

LAG combines multiple Ethernet Ports and exposes it as one link to the upper layers in the system.
It is Layer 2 concept.  Only trunk port would be assigned with IP addresses.  Links in the trunk don't have any Layer 3 information.  Only one MAC address would be used for the trunk.  Individual MAC addresses of the links don't appear in any communication other than control protocol (Marker protocol).

How does it work?

LAG contains two components - Distributor and Collector.
Distributor distributes the outgoing traffic across the links that constitute the trunk.  Collector collects the data in inbound direction coming from different links and tunnels through the trunk port to rest of the system.
LAG assumes that all links in the trunk are full duplex and point to point.
Simplest distribution is to distribute the packet by packet across the links based on weight configured on the links.  But there could be packet mis-ordering issues.


What are some critical items to be taken care by the Distributor:

Packet mis-ordering is one of the issues distributor would face if it distributes the traffic blindly on per packet basis.   To avoid mis-ordering,  distributors are expected to send all the packets of given flow (conversation) sent on the same link.  First generation distributors used to apply hash on source and destination IP and select the link based on hash value.  Though this ensures that the traffic belonging to one conversation goes on the same link, but the distribution may not be symmetric for some workloads.  Second generation distributors go one step beyond and apply the hash on TCP/UDP ports too.  This would give better distribution, but it may have some mis-ordering problem if the outbound packets are fragments.  Only first fragment would have transport header and other fragments don't have transport header. In those cases, there is a chance that non-initial fragments go on some other link and lead to mis-ordering.  Since fragments are not very common, some deployment accepts some level of mis-ordering to get the better utilization of the links.

Collector and Packet mis-ordering:  

To ensure that packets are delivered in order, collector should ensure to send the packets up in order it receives on any given link. There is no order to be maintained on packets coming in across links.  Collector also should ensure that it does not starve any link while receiving the packets.

Ipsec and LAG:

In some deployments, traffic is always encrypted via Ipsec and sent to the remote office.  If one tunnel  is used to send the traffic, all the traffic going from the local network to remote gateway contains same source, destination IP addresses.  In case UDP traversal is applied, it would have same source and destination ports.  Even if there are multiple links in LAG, distributor hash will fall onto only one link and other links would not be used. 

Same is true with Reverse traffic.  Also note that Links in Aggregation group are with local ISP.  Remote gateway under same admin control will not know about local Link aggregation.  That is, incoming traffic balancing across the links is in the hands of service provider.  It is okay to assume that most of 802.3ad distributors are configured with to use IP addresses and in some cases even ports.

Since distributors only know the IP addresses and ports of the packet, links would be utilized well in both directions if there are large number of flows with different IP addresses and Ports. 


Solutions

There are two solutions I can think of.

Solution 1:  Using Multiple IP address on the trunk link. 

Create as many tunnels as number of IP addresses on the trunk link with remote gateway.  As described in the link here,  some software should distribute the flows across these IPsec tunnels.  Since each tunnel now has different source IP address in the outer IP header, LAG distribution hash may fall into different links and thereby utilizing the bandwidth well in outbound direction.  One should ensure that, there are many local IP addresses to ensure that all links are used and also all links are used evenly.

Reverse traffic also would be balanced fine as service provider switch also would see different IP addresses (Destination IP).

Getting or assigning  multiple public IP addresses to the trunk may not be possible.  In which case, second solution can be used. But second solution would have some packet overheads.

Solution 2 :   Usage forceful NAT-T

Even though NAT is not detected,  there are ways to force UDP traversal. That is, ESP packets are sent with in UDP payload.  Create as many tunnels as necessary for good distribution at the LAG level.  Each tunnel would have different UDP source port.  Some software in the device is expected to balance the traffic across these tunnels. LAG would distribute the tunnels across multiple links of the LAG.  Reverse traffic also would be balanced on different links due to different destination port values of tunnels.  Since it is expected that LAG distributor look at the transport header for distribution, it is necessary that there are no fragments.  So, it is mandatory that tunnels are configured with redside fragmentation. This will ensure that fragmentation is done before Ipsec encapsulation.


In both the solutions,  both remote and local gateways should have some logic to

• know that multiple tunnels are created for same selectors for distributing the flows.
• know how to distribute different conversations to different tunnels.

It requires more tunnel capacity in devices.  This should not be a problem as modern devices has good horse power and enough memory to create some more tunnels with peer gateway.


Comments?

Linux based fast path - Why is this needed?

I found two excellent blog posts on this subject. Please see them here:

http://www.multicorepacketprocessing.com/the-need-a-fast-path-on-multicore-cpu/
http://www.multicorepacketprocessing.com/os-networking-stacks-like-linux-are-not-well-adapted-to-multicore-packet-processing/

I see this statement in one of blog entries "However, the stack of any OS cannot go under ~2000 cycles per packets due to its design.".  It is really true.  Linux and other operating systems are generic and they need to cater to multiple different types of applications. Generality always comes with the price.  Forwarding process of IP packets in  Linux for example go through many layers of software from reception of the packet to the transmit of the packet.  These layers add to cycles.  As excellently listed down in above blogs,  higher layers of the operating systems don't have full control over the hardware (processor and accelerator) features. Since many layers of software comes in picture,  cache utilization is not very good.  On top of it, context switching and locks (for Multcore processors) in these generic layers would use up more core cycles.

Fastpath is nothing new. This has been adopted by networking vendors for a long time.  Fastpath is specific to applications running in the networking devices.  For example,  in networking devices,  fastpath layers are implemented for Firewall, IPsec VPN,  QoS and forwarding.  Fastpath concept is very simple and described below.

• When packet is received, it first checks for the matching context.
•  If no context, packet is given to the normal application modules.
• Normal path based on its policy rules may decide to handle the session by creating its own context in its application. In addition it either decides to create the context in FP, or decides to handle the context itself by not populating the context in FP
• If there is a matching context, packet is handled within fastpath and packet is routed without involving the normal path.

How/Why do fastpath modules get good performance?

• It runs right above hardware and it has access to all hardware features.
• Since FP is specific to given hardware, it can take advantage of all features without worrying about losing generality. Since fastpath module for any given application is expected to be very small in footpriint,  not having generality across all hardware devices is considered okay.
• Fastpath implementations follow run-to-completion model.
• Due to its small footprint,  most of the fastpath code might be in the L1 Cache of processors.
• They can do locking of critical data blocks in processor caches.
• Lock free implementation - Linear performance with cores

As the 'Multicore processing' blogs have indicated it is no surprise to see 3 to 10x performance improvement based on the type of application.

There are different types of fastpath implementations in the industry today.

• ASIC based fast path implementations.
• Network processor based fastpath implementation (Remember NPF. They even have set of API documents for fast path. You can still find there here )
• Control plane and Data Plane with some cores running CP and some cores running FP in DP.
• Linux Ethernet Driver based Fastpath for devices that use Linux SMP.

There are some pros and cons among different fastpath approaches. But the basic idea is same across them.  Do routine jobs of packet processing in fast path and let normal path handle special connections and packets.

I believe many  network device vendors would need to go for some kind of fast path if they need to support high volume of traffic with cost effective processors.

NAT and IPsec - Application sequence

To my surprise, after so many years of IPsec deployments,  I keep hearing questions related to NAT and IPsec.  It appears that there is some confusion still on the sequence of operations between NAT and IPsec. That is why I thought I will write a blob on this subject.

Before going further into the technical aspects, it is good to introduce two scenarios.

• Branch Office VPN :  This term is typically used to connect offices of an organization over public Internet with IPsec VPN security.
• Partner VPN :  This term is typically used to connect some part of network or machine(s) with partner network securely using IPsec VPN.

If IPsec VPN is being used to secure networks of different offices of same organization, one can assume that the private IP addresses of different networks are unique within the organization.  Hence IPsec VPN can be applied on the private networks of offices.

While connecting with Partners,  one can't assume that the private networks of partners are unique. It is very much possible that both the networks might have same private IP address networks.  Hence the IPsec VPN must always happen with public IP addresses.  In this case, typically source NAT is applied before IPsec VPN is applied on the packet.  Source NAT translated the local network IP addresses to public IP address and the IPsec sessions negotiated with public IP addresses.

Note that it is also possible that a given a branch office VPN router, not only used to connect to the other offices of the organization, nut also connect partner networks. So, VPN routers must be able to support multiple site-to-site VPN within its organization, but also support site-to-site VPN with partner networks.

Let us take a simple scenarios where one secure gateway  is securing 10.1.5.0/24 network. It is expected to securely connect to its head office VPN router which is securing 10.1.6.0/24 network. Let us also say that  this router is expected to secure traffic to/from two machines on its local network (say 10.1.5.5 and 10.1.5.6) with three partner machines in partner network.  For this purpose, let us say that two public IP addresses are allocated for the local security gateway -  190.1.2.2 and 190.1.2.3.   Let us also say that partner provided IP address to connect to the partner machines -  191.1.2.2,  191.1.3.3 and 191.1.2.4.

On the security gateway, following NAT rules are required:

• Source Range:  10.1.5.5 - 10.1.5.6,  Destination Range:  191.1.2.2-192.1.2.4,  Apply Source One-to-One NAT with 190.1.2.2 - 190.1.2.3 :   This rule does source NAT  on the connections that are originated from local network identified by source. It replaces source IP of the packet it is receiving from local network if it matches with 10.1.5.5-10.1.5.6 range and if destination IP is in the range of  191.1.2.2 - 192.1.2.4.  NAT IP address are given as 190.1.2.2 - 190.1.2.3.  Since it is One-to-One NAT,  it replaces 10.1.5.5 with 190.1.2.2 and 10.1.5.6 with 190.1.2.3.
• Source Range:  191.1.2.2 - 192.1.2.4, Destination Range:  190.1.2.2 - 190.1.2.3, Apply Destination One-to-One NAT with 10.1.5.5 - 10.1.5.6:  This rule is applied for connections that are originated by partner network.  It translated destination IP so that the connections land at the right machines in local network.

IPsec policy rules would look like this:

•  Branch office VPN rule:  Source:  10.1.5.0/24  Destination 10.1.6.0/24 Apply Security (Algorithms and proposals are not shown here).
•  Partner Office VPN rule:   Source :  190.1.2.2-190.1.2.3  Destination :  191.1.2.2 - 192.1.2.4, Apply Security. 

I hope I could make it clear on how NAT and IPsec can be used to connect with partner network. Note that private IP addresses can be used to talk to partner network too as long as administrators of both organizations ensure that there are no duplicate private IP addresses in their networks that need communication.

IPsec as WAN protocol for IPv6? Read on...

In recent past, I see the trend of using one Security Association (SA) for both IPv4 and IPv6 selectors. That is, if two sites are being secured using IPsec having IPv4 and IPv6 networks,  traffic from both IPv4 and IPv6 networks go on the same SA. Ofcourse, it is possible with IKEv2 only. IKEv2 has facility to send both IPv4 and IPv6 selectors together as part of CHILD_SA negotiation.

Remote Access Client (IRAC) traditionally used in mobiles and desktop/laptop end points. They connect to IRAS in corporate office, get private IP address and access corporate networks.

In IPv6,  IRAC is not only used in end points, but also increasingly being used in small office IPsec boxes. More interestingly, IPsec is being used to get the IP addresses for internal LAN machines from the ISP. In this case ISP runs IRAS.  That is,  IPsec is being used as WAN protocol.  The flow is some thing like this:

• IRAC in CPE making IKEv2 connection to the IRAS in ISP.
• IRAC requests IPv4 and IPv6 information from the IRAS using configuration payloads in one transaction.
• IPv4 configuration attributes typically involve IP address, DNS Servers,  Remote Networks etc..,  
• IPv6 configuration as described in IKEv2 RFC is not sufficient for this kind of deployment. RFC5739  defines the attributes which get
• Multiple IPv6 prefixes
• DHCPv6 Server address at IRAS end.
• This standard expects the IRAC to get rest of information such as DNS Servers and any other information via DHCP to DHCPv6 Server address it gets through this exchange.
• How IPv4 address would be used:
• This is typically public IP address.
• This is used for NAPT. 
• DNS Servers and WINS Servers IRAC gets would get configured dynamically into DHCPv4 Server.
• Local IPv4 machines are assigned with Private IP address via DHCP v4 Server.
• How IPv6 information is applied
• Since it gets multiple prefixes,  it can assign one address from each prefix to IRAC interface (Virtual Link interface) in additional to Link Local address.
• Prefixes are configured with DHCPv6 Server on the LAN interfaces if stateful addressing is being advertised via RA on the LAN or to the RA Proxy which sends the prefixes in RA messages. If there are multiple LAN interfaces than the prefixes, then prefix need to be divided further and assign subset to each LAN interface.
• Any information it gets through DHCP transaction with IRAS DHCP Server also might need to get populated in DHCPv6 Server on LANs (Stateful or Stateless).

Even though above deployment is mentioned as WAN access,  similar transaction can happen in corporate world.  Small home office or small sales offices in multiple locations can use similar mechanism to assign IPv6 address to local machines to communicate with IPv6 networks in corporate offices.

OSPF v3 IPsec requirements - Notes

RFC 4552 standard provides enough details about IPsec requirements.  Some additional details are given here.

OSPFv3 is mainly meant for IPv6 link state propagation.  Unlike OSPF for Ipv4,  OSPFv3 for IPv6 assumes that the authentication, integrity and optionally confidentiality is provided by inbuilt IPsec layer in IPv6 stack. What capabilities are expected to be provided by IPsec layer to provide authentication and confidentiality functionality to the OSPF?  Before we go further,  please refer to to OSPFv3 RFC specification to understand OSPF concepts. I also found this reading very useful to get faster understanding of OSPF. Here I try to give some quick introduction to OSPF concepts that are relevant for this discussion.

Basic purpose of OSPF is to create routing table in a given network node (typically routers) based on link state information it receives from the adjacent routers.  A router can have multiple physical or logical (VLAN) interfaces connected to different network segments.   OSPF routing protocol runs on the configured interfaces.  As many OSPF Hello protocol instances are run as number of configured interfaces.    A router can be in multiple areas.  A given interface is always in one area.  Router collects link state information across all interfaces of the area and runs Dijkstra algorithm to figure out the shortest paths to all known destinations. If the router is configured to be in multiple areas, then it is called Area Border Router (ABR).  In OSPF world, areas are created administratively to reduce the number of link state updates in the network.  If all routers in the organization is in one area, then amount of link state information exchanged can be very high and amount of time and memory required to run Dijkstra algorithms also can be high.  By creating multiple areas, number of link state updates when there is change in link information is reduced to its area.


OSPF has a concept of Area 0. It is called backbone network.  Every area is expected to have one ABR with the Area 0 (backbone network).  If an area does not have ABR with area0, then it can't communicate with other areas for which it does not have ABR.  By having ABR with area0,  continuity across multiple areas can be maintained.

In some cases, it may not be possible to have direct connectivity with area0.  When the area does not have physical connectivity with area0, then  it is expected that administrators create a virtual link to the area 0 via transit area.  Transit area is the area which is connected to the area in question and the area0.  Virtual link can be thought of as a point to point link.  It creates an p-to-p interface.  More often multiple virtual links are created for redundancy purposes.  There is one standard http://tools.ietf.org/html/draft-zhang-ospf-dvl-01  which proposed some changes to the OSPF protocol to discover the discontinuity with the area 0 and creates virtual links dynamically by figuring out the transit areas.  But this draft was not widely implemented and hence the virtual links are continue to be configured manually.  Virtual links are normally created with the router IDs which are not IP addresses of the each end of virtual link.  Some vendors provide virtual link configuration with peer IP address.  Every time there is a change in the configuration on the peer, it is required to change the IP address.  To avoid administration mistakes, it is typically left to the OSPF protocol to determine the IP address from router LSAs in transit arewa.   The IP addresses are discovered dynamically  via router LSA in transit area.  OSPF then activates the virtual link. From then on OSPF uses this virtual link as point to point link with the area0.  What it means is that the IP addresses of IP header used by OSPF packets on virtual link are not known to the administrator at configuration time.

Let us come back to OSPFv3 and IPsec based authentication.

It appears that CISCO is supporting both interface level authentication and area level authentication. Juniper seems to be supporting interface level authentication for sure.   In interface level authentication, all neighbor routers would need to have same SA information. Different interfaces  (network segments) of same area can have different SAs.   In case of area level authentication, all routers and interfaces in that area would need to have same SA information.  It is administratively simple to have one set of SA parameters (SPI and keys) for area.  But there may be other security reasons (such as different administrators for different network segments) to have separate authentication/encryption for each network segment.  In case of ABRs,  routers belong to multiple areas.  If area level authentication is used,  there would be as many SA authentication parameters as number of areas.   See this link to understand configuration on CISCO and this link to understand Juniper configuration. 

Note that OSPF uses both multicast and unicast addresses to send its messages to the adjacent routers. Whenever any new OSPF router is added to the network or removed from the network,  it is not expected by administrator to do any configuration changes on current OSPF routers ( Except for virtual links I guess).  One must ensure this even  when OSPFv3 being secured using IPsec. That is, IPsec configuration should not be expected to be changed on the current OSPF routers when new OSPF router is being added. I would think that administrators will not configure neighbor IP addresses in the selectors of SPD policy records.  SPD policy IPv6 selectors would have OSPF IP protocol,  source IP as ANY and Destination IP as ANY.

As we discussed there can be multiple SA requirement either due to having authentication requirement for every network segment or due to multiple areas.   So, one should have many SAs (SPD policy records) with same selector set.  If there are multiple SPD records having the same selectors set,  first record is always will be matched which is not desirable.  That is where,  interface level Security policy database come in handy.  For each interface, there is one SPD.  When  OSPFv3 IPsec is configured on interface, a policy record with  selector having ANY to ANY with OSPF protocol is created in interface specific Security Policy Database.  We also discussed that yet times the SPD records are not interface specific, but area specific where area level authentication is desired.   To take care of this,  one option is to duplicate SPD records in all interfaces corresponding to that area.  This will have administrative overhead and memory overhead.  It is better if IPsec supports area based SPD.  As a matter of fact,  some IPsec stack are capable of doing this, though the capability is not specific to OSPFv3.

IPsec stacks do support multiple SPDs for providing IPsec interfaces for route based VPN.  Since interface IDs are dynamic in nature,  the SPD policy database granularity is provided on fixed domain basis.  Interface ID to domain mapping is configured or programmed at run time.  Multiple interface IDs can be mapped to one domain. But one interface ID can't existi in multiple domains.   If interface level based SPD is required, then the interface to domain mapping is 1:1.  This feature can be mapped to OSPFv3 based authentication easily.  If interface level authentication is chosen,  then there is 1:1 mapping between interface and domain.  If area level authentication is chosen, then area is considered as domain.   OSPFv3 module at run time can generate the mapping of interface ID to the domain or administrator can configure interface names to the domain.

We discussed about virtual links. Virtual links are also interfaces and hence SPD can be configured on per interface basis.   But there is one problem though as described in section 11 of RFC43552.  See the descriptoon about rule 4 and 5.  To protect against clear OSPF packets going to the OSPF daemon through via normal interfaces or any other interface, it is necessary to program the virtual link IP addresses in all interface specific SPD records. That is the reason you require IP addresses for virtual links.  This is not required in case of other SPD records as IPsec packet processing module knows which inbound SPD to look at once the IPsec decapsulation is done.

OSPF requirements for IPsec are :

• Common Requirements
•  Transport code.
• AH Support is MUST as ESP is not implemented by some popular routing vendors.
• Manual Key Management support  :  Please see RFC4552 on why manual key management is required. Basic need is due to Multicast support.  Since multicast key distribution protocols are not popular,  Manual key management is expected to be supported. 
• Disable Anti-Replay check.
• Configuration to replace keying material where administrators will not be able to change the configuration of all routers at the same time. See RFC 4552 Section 10.1 'Rekeying procedures'.
•  IPsec module should be able to take KeyRolloverInterval. Typically it is in hours/days.
• Once the new keys are configured at different times but within KeyRolloverInterval,  without any manual intervention, IPsec should be able to bring new SAs and delete Old SAs.
• OSPF - Internal Router :  In this case, all interfaces belong to only one area. 
• If area level authentication is good enough:  Then global SPD is good enough.
• If interface level authentication is required, then it requires to have interface level SPD.
• OSPF - Area Boundary Router requirements.
• If interface level authentication is configured for all interfaces,  interface level SPD capability is required.
• If area level authentication is configured and if there are multiple interfaces to each area, then domain level SPD capability is required.
• If Virtual links are required, then IPsec stack should have ability to create policy records associated with the virtual link addresses on both ends in all interface/domain specific SPDs.

What changes are required in OSPF implementation?  If virtual link capability is not required, then there is no change required in OSPF implementations as long as administrator ensure to configure  interface level/area level authentication and configure the interfaces domain mapping.   When virtual links are configured, then OSPFv3 module may require to inform IPsec module with IP addresses of virtual link so that IPsec module can create the policies in all interface/area (domain) specific SPDs. By the way, this is also optional as OSPFv3 anyway is going to drop the packets if they don't come on virtual link.  So, in essence there is no real requirement to change the OSPF implementation to provide IPsec authentication.

IPsec VPN load balancing - technical bit

Many Enterprises are having multiple WAN links for sharing the WAN load and also to provide redundancy. Enterprises are increasingly going for WAN links from different providers for continuous connectivity even when there are failures at service provider end. There are many solutions in the market which take advantage of multiple links - by bonding these WAN links. Since these WAN links belong to different service providers, load sharing is done typically at the connection level. That is, all packets of a 5 tuple connections go through the same WAN link, but packet belonging to different connections go via different links.

IPsec VPN tunnel between two offices is considered as one connection by these solutions. Due to this, even if both offices have multiple links, the bandwidth for IPsec traffic is limited to maximum bandwidth provided by selected link. Actually, it would be the minimum of office 1 link and office 2 link bandwidth. In many situations, the traffic among offices (branch offices to head office) is mainly IPsec traffic.

IPsec VPN load balancing functionality is expected to solve above problem and fully utilizes the maximum bandwidth provided by multiple links.

Let us examine different deployment scenarios:

• Sceneario 1: Branch office having one WAN link and Head office having multiple WAN links. And entire VPN traffic in each side serviced by one VPN router (or UTM box)
  
• Scenario 2: Both offices having multiple WAN links and all the VPN traffic in each side serviced by one VPN router (UTM box)
• Scenario 3: Branch office having multiple low bandwidth links and head office having one high bandwidth link.
  
• Scenario 3: Both offices having multiple WAN links and as many UTM VPN routers as number of links.

IPsec VPN functionality is typically used in tunnel mode. In tunnel mode, IP packets traversing between the sites are encapsulated. As part of encapsulation, newer IP header is added and it is called outer IP header. Outer IP header is mainly used for routing of packets across Internet to reach the right VPN router. The IP addresses of outer IP header, hence must be public IP addresses. These IP addresses are called 'gateway' IP addresses. Local gateway IP address is used as 'source IP' and remtoe gateway IP address is used as 'Destination IP' in the outer IP header. These two gateway IP addresses are called 'gateway pair'. WAN link load balancing function uses these IP addresses (mainly source IP address, as both WAN links have default routes) to route the packets on appropriate WAN link. Hence, it is required that VPN router create multiple tunnels with different local gateway IP addresses and balance the outbound traffic across these tunnels to use the bandwidth of local WAN links efficiently. It is expected that the remote router also does same thing on its outbound traffic.

In all scenarios, the IPsec VPN functionality of UTM box should take multiple gateway pairs configuration in appropriate SPD policy records. In theory, there could be M * N pairs with M being number of links in site 1 and N being number of links in site2 2. But, in reality, number of gateway pairs one would configure is some where between minimum of M and N to M*N. For example, In scenario3, to take advantage of multiple WAN links in branch office, both head office router and branch office router can be configured with two gateway pairs. On head office side, local gateway IP address is same across these two gateway pairs and in branch office side remote gateway IP address is same across gateway pairs.

How to load balance traffic among multiple tunnels between two sites:

Many VPN routers today have stateful security functions such as firewall, IPS etc.. UTM boxes when used in place of VPN routers have many stateful security functions. If entire VPN traffic is handled by one device, packet based load balancing works fine. But, if the WAN links are handled by two different VPN routers having stateful functionality, then packet based load balancing does not work. Even connection based load balancing can have problems for applications having multiple connections. Hence, I suggest not to use load balancing, but use manual load balancing by creating multiple SPD policy records between two sites, with SPD policy record having different gateway pairs and with different selectors.

In summary: Packet based load balancing across different tunnels of one SPD policy record is applicable if VPN routers don't have stateful security function or if the VPN router is terminating all tunnels.

UTM Message logging capabilities

UTM devices are multi function security devices. These functions include firewall, IPsec VPN, SSLVPN, IPS, Anti Virus and Anti Spam. Some devices even include web application firewall function. Each security function is different from others. So, one would expect the information in logs is different across different security functions and even sub functions within a security function. Logs are generated for many reasons. Logs are generated not only to indicate the policy violations, intrusion detections, virus or spam detections, but also generated to indicate session information, configuration changes, login failures, system errors, system warnings etc.. Type of information in each of these types of log messages is different from each other. To facilitate analysis of logs by external log analyzers, each family of log messages should have its own format. That is, each family should have its own keywords to represent the values of different parameters. There could be many message families.

Each message family contains multiple different types of logs. For example, there could be multiple types of logs in intrusion message family. Typically, each signature in IPS is one type of log. In signature based IPS, there are as many logs as number of signatures. Each type of log message is typically represented by message IDs.

UTM devices generate logs during its operation. There could be large number of incidents which generate similar type of log multiple times. Messaging system component of UTM devices provide multiple controls for administrator to control logs. Some of the controls are:

• Message ID level control on enable/disable: If a message ID is disabled, any log incidents of that message ID are not processed, that is, they are not stored or exported to external log collectors.
• Message ID based Log frequency control: It allows logs, but controls the number of logs generated for processing. Typically it takes two parameters - Log threshold count and log threshold time. At most one log is processed for every 'log threshold count' logs or within the 'log threshold time'. That is, if log threshold count is 100 and log threshold time is 5 minutes, then if number of logs generated are more than 200, but less than 300 within 5 minutes, then it emits three logs - 1st log, 101st log and 201st log.
  
• 5 Tuple based Log frequency control: Message ID based log frequency control is good for non-network based logs such as system errors, warnings etc.. But, for connections, this kind of control is not good enough. If there is same intrusion detected in traffic going to multiple victims, then administrator would like to know at least one instance of intrusion going to each victim. With message ID based log frequency control, all victims will not be reported if the intrusions happen within 'log time threshold'. To avoid this, 5 tuple based log frequency control is required. 'Source IP', 'Destination IP' , 'Protocol', 'Source Port' and 'Destination Port' constitute 5 tuple. Each item in tuple can be enabled/disabled. 'Log threshold count' and 'log threshold time' described above is individually valid for each combination of 5-tuple items enabled. Let us say that, if administrator enabled 'source IP' and 'destination IP' addresses in the 5 tuple for a given message ID, then for logs generated for this combination of network connections are processed as per 'log threshold count' and 'log threshold time'. Using this, log systems don't miss attacks and other events happening on individual victim machines and also don't miss reporting 'attackers'.

In multi function security devices like UTM, there could be many message IDs. It is no surprise, even the number of message IDs are in the tune of 1000. Controlling individual message ID would be night mare for administrators. UTM devices need to provide control of logs at higher level than the control on message ID basis. That is where, sub-family of message IDs come in handy. Sub-family is nothing but group of message IDs. This grouping is mainly for log control. This grouping does not define the message format. The controls 'Enable/Disable', '5 tuple based log frequency control' can be specified on per sub-family basis. All message IDs inherit these controls. Of course, administrators are provided control on message ID basis too, when he/she thinks that some message IDs can't inherit the controls from their sub-families.

Though many aspects of this article is based on Intoto iGateway product family concepts, these are generic concepts and valid for any security products. The typical flow of logs is:

• Logs are generated by applications such as firewall, IPS, AV/AS, WAF etc..
• Log throttling system discards logs and processes only some logs based on controls configured by administrator.
• Logs are then stored, exported.
• Log analyzers (local or external) analyzes logs and even create reports. Also, they provide 'search' functions based on different criteria.

Logs are exported by either using 'syslog' or 'email'. iGateway UTM also has facility to store logs locally in a database (postGres).

Exported logs are sent in a format for log analyzers to easily extract values for different fields. WELF is one format that became quite popular. Though iGateway product family uses WELF syntax, but it does not use keywords as specified by WELF as they are incomplete. Some of the rules it follows are:

• Each log forms one line with each line containing multiple fields.
  
• Each field is formed as keyword=value. Keywords are defined by corresponding message family. Each keyword and value pair is separated by one or more spaces.
  
• keyword and values should not have any spaces. They must not have any quotes and '=' sign characters. If the value needs to have spaces, then the value string must be enclosed in double quotes.
• Mandatory keywords across all message families
• time: Date and time in double quotes.
  
• priority: Priority of the message. One of values from 1 to 7.
  
• id : Identity of the device sending the logs. It is configured by administrator on each device.
• mtype: Message family.
• mid: Message ID
  
• Generic keywords: There are some generic keywords which are valid across multiple message families. Though these are not mandatory, these generic keywords must be used wherever they are needed.
  
• vsg : Virtual instance.
  
• fromzone: Zone in which the connection is originated.
• tozone: Zone in which the connection is terminated.
• userid : User name, if this information is known.
• usergroup: User group.
  
• sip : Source IP address in dotted decimal form.
• dip: Destination IP address in dotted decimal form.
• protocol: Protocol of the connection as integer.
• sport: Source port
• dport: Destination port.
• Other keywords are specific to each message family.

PCI DSS - UTM technology requirements

Many security vendors claim that their products can be used to comply with PCI DSS requirements. But, it is difficult to find exactly what is it one should look for in these kinds of products. I try to address this in this blog entry. My focus in this article is on network firewall, IPS, VPN , AntiVirus and web application proxy firewall products.

PCI DSS (Payment Card Industry Data Security Standard) defined 12 requirements. For more information on these requirements, please refer to PCI DSS standards. Those 12 requirements are:

• Requirement 1 - Install and Maintain a firewall configuration to protect cardholder data.
• Requirement2 - Don't use vendor-supplied defaults for system passwords and other security parameters.
• Requirement3 - Protect stored cardholder data
• Requirement4 - Encrypt transmission of cardholder data across open, public networks.
• Requirement 5 - Use and regularly update anti-virus software or programs.
• Requirement 6 - Develop and maintain secure systems and applications.
• Requirement 7 - Restrict strong access to cardholder data by business need-to-know.
• Requirement 8 - Assign a unuqie ID to each person with computer access
• Requirement 9: Restrict physicall access to cardholder data
• Requirement 10: Regularly monitor and test networks
• Requirement 11: Regularly test security systems and processes
• Requirement 12: Maintain an Information Security Policy

Security infrastructure products like firewall, IPS, VPN, Web application firewalls are used to satisfy the PCI DSS requirements.

Expectation from Network based Firewalls:

• Ability to support multiple zones: Cardholder data servers must be separated from the frontend servers such as Web Servers. Thereby, any compromised front server does not give access to cardholder information to attackers. Typically, frontend servers are kept in DMZ of firewall. Cardholder information must be kept in other zones. Firewall should have facility to define new zone and firewall must have facility to setup rules for each zone. All cardholder servers can be kept in the zone separate from 'corporate zone', 'dmz zone' and 'untrusted zone'.
  
• Ability to create access control policy rules: Firewalls must support creation of rules among zones. For example, access to cardholder data servers should only be allowed from 'frontend servers' in 'DMZ' zone. In addition, only specifc ports should be allowed. Firewalls must have capability allowing rule creation with 'From' and 'To' Zones, 'From' and 'To' IP addresses and Service Ports.
• Ability to define rules with named objects: Firewall rules can become pretty complex. For readability and manageability, administrators would like to create rules with named objects (IP address objects and Service Objects).
  
• Ability to generate logs and alerts: Any attempt to card holder data other than the allowed policy rules must be notified. It is also required that firewalls have ability to generate log events for each connection that was allowed. Firewalls should send enough information along with the notification for analysis.
  
• Ability to detect and protect from DOS & DDOS attacks.
• Ability to hide internal IP addresses using NAT capability : Any outbound connection from any zone should hide its internal addresses.
• Factory defaults:
• 5 zones - DMZ for front end servers, PCI for payment systems, CORP for internal users, VISITOR for visitors and MGMT for management traffic.
• ACL rules:
• Allow only DMZ to PCI traffic. Deny PCI to ALL zone traffic.
• Allow only HTTP and HTTS traffic from ANY to DMZ.
• Allow only Enterprise protocol from CORP to ANY and VISITOR to ANY.

Expectation from Network based Intrusion Prevention Systems:

• Ability to protect all services of machines in corporate network, card holder data zone, DMZ zone from attacks exploiting known vulnerabilities: IPS devices are expected to have up-to-date signatures to protect internal resources from exploiting known vulnerabilities.
  
• Ability to differentiate from patched and un-patched services in internal machines: IPS devices are expected to have facility to control generation of logs based on targeted system version.
  
• Ability to detect and stop connections and packets transporting malware to internal network: IPS devices are expected to have up-to-date signatures to detect malware by monitoring the connections and packets.
  
• Ability to detect infected machines in internal zones by monitoring the traffic and connections from the internal machines to other zones: This typically happens if an infected system was brought into the network. IPS devices are expected to detect these machines by monitoring the traffic using up-to-date signatures.
  
• Ability to provide signature additions and editions: Beyond signatures which are auto updated, IPS systems are expected to provide facilities to create custom signature rules. It facilitates the administrator to create rules which are specific to the deployment. Administrator can create rules to detect default passwords of the applications and other characteristics of applications. Also administrator can create rules to detect specific information about cardholders.
  
• Ability to detect protocol anomalies of HTTP.
  
• Ability to detect traffic anomaly: Especially this is required to detect anomaly between frontend servers and cardholder data servers. Administrator can setup normal traffic profile for day and night and check for any anomaly. IPS deviecs are expected to provide this function.
• Ability to detect and prevent from SQL, XSS injections: IPS devices are expected to monitor HTTP POST values, GET values and XML data to detect SQL or XSS injections and check for any blind SQL and XSS injections using its up-to-date signatures.
• Ability to generate logs and alerts: IPS devies are expected to generate logs with comprehensive information for administrators to analyze the issue and take corrective actions.

Expectation from IPSec VPN:

• Ability to create Site-to-Site tunnels: Yet times, frontend servers and cardholder data servers may not be in same location. In these cases, it is expected that Ipsec VPN tunnels are used to transport data among these servers to provide data security on wire.
• Ability for Remote Access VPN: Administrators and other privileged users may need to get to the cardholder data servers remotely. IPSec VPN component of security appliance is expected to allow remote access via VPN tunnel.
• Ability to create new key generation periodically ( as small as 5 minutes).
• Ability to create IKE policies without preshared keys. When preshared keys are used, IPsec VPN should check for the strengh of the preshared keys.
• Atleast one digit.
• Atleast one non-alphabet and digit.
• Minimum length of 6 characters.
  
• Ability to support EAP authentication in remote Access VPN cases.
  

Expectation from Network based AntiVirus:

• Ability to scan email/HTTP traffic for virus and stop virus reaching internal machines: Internal machines are eventually used to access cardholder servers. Network based Virus scanning is one way to stop viruses reaching the internal servers and machines. Note that, network based AV can only do limited job. It is required that host based AV software is installed on all endpoints and servers.

Expectation from Web Application Firewalls:

• Ability to look into SSL traffic: All transactions are based on SSL. Any deep inspection requires decoding SSL traffic.
• Ability to provide HTTP protocol anomaly detection
• Ability to detect exploit traffic
• Ability to detect SQL and XSS injections.
• Ability to provide access control for different services running on HTTP.
• Ability to look deep into XML for SQL, XSS, command injections.
  
• Ability to generate log events and alerts.

Since these security devices are part of merchant network, these also must adhere to generic requirements of PCI DSS. They are:

• Configuration management of these devices should only be allowed by privileged users.
• Auditing of configuration changes: If there are multiple administrators, then each administrators must have unique ID. Each change in the configuration must be logged with information such as 'name of user' , ' change made', 'time & date of change' etc..
• Security devices also can be vulnerable. It must have facilities to upgrade new firmware.
• Mandate change of default passwords first time administrators log in, if local user database is used.
• Check for strong passwords, if local database is used.
• 6 Character length.
• Atleast one digit
• Atleast one non-alphanumeric character.
  
• Security devices predominantly should show alert message in their home page of default passwords.
• Mandate the password change if unchanged for a month or as per company policy.
  

Why is IKEv2 better than IKEv1?

IMO, any new deployment should go with IKEv2. It provides several advantages over IKEv1.

• IKEv2 is light on bandwidth and faster
• Less number of messages to establish tunnel.
• IKEv2 provides inbuilt NAT Traversal.
• IKEv1 does not provide this facility. But an internet draft was created to enhance IKEv1 with this functionality. Since this draft is not standardized, there may be interoperability issues.
• IKEv2 has inbuilt tunnel liveness checks.
• If tunnel is broken down on peer, it has facility to detect and re-establish the tunnel.
• IKEv1 does not have this functionality. There is an internet draft available though.
• IKEv2 provides comprehensive authentication capabilities.
• It supports Pre-shared key authentication, certificate authentication. IKEv1 also has them.
  
• More importantly, it provides EAP authentication and hence it is suitable to integrate with existing authentication systems in Enterprises. IKEv1 does not have this capability.
• IKEv2 has companion document to work with changing IP addresses on devices .
• MOBIKE standard is only supported on IKEv2.
• IKEv2 has facility to negotiate multiple sets of selectors.
• Many networks/ranges can be negotiated in one exchange. Hence, number of policy records can be very less when sites have multiple networks.
• In IKEv1, each pair of networks need to be defined in one policy record in SPD.
• IKEv2 has clear method to choose subset of selectors when both sites are not configured with exact selector values.
  
• In case of mismatch, IKEv2 has better mechanisms to converge.

If you are newly deploying IPsec gateways or thinking of upgrading Ipsec based security gateways, consider using IKEv2.

IPsec and DDNS - Where do they meet?

If some one tells you that you require permanent public IP address for connecting offices via IPsec tunnel, then he/she is not up-to-date with the latest offerings.

It is true that original IPsec security gateways do require static public IP address for establishing tunnel. With Dynamic DNS capability built into these gateways, this is no longer the case. As long as there is public IP address (static or dynamic), IPsec tunnel establishment is possible. Both sites participating in the tunnel can have dynamic public IP addresses.

What is Dynamic public IP address?

IP address assigned by ISP is not constant. ISPs can change the IP address that is being assigned from time to time. More often, every time CPE device reconnects (in case of PPP) or require new lease (in case of DHCP), ISPs provide new IP address.

What dynamic DNS?

Dynamic DNS provides fixed domain name, but changing IP address. There are large number of providers (One example: www.dyndns.org) provide services to register for domain name and change the IP address for this domain name through proprietary but well published protocols. CPE device, whenever public IP address changes informs dynamic DNS providers through the protocol. Any peer who needs to communicate with the CPE device will get the latest IP address via DNS protocol. Only limitation is that all peers must communicate with the CPE using domain name.

IPsec and DDNS:

IPsec as an application can make use of domain name to communicate with the peer. IPsec component must have domain name resolution facility to get IP address of peer before connecting to it via IKE. IPsec tunnel establishment can happen even if both the sites have dynamic public IP addresses as long as they have DynDNS capability.

With gateway adoption feature as described in one of previous blog entries, tunnel need not get torn down even if the IP address changes, thus providing seem less connectivity.

Fragmentation before encapusulation (Red side fragmentation) - IPsec

I don't know why it is called 'red side fragmentation'.

What is red side fragmentation?

Fragmentation of IP packets is done before ESP/AH/IPCOMP encapsulation in IPsec such that there is no fragmentation required to transmit the encapsulated packets.

Why is this required?

I believe some of service providers tend to give less priority to packets that are fragmented. When they detect congestion, fragmented packets are given lesser priority and they may get dropped.

IPsec carries many important protocol traffic across sites. Any IPsec packet drops cause degradation of application performance. IPsec gateways need to ensure that this traffic is not dropped. As you are aware, IPsec adds additional encapsulation headers such as ESP/AH/IPCOMP and another IP header in case of tunnel mode. It increases the packet size by few more bytes. If big size packet is received from local hosts and any additions to this packet requires fragmentation. If the fragmentation is done after encapsulation, the resulting packets have fragmentation variables set (MF bit and Offset bit) in IP headers. To avoid this, fragmentation is done on clear packets and IPsec encapsulation, outer IP header insertion is done on each of clear fragments. In this case, the outer IP header does not have fragmentation variables set. By this, service providers don't give lesser priority to these packets upon congestion.

Many security devices supporting IPsec have this feature. As an evaluator of security devices, you should look for this feature.

Encapsulation-less Route based IPsec VPN

Some one who read my blog asked me a question on mechanism to identify right Security Policy entry, now that there could be multiple policy entries with the same selectors.

In a deployment where there are multiple branch offices, head office security gateway has as many Encapsulation-Less Routerbased VPN (ELR VPN) interfaces as number of branch offices. It is quite possible that each branch office has different security requirements, hence there would be multiple SPD policy entries in head office router. When a branch office router initiates the IKE exchange, head office securty router should identify the right SPD (Security Policy Database) policy entry to negotiate security parameters for data Security associations. In typical policy based VPN and route based VPN, the SPD policy record selection is done based on selector values being negoatiated. But in Encapsulation-less Route based VPN (ELR VPN), selectors are always 0.0.0.0. Hence, this can't be used to select the SPD policy record.

ELR VPN implementation must use some other paramters to select the SPD policy entry. At the same time, implementations should not be creating proprietary extensions to IKE. So, we decided to go with 'remote ID'(ID of the initiated party) as index parameter to select right SPD policy record from SPD policy database.

In above example, each ELR VPN is associated with branch office identities. When IKE exchange is initiated by branch office router, it sends its identitity. Using this identity, head office router selects policy record that corresponds to branch office.

Hope this helps.

Route based VPNs - Explained

IPsec standard predominantly talks about policy based VPN. Policy based VPN contains multiple policy records, with each policy record having source, destination networks/hosts, service port and security paramters such as encryption, authentication algorithms etc.. These records are arranged in ordered list. Packets are matched against the policy table. If there is a policy record match, appropriate secuirty is enforced on the packets such as encryption and adding additional headers to reach the remote gateway. Remote gateway decrypts the packets and sends clear packets to intended host.

Thought policy based VPN gives granular control for administrators, it has its own disadvantages. If pair of security gateways have multiple networks and services for which data security need to be applied, then policy record(s) must be configured with these networks on the gateways. Every time new network is added in a site, policy records should be updated. Configuration update should be done not only on local gateway, but also in remote gateways. Source IP selector of policy record gets modified with new network on local gateway and Destination IP selector needs to get modified with new network on remote gateway. Admins not only need to configure routes to reach new networks in remote gateways, but also add or modify IPsec policy records. Since networks are to be added to the policy records explicitly, dynamic routing protocols also can't be used across sites.

Route based VPN solves above problems. In route based VPN, a point-to-point L3 interface is created and all traffic sent to this interface are tunneled to the remote gateway. As many interfaces are created as number of remote gateways. For a given pair of gateways, only one tunnel is created. Once this is done, administrator only needs to add routes to remote networks via tunnel interfaces. If dynamic routing protocols are used, admin need not even create routes explicitly.

Multiple types of route based VPNs are implemneted by appliance vendors. They are:

IP-in-IP route based VPN: Interfaces are created with IP-in-IP. IP packets are encapsulated with new IP header. Outer IP header IP addresses are gateway IP addresses. One IPsec policy record is created with these two gateway IP addresses as selectors. Any packets sent to this interface is encapsulated with outer IP header and then IPsec processing with ESP/AH encapsulation is done. When the encrypted packets are received, packets are decrypted, outer IP header is removed and then internal packet is routed to intended host.

GRE Route based VPN: It is similar to IP-in-IP route based VPN, except that it gets encapsulated with GRE+IP header. IPsec policy is created with GRE protocol and gateway IP addresses.

Encapsulation-less route based VPN: In this mode, there is no additional encapsulation such as IP-in-IP or GRE. Only IPsec ESP tunnel is used to encapsulate the packets with gateway IP addresses. In this case, Ipsec policy record is created with Source IP and destination IP selectors as 0.0.0.0 (ALL). So, selector negotiation happens with 0.0.0.0 IP addresses.

Since L3 interface is created, any dynamic routing protocols such as RIP, OSPF and BGP work on these interfaces and don't require any changes to the routing protocol implementations.

Mobiles - Secure Remote Access & firewalls in middle

It is well known that IPsec with IKEv2 is typically used to tunnel remote user traffic to Enterprise networks with IPsec client running in laptops and IPsec Gateway installed in Enterprises.

Mobiles are being equipped with IPsec client to access company network resources remotely. Wifi enabled mobiles can be used to access company resources from homes, using visitor networks of other companies, from airports, from coffee shopts etc..

IPsec uses UDP 500, UDP 4500 for key exchange. Data packets are sent using UDP 4500 or IP protocols ESP, AH, IPCOMP. These ports and protocols must be opened in firewalls that are between mobiles and Enterprise IPsec gateways. Without these ports open, traditional IPsec communication does not work.

It appears that some administrators are willing to open UDP 500 and 4500 ports permanantely, but unwilling to open ESP, AH and IPCOMP protocols. Due to this, I have a feeling that eventually all IPsec communication from mobiles would happen on UDP 4500, even if there is no NAT device in between.

In some cases, administrators are unwilling to open any outbound UDP port other than DNS port. If mobiles are behind these firewalls, using SSL (TCP 443) would be a good choice. Since SSL is dependent on TCP, this type of communication is not ideal for real time traffic such as voice and video.

IMO, Mobiles in future will have both IPsec and SSL VPN tunnel support. I propose Firewall traversal discovery mechanism that can be used to select either IPsec or SSL VPN tunnel. Firewall discovery involves both client and server - Client residing in mobiles and Server residing in Enteprise gateway. Client sends UDP 500 or UDP 4500 'echo request' message to find out reachability with Enteprise gateway. Server is expected to send 'echo reply' message on the UDP session upon receiving 'echo' message. If client receives 'echo response' message, it can use Ipsec tunnels. Otherwise, Mobiles need to fallback to SSL VPN tunnel. Firewall discovery echo messages must use its own 'payload' types to avoid conflicts with existing payload types in IKEv2.

Summary:

• Secure Remote Access from Mobiles to Enterprise networks will happen on either IPsec tunnel or SSL tunnel.
• Firewall traversal discovery mechanism would be used to select IPsec tunnel or SSL tunnel.
• IPsec tunnel is preferred over SSL tunnel.
• IPsec tunnels would use UDP encapsulation even if NAT devices are not detected.

Mobility with IPsec

IPsec protocol is being increasingly used by Enterprises to provide secure remote connectivity to internal networks thereby enabling access to internal resources to 'on road' employees and tele-commuters. IPsec gateways assign private IP address and related information (IP address, Internal networks, DNS Server and WINS Sever IP addresses) to the remote clients. Note that this is in addition to the IP address provided by service providers. Applications use private IP address to communicate with Enterprise servers and ISP provided IP address (Outer IP address) is used to tunnel the traffic securely to Enterprise gateway.

Since applications on client use private IP address, any change in the ISP provided IP address does not destroy the IP connections to Enterprise networks. This is in particular helps mobile users. While traveling by car, Mobiles may get different IP addresses across different point of attachments to the cellular network. As long as private IP address does not change, there is no disconnection to voice calls and data sessions. These sessions and calls work even in cases where mobile point of attachment changes from cellular to wi-fi and vice versa.

When outer IP address changes, IPsec client creates new tunnel to the IPsec gateway. Tunnel establishment typically takes hundred of milliseconds to a second. Though this is fine for data sessions, it introduces big jitter and that may not be acceptable for voice and other real time multi-media applications. At Intoto, we have created a proprietary solution (We call it IPsec address adoption) to adopt the IP address change in existing tunnel, there by avoiding new tunnel creation. Both IPsec Gateway and IPsec client adopt to change in IP address and change the tunnel addresses securely, avoiding expensive tunnel creation and thereby providing smooth transition.

Assumptions and Limitations of 'Address adoption' technique:

• It always uses tunnel mode.
• Forces NAT-T even when there is no NAT device in between
• Always does UDP encapsulation.
• Always sends Keep alive messages.
• Since it detects the address change based on incoming packets, it is assumed that there is traffic in both directions.

One of the advantages of 'Address adoption' solution is that it works with both IKEv1 and IKEv2. This solution is simple compared to MOBIKE. Hence, this solution can be called 'SMOBIKE' (Simple MOBIKE).

In summary, IPsec with its inherent capability of tunneling, provides seamless mobility to Enterprise resources. With additional Intoto proprietary address adoption technique, it reduces jitterness in the real time traffic.

Srini