IP Fragmentation versus TCP segmentation

Ethernet Controllers are increasingly becoming more intelligent with every generation of NICs.  Intel and Broadcom have added many features in Ethernet NIC chips in recent past.  Multicore SoC vendors are adding large number of features into Ethernet IO hardware blocks.

TCP GRO (Generic Receive Offload - It used to be called Large Receive offload too) and GSO  (Generic Segmentation Offload and it is used to be called Transport Segmentation Offload)  are two new features (in addition to FCoE offloads) one can see from Intel NICs and many Multicore SoCs.  These two features are  good for any TCP termination applications on the host processors/cores.  These two features reduces the number of packets traversing the host TCP/IP stack. 

TCP GRO works across multiple TCP flows where it aggregates multiple consecutive TCP segments (based on TCP sequence number) of a flow into one or few TCP packets in the hardware itself, there by sending very few packets to the host processor.  Due to this,  TCP/IP stack sees  fewer inbound packets.  Since the packet overhead is significant in TCP/IP stacks, lesser packets uses lesser number of CPU cycles, thereby leaving more CPU cycles for applications, essentially increasing the performance of overall system.

TCP GSO intention is similar to TCP GRO,but for outbound packets.  TCP layer typically segments the packets based on  MSS value. The MSS value is typically determined from PMTU (Path MTU) value.  Since TCP and IP headers take 40 bytes of data,  MSS is typically ( PMTU -  40 ) bytes.  If PMTU is 1500 bytes, then the result MSS value is 1460. When the application tries to send large amount of data,  then the data is segmented into multiple TCP packets where each TCP payload carries up to 1460 bytes.  TCP GSO feature in the hardware eliminates the need for TCP layer to do the segmentation and thereby reduces the number of packets that traverse between TCP layer and to the hardware NIC.  TCP GSO feature in the hardware typically expect the MSS value along with the packet and it does everything necessary internally to segment and send the segments out.

Ethernet Controllers are increasingly providing support for IP level fragmentation and reassembly.  Main reason is being  increasing popularity of tunnels.

With increasing usage of tunnels (IPsec, GRE, IP-in-IP,  Mobile IP, GTP-U and futuristic VXLAN and LISP), the packet size is going up.  Though these tunnel protocol specifications provides guidelines to avoid fragmentation using DF bit and PMTU discovery,  it does not happen in reality.  There are very few deployments where DF (Don't Fragment bit) , which is required for PMTU discovery, is used.   As far as I know,  almost all IPv4 deployments fragment the packets during tunneling.  Some deployments configure network devices to do red-side fragmentation (fragmentation before tunneling so that the tunneled packets appear whole IP packet) and some deployments go for black-side fragmentation (fragmentation after tunneling is done).   On receive direction, reassembly happens either before detunneling or after detunneling. 

It used to be the case where fragmented packets are given lesser priority by service providers during network congestion.  With high throughput connectivity and increasing customer base for networks, service providers are competing for the business by providing very good reliability and high throughput connectivity. Due to popularity of tunnels,  service providers are also realizing that dropping fragmented packets may result in bad experience to their customers.  It appears that service providers are not treating the fragmented packets in a step-motherly fashion anymore.

IP fragmentation and TCP segmentation offload methods can be used to reduce the number of packets traversing the TCP/IP stack in the host.  Next question that comes to mind is how to tune the TCP/IP stack to use these features and how to divide the work  between these two HW features. 

First thing to tune in the TCP/IP stack is to remove the MSS dependency on PMTU.  As described above, today MSS is calculated based on PMTU value. Due to this, IP fragmentation is not used by TCP stack for outbound TCP traffic. 

TCP Segmentation adds the both TCP and IP header to each segment.  That is, for every 1460 bytes, there would be overhead of 20 bytes of IP header and 20 bytes of TCP header.  In case of IP fragmentation,  each fragment would have its own IP header (20 bytes of overhead).  Since TCP segmentation has more overheads,  one can say IP fragmentation is better.  Here, MSS can be set to a bigger value such as 16K and let IP layer fragment the packet if the MTU value is less than 16K.   This is certainly a good argument and it works fine in networks where the reliability is good.  Where the reliability is not good,  if one fragment gets dropped, TCP layer needs to send entire 16K bytes in retransmission.  If TCP had done the segmentation, it would only need to send fewer bytes. 

There are advantages and disadvantages with both approaches. 

With increased reliability of networks and with no special treatment on fragmented traffic by service providers,  IP fragmentation is not a bad thing to do.  And ofcourse, one should worry about retransmissions too. 

I hear few tunings based on the deployments.  Warehouse data center deployments where the TCP client and servers in a controlled environment are tuning MSS to 32K and more with 9K (jumbo frame) of MTU.  I think that , for 1500 bytes MTU,  going with 8K of MSS may work good.

Residential CPE Devices - Conditional DHCP Server, IPv6 Prefix Inheritance from WAN and Network Label

Even though there are large number of IP addresses in Ipv6 world,  RG environments would still get the IPv6 addresses from ISPs dynamically.  There are advantages of doing this.  Home users don't have to be worry about renumbering their routers and internal machines in the home LAN when they switch to new service provider.  In general, it also reduces the amount of configuration one needs to make on the router.

As explained briefly in the article, WAN interfaces of CPE are configured to get the IP prefixes from the service providers.  These IP prefixes are programmed automatically in the DHCP Servers of the CPE. DHCP Servers in turn assigns IP addresses from these prefixes to LAN machines, Media Servers, NAS Servers, VOIP terminals etc.. 

In case of IPv4, this is done somewhat differently.  Service providers don't provide the IP addresses needed for the local LAN machines via WAN interface.  Home user is expected to configure private IP address range in the DHCP Server.  Outgoing traffic would undergo NAT with public IP address given through WAN connection.

In IPv4 world,  different IP address ranges (pools) can be assigned to DHCP Server and provide IP addresses from different pools based on conditions - DHCP User Class & Vendor Identifier Class options values. This is done to identify different types of devices in the LAN for providing differential treatment by CPE functions such as Security functions and QoS functions.  For example,  VOIP TA boxes can be served IP addresses from a separate pool of IP addresses.  This pool of IP addresses can be used in QoS rules to provide higher priority for the traffic coming from VOIP boxes while forwarding the traffic onto bandwidth constrained WAN interfaces. Administrator (home user) configures both DHCP conditional pools as well as QoS policy rules.

In IPv6 world,  there is no NAT.  And administrator does not configure the DHCP IP address pools - Whether it is general pool or conditional pools.  These IP prefixes are inherited from the dynamically assigned prefixes by Service provider. How does administrator configure security or QoS function to provide differential treatment on the traffic coming from different types of machines in the LAN, if he/she does not know  a priori  IP addresses that get assigned to different types of devices?

Fortunately, there is a way.  Many security and QoS policy rules not only take immediate IP addresses as source or destination IPs, but also they take named objects - Network objects. 

How does this work?

• CPE software should provide facility for administrators to enter network object names in DHCP common and conditional pools.
• CPE Software is expected to create this object when configured.
• When WAN Interface gets dynamic IP prefixes from ISPs,  it informs the LAN Device to inherit the prefixes and program the DHCP pools.  As part of this, CPE software divides the WAN prefix into multiple sub-prefixes and each sub-prefix is assigned to DHCP pools across multiple LAN devices.  As part of this assignment,  CPE software is expected to program the IP addresses in the corresponding Network object records.
• When related WAN interface  loses the connection, it is expected that CPE software removes the IP prefixes from the network objects too.
• CPE Software security and QoS functions would need to have facility to take network object names in its source and destination IP fields of policy rules.
• Since network objects are programmed with right IP prefixes,  Security and QoS functions would provide differential traffic treatment.

So, don't forget to add "network object record' to the DHCPv6 server pools while defining data model.

LAN & WAN Interfaces with IPv4 and IPv6 - Prelude to Data Model definition

In any network routing device, you would have LAN interfaces and WAN interfaces.  LAN interfaces are connected to networks with Desktops, Servers etc.. WAN interfaces are connected to network which is connected to Edge Routers or devices which are towards Internet.  In case of Edge router,  WAN interfaces are used to connect to service provider network.  In non-edge router cases, there can be WAN interfaces or in some cases there may not be any WAN interfaces.

So far the interfaces are used to connect to IPv4 network and hence used to accept only IPv4 addressing. IPv6 networks in Enterprises are becoming common.  Now interfaces need to be configured with IPv6 addresses and provide IPv6 addresses to local network machines.  Configuration is becoming complex. But understanding the concepts makes it easier.  This article tries to provide requirements from both IPv4 and IPv6 perspective on LAN and WAN interfaces.  I hope that this article is useful for both developers and administrators.

LAN Interfaces:  Routers have following types of LAN interfaces that take IP addresses.

• Ethernet Interfaces :  Some Ethernet interfaces may become part of Bridge interface. If the interface becomes part of bridge i.e bridge port, then it is no longer called LAN interface. It is simply called bridge port.  Similarly, if the interface becomes part of bonding interface, then also the interface is not called LAN interface.
• Bonding Interfaces:  Multiple Ethernet interfaces are bonded together into one interface.
• VLAN interfaces :  This is reverse of Bonding interface. Here one Ethernet interface is divided into multiple LAN interfaces. VLAN ID is used to de-multiplex incoming traffic to different interfaces. Since VLAN is used,  these interfaces are called VLAN interfaces.
• Bridge Interfaces :  Multiple Ethernet interfaces become a  bridge interface using 802.1D protocol. These are also LAN interfaces.

LAN interface requirements:

• Multiple IPv4 addresses (Address, Subnet) can be configured to enable multiple IPv4 networks on the same physical LAN. 
• DHCP IPv4 Server Configuration :  LAN interface can be configured to serve the IP addresses to machines in the LAN.  There are multiple requirements here.
• There are multiple different types of LAN machines -  VOIP phones, Media Servers,  Desktops, Laptops,  Smart phones etc..  Each type of machine might have different kinds of QoS requirements.  By providing IP addresses from different ranges of IP addresses to each type of LAN machines, QoS policies can be configured easily by having QoS rules with appropriate IP address range.  It is my understanding that different types of machines send 'Vendor Class identifier' differently.  This can be used to select the IP address range to server IP address.  There are other options that can be used to select the IP address range. So, the DHCP Server configuration on the LAN interface should have facility to take multiple IP address ranges with associated DHCP Option values.  Ofcourse, it also should take default IP address range that can be used to serve IP addresses when the DHCP client sends options and values that don't match the conditions set on the server side.
• DHCP server is not only used to assign the IP address, but also other IP configuration such as DNS Server IP addresses, WINS Server IP address,  Default Router IP addresses etc..  Some of these can be configured manually at the server. But some of them might need to be learnt from the WAN connections.  But note that WAN connections may not be UP when the LAN machines connect to the DHCP Server.  In these cases, it is necessary that 'lease time' is set to very less time (such as few minutes) so that the client initiates the DHCP connection again.  When all the information is available with DHCP Server (ie when the WAN connection is UP), then it can give higher lease time to the DHCP clients. In IPv4 world, typically DNS Servers from WAN are not propagated to DHCP clients.  DNS Servers are configured in local DNS relay and provide local LAN interface IP address as DNS Server to DHCP Clients. There by,  there is no dependency on when the WAN connection is UP. 
• Yet times, there may be DHCP Server elsewhere.  In this case, it is possible to set up DHCP relay on the LAN Interface.  
• Some network devices also have DNS Proxy/Relay. In these cases, it is expected that the DHCP Server upon giving lease to a machine configures the FQDN with the given IP address in the DNS Proxy/relay. 
• In addition to configuration,  it is required that the device provides statistics information and listing of 'attached devices (dhcp leases).
• Dynamic Routing configuration:  In Enterprises, configuration of static routes in each device is discouraged.  Typically RIP or OSPF are used to learn the routes.  So, it is required to configure any thing necessary to enable routing protocol on the interface.
• Some operating systems don't give flexibility of configuring the name of interface. So, it is good if some facility is provided for administrator to configure interface label and let the operating system choose the interface name.  This label can be intuitive name.  Any other configuration (such as creating routes etc..) requiring LAN interface can be referred by 'interface label'.  
• Multiple IPv6 addresses can be configured statically.
• IPv6 address assignment : In IPv4 world, DHCP Server is only way to serve the IP address and other networking information to the machines in LAN.  In IPv6 world,  IP address information is served in two ways - DHCP Server and using SLAAC (State less Auto Address configuration).  
• SLAAC:   IPv6 prefixes to be advertised can be configured.  It is expected that the machines create its own IP address with this prefix and rest of it from the MAC address of the interface. Router advertises the prefixes in RA (Router Advertisement) messages.  Rest of networking information (such as DNS Servers etc..) is normally served via DHCP Server.  Since DHCP Server is not assigning IP addresses, this scheme is called DHCP Stateless configuration as described in RFC 3736.  In some deployments,  the prefixes that need to be advertised to the local clients need to be derived from the prefixes WAN connections gets from the ISP.  Since there could be many WAN interfaces,  there can be a requirement to configure the WAN interface label from which to derive the prefixes.
• DHCP Server configuration:  Here it is similar to IPv4 DHCP Server. There are some minor differences.  In Ipv4, all the time the IP address ranges are configured by administrator. But in this case,  IPv6 prefixes are learnt from the WAN connections. As indicated above as part of SLACC,  it may be required to configure the WAN interface label from which to derive the prefixes and other information. DHCP Server is specified in RFC 3315.
• As discussed above, if WAN connection is not UP, then the internal machines will not be advertised with the prefixes and hence internal machines may not be able to communicate among themselves. Note that Link local addresses are not expected to be used by applications. Link local addresses are expected to be used only for Neighbor discovery and Route discovery protocols.  It is not good if local machines can't communicate among themselves if there is no WAN connectivity.  Of course, there is no issue if the global prefixes are known and configured statically.  In other cases where WAN connectivity provides the prefixes,  a provision is made to assign ULA (Unicast Local Address) and is described in RFC 4193.  This particular ULA prefix configured should be same across the reboots of the CPE device.  Though the ULA prefix is generated using random number, it should be saved so that it stays across reboots.  Due to randomization,  this prefix may be unique, but there is no surety. Hence it is necessary that addresses starting with FC00:/7 are filtered out at the site boundary router towards Internet. But note that these can be used for inter-site VPN.  For all practical purposes, this is like any globally unicast prefix.  Note that, this address can co-exist along with other global unicast addresses which the router advertises to the local LAN machines.

 WAN Interface requirements:

• Multiple physical interfaces can be WAN devices.
• There are some kinds of WAN connections that require physical interface to be used such as PPPoE and normal IP connections.  There are some WAN connections which send data based on routing information such as IPSec-IRAC and PPTP. 
• Each WAN device might have multiple WAN connections. Each WAN connection itself becomes an interface. 
• Each WAN Connection can be  configured to make connections to ISP using one of following:
• IP Connection
• PPP - PPPoE, PPTP
• IPsec - IRAC
• Each WAN Device mostly would have statistics information and very less configuration. Statistics information mainly contains packets or bytes sent/received,  interface label etc.. 
• IP Connection Mode:
• WAN Device :  Identified by Interface label.  This connection uses this  WAN Device.
• IPv4 Addressing
• Sub Modes:  Static,  Dynamic.
• Static:  Multiple IPv4 addresses with each IPv4 address having associated Subnet prefix.
• Dynamic (DHCP Client) Mode:  It should request for IPv4 address, prefix,  DNS Servers, WINS Servers,  SNTP Servers. It is also should be possible for administrator to enter other options (for send and receive) such as Vendor Class Identifier.  The DNS Servers which it gets are typically programmed in DNS Relay.
• IPv6 Addressing :  This is some what complex compared to IPv4. 
• Sub Modes:  Static,  Dynamic Static IPv6 addresses can be configured. 
• In Dynanic mode,   it starts with SLACC (RFC 4862). If the upstream router indicates the address needs  using stateful  way(M flag), then DHCP client is initiated with IA_NA option.  DHCP Stateful addressing (RFC 3315) and DHCP Prefix Delegation (RFC 3633) is always required to get other networking information (DNS Servers, SNTP Servers, SIP Servers etc..).  Prefixes which it gets would be used to divide across multiple LAN interfaces.  The division to be used can be configured.  As in IPv4, it should also take configuration for options that need to be sent or received.  Note that DNS Server information may be used by LAN Device DHCP Server and hence it should be possible for the CPE device to program the LAN Device DHCP Server or SLAAC server with learnt prefixes and DNS Servers.  Note that, if IA_NA option is not fulfilled by the server, it should assign one of the IP address from delegated prefixes to the WAN interface.
• PPP Connection:
• Sub Modes:  PPPoE,  PPTP
• WAN Device Interface Label:  WAN Device to use. Valid only if it is PPPoE. In case of PPTP, it uses interface identified by routing entry which itself is found using PPTP Server IP address.
• Generic PPP Configuration required:
• User name, password in case of PAP/CHAP
• Other PPP information (Like MTU, MRU, Compression Control etc..)
• Sub Mode Specific configuration:
• In case of PPPoE:  AC Name, Service Name etc..
• In case of PPTP:   PPTP Server IP address and other information.
• IPv4 Addressing:
• Static or dynamic.
• It also can get DNS Server IP addresses. As in 'IP Connection' mode, these addresses can be programmed in DNS Relay.
• IPv6 Addressing
• Using PPP, only link local addresses are negotiated. 
• Using RA (SLAAC), it can get the Prefixes. If O flag is set, it gets DNS and other information via DHCP.
• If RA indicates M=1, then it tries to get the IP address using DHCP IA_NA.
• In any case, it initiates DHCP PD to get the prefixes.
• If IA_NA is not successful, it uses one IP from the prefixes and assigns to its interface.
• Only configuration require for above operation is to configure DHCP options to send and receive.
• IPsec-IRAC Mode (RFC 5739) :
• IPv4 Addressing:
• If enabled, it gets the IP address and gets used as NAT IP address.
• IPv6 Addressing:
• All IKE and SPD policy rule configuration is required to be configured.
• As part of IRAC, it is expected to get the IPv6 prefixes and DHCP Server IP address.
• Using  DHCP Stateless configuration, it gets other networking information.
• As in other modes,  it assigns the prefixes to LAN Devices (SLAAC and DHCP Server configuration of LAN Devices) and also programs the DNS Servers in DHCP Servers of LAN Devices. 
• Like any good data model,  there should be enough information provided to administrator :
• Statistics
• Dynamic information that is learnt.

This is only prelude to define the data model.  Expect detailed data model soon.

UDP Broadcast Relay : TR-069 Support

UDP broadcast relay functionality became very popular due to NetBIOS. Broadcast packets are used by NetBIOS for name resolution. Windows Network neighborhood is one functionality that makes use of NetBIOS name service. Due to broadcast functionality, NetBIOS name service works within subnet. If there are multiple subnets, then WINS Server is required. Broadcast relay functionality in routers separating subnets eliminates the need for WINS Servers. Name resolution using UDP broadcast relay function can even be extended to networks in remote offices by relaying broadcast packets over VPN tunnels.

UDP broadcast relay functionality in routers receives broadcast packets and send to other subnets by replacing destination IP of original packet with destination subnet broadcast address.

Since firewall/VPN gateways are also routers, this functionality is implemented in many firewall/VPN gateways. These gateways provide control for administrators on type of broadcast packets to relay and destination subnets to relay to. Multiple of these rules can be created for different types of broadcast addresses.

Configuration consists of set of rules. Each rule containing incoming braodcast IP address/interface and relay subnets/interfaces. Rules can be created and deleted by administrator.

TR-069 profile:

• internetGatewayDevice.security.VirtualInstance.{i}.UDPBroadcastRelay.{i} PC
• name: String(32), RW, Mandatory - Identification of broadcast relay rule. Once rule is created, this value can't be changed.
  
• description: String(128), RW, Optional
• enable: Boolean, RW, Mandatory: Value 1 indicates the record is enabled and 0 is used to disable.
• incomingBroadcastAddressType: String(16), RW, Mandatory - Indicates whether the broadcast address is represented as an IP address or Interface identifier. Takes one of the values "ipaddress", "interface".
• incomingbroadcastAddress: String(128), RW, Mandatory - Either dotted IP address or Fully qualified TR-069 instance of VLAN, LANDevice, WANPPPConnection or WANIPConnection etc.
• incomingbroadcastPort: Integer, RW, Mandatory - Destination Port of incoming broadcast packet.
  
• internetGatewayDevice.security.VirtualInstance.{i}.UDPBroadcastRelay.{i}.relayTo.{i} PC
• relayBroadcastAddressType: String(16), RW, Mandatory - Indiacates whether the relayTo broadcast address is specified as IP address or interface - Takes one of the values "ipaddress", "interafce".
• relayBroadcastAddress: String(128), RW, Mandatory - Either dotted IP address or fully qualified instance of interafaces from VLAN, LANDevice, WANPPPConnection or WANIPConnection.

In case of remote subnets, relayBroadcast is specified as remote subnet broadcast IP address. If the subnets are directly attached to the router, then interface names can be used in relayBroadcastAddress field.

Trace Route Diagnostics - TR-069 profile

Tracing the routing path to a given destination machine is very important diagnostic tool for administrators and service providers. TR-098 Amendment 1defined 'IP PING' diagnostics, but did not define trace route diagnostics.

Trace route utility is provided in many operating systems today. Its main purpose is to find out the route IP packets take to reach a specific destination. It gives indication on the routers in between and round trip time of each probe. This utility sends UDP, TCP or ICMP probes with small TTL and listen for ICMP 'time exceeded' reply. It starts with TTL 1 to find out the first hop, TTL 2 to find out the second hop and so on.

Trace route can happen by sending ICMP, UDP or TCP packets. Trace route utility provides this option for traversing through firewalls. When firewalls don't allow ICMP, trace route can be used with UDP or TCP.

Since, a given host given may have multiple outbound interfaces, trace route utility gives facility to use specific link for its source IP address to ensure that responses come back to this IP address. It also provides option of specifying the gateway to route the packets through a specific link of the host.

Trace route utility provides options such as:

• Destination Host IP address or FQDN.
  
• Use ICMP, UDP or TCP.
• Use IPv4 or IPv6
  
• Packet size : Valid in case of ICMP and UDP. Not applicable if TCP is chosen. TCP probe always goes with SYN flag.
• Port: Port number to use for destination port of the UDP or TCP probe. In case of ICMP, it is used as sequence number.
• TOS: TOS value to use in the IP header of probe packet.
  
• Link interface to use: Utility uses IP address of this link as source IP of the probe packet. if not specified, default is to use link determined by route.
• Gateway IP address: This is mainly to select the right link, if there are multiple links on the host. Default is 'based on route'.
  
• Maximum TTL: Indicates the maximum number of hops to discover. Default : 30
  
• Number of Probes: Number of probe messages to each hop. Default 3.
• Wait time: wait time to wait for the response to probe. Default : 5 seconds.
  
• Send interval: Interval between probe messages to a hop: Default 0 seconds.
  

Results of trace route utility can be represented as:

• Destination Host: For which trace route diagnostics was run.
  
• Number of hops.
  
• Sequence of hops. Each hop consisting of
  
• IP address of router: Display * if no response from that hop.
• FQDN of the router: By doing Reverse DNS lookup
  
• Probe1 round trip time in milli seconds.
• Probe2 round trip time in milli seconds.
• Proble3 round trip time in milli seconds.
• Rest of round trip times in comma separated string (upto 32 bytes).

With this TR-069 profile could be as follows:

• internetGatewayDevice.IPTraceRouteDiagnostics P
• diagnosticsState: RW, String, It takes values of "None", "Requested", "Completed", "Error_HostNameResolutionError", "Error_HopCuntExhausted": Similar to PING diagnostics stage.
  
• destination host : RW, String, 256 bytes max, IP address in dotted decimal form or fully qualified domain name.
• probeProtocolSupported: R, String, comma separated strings. "ICMP", "UDP", "TCP", this is the capability of device.
  
• probeProtocol: RW, String, Takes one of values of "ICMP", "UDP" or "ICMP".
  
• IPv4OrIPv6 : RW, String, Values are "IPv4", "IPv6"
• packetSize: RW, Integer, Not applicable in case of TCP.
• Port: RW, Integer, Destination Port to use. In case of ICMP, it is used as sequence number.
• TOS: RW, Integer
• LinkInterface: RW, String, Fully qualified instance from VLAN, LANDevice, WANPPPConnection or WANIPConnection etc.. IP address of this interface is used as source IP of the probe packet.
  
• gatewayIPAddress: RW, String, Dotted decimal form. It is to select the link.
• maximumHops: RW, Integer
• numberOfProbles: RW, Integer, Number of probes to use for each hop.
• responseWaitTime: RW, Integer, in seconds.
• sendInterval RW, Integer, in seconds.
• internetGatewayDevice.IPTraceRouteDiagnostic.response P
• hostName
• status: "Error_UresolvedHost", "Error_MaxHopCountReached", "Success"
  
• numberOfHopEntriesDiscovered: RW, Integer
• hopIPAddress: RW, String
• hopFQDN: RW, String
  
• probe1RTT: RW, Integer in milliseconds.
  
• proble2RTT: RW, Integer
• proble3RTT: RW, Integer
• OtherProbeRTT: RW, string, comma separated. Upto 32 bytes.

Virtualization and Zones in Secuirty Appliances - A brief

I see this question being asked frequently. There is confusion between virtual instance Versus security zone. This is my small attempt to give some brief on virtualization in security appliances. It is not meant to describe this feature in detail :-).

For a brief on Security Zones, please see this entry:
http://srini-addepalli.blogspot.com/2008/02/mutliple-security-zones-in-enterprise.html

Virtual Instances:
Newer Security appliances have this feature whereby one single hardware box supports multiple security instances. They are called virtual instances. Each virtual instance has its own configuration of security functions. Some security vendors call these instances as VSGs (Virtual Security Gateways). Note that, the term Virtual is not same as the terms used by Vmware and Xen. Each virtual instance has its own security zones, firewall policy configuration, IPsec SPD, IPS configuration and so on. For all practical purposes, you can assume that multiple traditional security appliances are kept in one single box with common user interface.

As you understand by now, security zones are part of each virtual instance.

Link layer Interfaces (Ethernet, Wireless etc..):
Each virtual instance has its own interfaces. A given interface can't be part of two virtual instances. Again, imagine multiple traditional physical security appliances. Each traditional appliance has its own Ethernet ports and wireless ports. Similarly, each virtual instances has its own interfaces. If the security appliance is supporting, say 16 virtual instances with each instance supporting 4 security zones, you require 16*4 = 64 link interfaces. This number goes up very high if more virtual instances or more security zones are required. To reduce the hardware cost of adding these many physical ports, vendors support VLAN based interfaces for virtual instances. 4K VLAN interfaces can be created on each physical port. Virtual instances (VSGs) can make use of VLAN interfaces to provide virtualization with less number of physical ports. In some cases, it is not possible to have VLAN ID to map to virtual instance. One example of this type is 'packets coming from Internet' in data center environment. In these case, virtual instance needs to be identified by the destination IP address of the packet. In these cases, the IP address in 'destination IP address feild' of IP header is used to identify interfaces.

Link interfaces are assigned to each Virtual instance and Security Zone. Interface on which packets are being received is used to determine the virtual instance and security zone.

Virtual Security Gateway feature, in my view, is mainly useful for service providers and data centers who want provide security services for business customers. Each customer is treated as one VSG. With one security appliance, SPs can provide security services for many customers, thereby reducing cost dramatically.

With this background, let me describe the relation between VLAN, VSG and Security Zones by taking an example. Let us assume that a 'data center' is hosting services for 64 customers. Data center provider wants to deploy Firewall/IPS appliances to protect servers of these 64 customers. Let us also assume that the appliance has two physical ports. One physical port is connected 'Server' network (Call it as 'Server Port) and another port is connected to untrusted network (Internet Port). With this scenario Admin of this box need to do following:

• Install 64 port switch - Connect Uplink port to security appliance's Server port.
  
• Configure the switch to add VLAN ID 1 to VLAN 64 to packets coming from 64 ports and redirect them to uplink port (VLAN switching).
  
• Configure VLAN module of appliance to create 64 interfaces on VLANA1 to VLANA64.
• Install another 64 port swtich - Connect Uplink port to security appliance's Internet Port.
• Create VLANB1 to VLANB64
  
• For all 64 customers
• Associate VLANAx to 'Server Zone' and VSGx
• Associate VLANBx to 'Internet Zone' and VSGx
• Add security service configuration for each Virtual instance.
• That is all is required I guess.

What it the configuration support required for this?

• VLAN Interface configuration. I covered this in before as part of this.
• Virtual instance administration
• It is set of records and each record having
• Virtual InstanceID.
  
• Virtual Instance name.
• Enable/Disable
• Description
• Zone Description
• Zone ID
• Zone name
• Interface mapping
• Set of records. Each record having
• Interface name(Could be VLAN interface name, bridge interface name or WAN interface name etc.. )
• VSG ID
• Zone ID

Enterprise security appliances typically don't need to support virtualization and hence it can be considered to have one virtual instance.

With above back ground, I feel that TR-069 model may look like this:

• internetGatewayDevice.security P
• maxZonesPerVirtualInstance (Read Only)
• maxVirtualInstances (Read Only)
• currentNumberOfVirtualInstances
  
• internetGatewayDevice.securityDomains.VirtualInstance.{i} PC
• ID
• Enable
  
• Name
• Description
• internetGatewayDevice.security.VirtualInstance.{i}.Zone.{i} P
  
• Zone ID
  
• ZoneName
• internetGatewayDevice.mapping.{i} PC
• InterfaceReference (Fully Qualified instance from VLAN table, LANDevice and WAN Link).
• VirtualInstanceID
• ZoneID

TR-069 Dynamic DNS (DDNS)

Let me give some introduction on DDNS before giving some ideas on DDNS data model.

Many small offices would not want to pay for static public IP addresses and yet host servers for public access. For public access of servers, it is necessary that reachability information is constant. DDNS functionality enables this. DDNS providers facilitate this by allowing CPE devices to advertise IP address changes to DDNS providers. DDNS providers internally update their DNS servers with this new IP address.

DynDNS is a popular protocol used by many DDNS providers. Though it is not a defined by IETF or any other standard body, this seems to be quite popular. DDNS providers such as www.dyndns.org and 3322.org use this protocol to provide dynamic DNS service. TZO is another protocol used to update the service. DDNS provider www.tzo.com provides DDNS Service using this protocol.

As an end user, following steps are to be followed to get hooked into this service:

• Create account with DDNS provider using their web site. You may be asked to provide your email address, user name and password.
  
• Register your domain names with the DDNS provider by visiting their web site or by using one of their clients.
• Configure your router with DDNS provider information to update IP address automatically.
• Set up your internal servers for public access.
  
• Configure your CPE router to forward the traffic to your servers.

CPE router configuration: CPE routers normally support multiple instances of DDNS - One instance for each WAN link. For example, If there are two WAN links, then CPE devices will have two DDNS client instances. Each client instance can update the IP address for multiple domain names. Each record would need to have following information configured.

• Name of the record : To identify the record.
• WAN interface (link): DDNS client monitors the IP address of this link. Wheneverthe IP address of the link is changed, then it starts the process of updating.
  
• Update time period: This time period indicates the periodic interval to update the IP address, even if the link address is not changed. This configuration parameter is not used by some DDNS protocols such as 'dyndns'.
  
• Domain name 1 : Name of the domain name that was registered.
  
• Domain name2
• Domain name3
• Domain name4
• DDNS protocol to be used: Protocol that is to be used to update IP address. 'dyndns', 'tzo', 'dhrp' etc..
• Provider details
• In case of dyndns, the additional configuration required to contact DDNS provider are:
• DDNS provider IP address or FQDN: Reachability information to reach DDNS provider.
• Relative URL (Script name): DynDNS protocol is HTTP or HTTPS based. This parameter indicates the CGI script to be used to send update information. Unfortunately, this name is not standardized. Different providers are using different script names. Hence, this should be taken as configuration parameter.
• Protocol : HTTP or HTTPS
• Port : Typically, it is 80 or 443.
  
• User name and password: User name and password used to create account with DDNS provider.
• Trusted Certificate in PEM form: When HTTPS used. This certificate is used to authenticate the DDNS provider to avoid MITM attacks.
  
• Note: DDNS client always should send MX=NOCHG and wildcard = NOCHG to ensure that configuration done using DDNS provider website is not erased.
• TZO provider information: As I understand, TZO protocol defines server discovery. I guess this is mainly for load balancing. It involves following stages - Getting IP addresses of clusters and getting IP addresses of update servers and then updating IP address to one of the update servers.
  
• ProviderIP address/host name (Cluster lookup host): Using this IPaddress/host name, the TZO client in the CPE gets the all IP addresses of cluster. TZO client makes a TCP connection for this purpose.
  
• Cluster lookup port: Port used by TCP connection to do cluster lookup. Default : 21340
  
• Update Server lookup Port: Once TZO client gets the cluster IP addresses, it is expected to choose one of them and make another TCP connection to get the list of servers that can be used to update the new dynamic public IP address. This port is used by client to make the connection. Default : 21344
  
• Email address: Mail address used when the TZO account was created.
  
• Key: Key generated when the TZO account was created.

Based on above explanation, the TR-069 data model for DDNS could be:

• internetGatewayDevice.DynamicDNS.{i} : PC - New instances can be created.
  
• Name
  
• Enable
• LinkReference : Fully Qualified name of WanDevice->WanConnectionDevice->WanIPConnection or WANPPPConnection.
• UpdateTimePeriod : In seconds.
• DomainName1
• DomainName2
• DomainName3
• DomainName4
• DdnsProtocol : Takes one of the values of 'dyndns', 'tzo'
• internetGatewayDevice.DynamicDNS.{i}.dyndns
• ProviderFQDN
• ProviderURLScript
• ProviderProtocolSupports: Read Only, String, It takes comma separated protocol strings such as http, https etc..
  
• ProviderProtocol : take 'http' or 'https'
• ProviderPort
• Username
• Password
• ProviderCACertificate : Valid only if 'https'
  
• internetGatewayDevice.DynamicDNS.{i}.tzo
• ProviderClusterLookupFQDN
• ProviderClusterLookupPort
• ProviderUpdateServerLookupPort
• RegisteredEmailAddress
• KeyProvided

IGMP Proxy

IGMP Proxy is defined in rfc4605.txt.

IGMP proxy is typically used in Edge routers - Either office edge such as gateway routers and provider edge such as DSLAM. One main point to keep in mind is that IGMP proxy is useful only in simple tree topology where tree can be configured manually. For complex trees where manual configuration is not possible, multi cast routing protocols such as PIM should be used. IGMP proxy is very simple in the sense that it acts as IGMP router for downstream interfaces and as a host on upstream interfaces. It listens for IGMP reports coming on downstream interfaces from hosts interested in listing to multicast traffic. It consolidates report information coming from different hosts across different downstream interfaces and send consolidatedreport to upstream routers.

In Office edge routers, the interfaces connected to inside machines are typically downstream interfaces and WAN interfaces connected to ISP routers are upstream interfaces. In case of DSLAM, WAN interfaces towards its customers are downstream interfaces and upstream interface is connected towards Internet.

Multicast stream providers get unique Multicast IP address for each type of stream from IANA. This IP address is advertised. Any machines willing to receive this stream send IGMP membership reports to routers indicating their willingness to receive packets coming on a particular multicast IP address. IGMP protocol is used to send membership reports.

IGMP proxy devices receives multicast stream from upstream interfaces, it knows the downstream interfaces interested in this traffic based on membership reports it received before. It duplicates the traffic and send it to these downstream interfaces. It provides great advantage on cost savings as only one copy of stream is sent on WAN link. Gateway implementing IGMP proxy duplicates on multiple downstream interfaces.

IGMPv3 support source specific multicasting. IGMPv3 hosts can request the stream only when it is generated from a particular source. It provides some kind of security for hosts (note that IP address can be spoofed by rogue multicast stream generator) and more importantly it eases the network routing problems. IGMP proxy should honor v3 memberships with specific source IP address. If proxy receives v2 or v1 membership from local hosts on the same interface, then source specific multicast membership from v3 hosts will not have desired effect.

As discussed before, IGMP proxy acts as IGMP router on downstream interfaces. As an IGMP router, it sends IGMP queries periodically to ensure that hosts are still interested in multicast streams. IGMP routers typically send generic query periodically. Group specific query is sent upon a state change such as host leaving the group.

IGMP Proxy takes responsibility of creating multicast routes based after consolidating membership reports.

With the above background, let me present the IGMP proxy configuration required in gateways.

• Enable/Disable: Whether IGMP proxy operation is to be enabled or disabled. If it is disabled, this router does not even entertain reception of IGMP messages.
• Maximum number of groups : This parameter represents the number of groups allowed by IGMP Proxy. Note that 'groups' have one to one association with multiple IP addresses.
• Robustness: Please rfc3376 for explanation. It is mainly intended for lossy network. This parameter value is default value for 'Startup Query Count' and 'Last Member Query Count'. The queries are sent as many times as 'robustness' count separated by 'Startup Query Interval' and 'Last Member Query internval respectively. Default value is 2 as per rfc3376.
  
• Query interval: The number of seconds between two IGMP general query messages. Default : 125 seconds as per rfc3376.
• Query response interval : Amount of time the querier waits for response to general query. Default is 100 (10 seconds). Each unit corresponds to 100milliseconds.
  
• Startup Query Interval : Number of seconds between general queries during startup.
• Startup Query Count: Number of queries sent out upon startup.
• Last Member Query Interval: Amount of time router can wait before sending group specific query or group-source specific query in response to leave group messages. Default is 10 (1 second). Unit is equivalent to 100 milliseconds.
  
• Last Member Query Count: Number of queries sent upon receiving leave group message.
• Host Unsolicited Report Interval: This variable used by host portion of IGMP proxy. This value is represented in seconds and is used by host to send reports. Host is expected to send 'robustness variable' number of reports within this time to the routers to cover the possibility of report misses by routers.
• Downstream interfaces: List of downstream interfaces. Interfaces are typically of type: wireless LAN, USB Ethernet, Ethernet, VLAN and PPP etc..
• UpStream interfaces: List of upstream interfaces.
• Log : YES or NO. IT departments always would like to know the network view. IGMP membership information provides valuable information about usage patterns etc.. To provide historical information to know the join and leave intervals of machines for different groups, it is required that log entries are generated. This variable controls whether log is to be generated or keep quiet. Log entry should contain
• Machine IP address
• Group IP address (Multicast IP address)
• Time at which it joined or time at which it left.
• Interface used for join or leave.

Run time information: Multicast member ship view is very important for administrators to know what is happening at any given time. One would like to know things like - Machines and their membership information and interface information on which the machines are present etc.. To be precise:

• For each group in IGMP
• Group Address (Multicast address)
• For each downstream interface
• Interface name
• Time at which this group was added.
• Source IP addresses included.
• Source IP addresses excluded.
• Machines that are interested in this group.
  

• Statistics :

• Number Of General Queries Received,
• Number Group Specific Queries Received,
• Number Of Group & Source Specific Queries Received,
• Number Of Queries Transmitted
• Number Of V1 Reports Received,
• Number Of V2 Reports Received,
• Number Of V3 Reports Received,
• Number Of Leave messages Received,
• Number Of Reports Transmitted,

With above in mind, TR-069 model can be represented as:

• internetGatewayDevice.IGMPProxy P
• Enable : RW, Integer.
• MaxNumberOfGroupsAllowed: R, Integer
• LogEnable: RW, Integer.
  
• RobustnessValue: RW, Integer
• QueryInterval: RW, Integer
• QueryReponseInterval: RW, Integer
• StartupQueryInterval: RW, Integer
• StartupQueryCount: RW, Integer
• LastMemberQueryInterval: RW, Integer
• LastMemberQueyrCount: RW, Integer
• LastMemberQueryCount: RW, Integer
• HostUnsolicitedReportInterval: RW, Integer
• internetGatewayDevice.IGMPProxy.Interface.{i} PC
  
• InterfaceReference: RW, String : Once assigned, can't be changed. This full qualified name of interface instance of LANDevice, WANIPConnection, WANPPPConnection, VLANInterface.
• InterfaceType: RW, String. "UPSTREAM", "DOWNSTREAM". Once set, it can't be changed.
• internetGatewayDevice.IGMPProxy.GroupInfo.{i} P
  
• GroupAddress: R, String.
  
• internetGatewayDevice.IGMProxy.GroupInfo.{i}.interfaceInfo.{i} P
  
• InterfaceReference: Read Only, String.
• Time : At which group is added. Date & time. Read Only, String type.
• IncludedSourceIPs: R, String. Comma separated IP addresses.
• ExcludedSourceIPs: R, String. Common separated IP addresses.
• internetGatewayDevice.IGMPProxy.statistics P
• internetGatewayDevice.IGMPProxy.statistics.hostside P
  
• NumberOfGeneralQueriesReceived : R, Integer
  
• NumberOfGroupSpecificQueriesReceived : R Integer
  
• NumberOfGroupAndSourceSpecificQueriesReceived : R, Integer
  
• NumberOfReportsTransmitted: R, Integer
  
• internetGatewayDevice.IGMPProxy.statistics.routerside P
• NumberOfQueriesTransmistted : R Integer
  
• NumberOfV1ReportsReceived: R, Integer
  
• NumberOfV2ReportsReceived: R, Integer
  
• NumberOfV3ReportsReceived: R, Integer
  
• NumberOfLeaveMessagesReceived: R Integer

WAN Links - Sharing Load and Failover - TR069 based configuration

CPE devices provide a feature called 'WAN link load balancing and fail over' facility where they have multiple WAN links to reach Internet. These WAN links are either taken from one ISP or from different ISPs by offices/homes. They provide redundancy capability and also can be configured to provide load sharing.

As far as I know, DSL forum did not define profile for this. I tried to give profile and required configuration parameters needed to configure this feature in CPE from ACS using TR-069 standard.

What constitutes this feature:

• Multiple WAN links in CPE.
  
• WAN links grouped together into multiple bundles.
• Each bundle having following properties
• Share the TCP/IP connections load across links in the bundle or just use only one link, user others when current link fails.
  
• In case of TCP/IP connections sharing, whether to bring up the all links always or bring up/down based on number of TCP/IP connections at that time - Define high threshold and low threshold in percentages. When number of TCP/IP connections reach high threshold, bring up new link in the bundle. When the connections reach low threshold, bring down one of the expensive links.
• Each link having following properties
• Method to use to check liveness : Ping, DNS resolution, None.
  
• Domain name, if method chosen is Ping or DNS.
• Liveness check interval in seconds: Liveness check is done using this interval period.
  
• Number of times the livness check should fail before marking the link 'Down'.
• Link status : UP, Down
• Cost of the link : 1 to 10 - 1 being highest cost and 10 being lowest cost.
  
• WAN interface: Interface identifier or logical identifier identifying the link.
  
• Protocol Bindings: Yet time, it is required that some protocol traffic go through some particular link in the bundle. For example, if email server is hosted by one ISP, all email connections should be sent via that link. ISP may not accept emails coming from the site via other ISP links.
• Each protocol exception record takes following parameters
• Destination IP address - Range of IP addresses.
• Protocol - UDP, TCP or UDP & TCP and any other protocol value.
• Port range: Valid in case of TCP and UDP.

Inner workings:

• Route dictates the outbound interface for packets.
• If outbound interface is WAN link, bundle is determined.
• If this is first packet of connection, then least loaded link is chosen in the bundle. If needed new link is brought up. IP address of chosen link is used for SNAT. Consider protocol bindings in choosing the link.
  
• When there are multiple WAN bundles, it is expected that ACS configures routes with source IP address - Source based routing. It enables usage of multiple WAN bundles. Note that there would be multiple default routes - one for each bundle. It is expected that ACS configures source based routes to make use of multiple bundles.

Profile as per above description:

• internetGatewayDevice.WanSharingBundle.{i}
• Enable
• Share or Failover only?: Indicates whether the load is expected to shared or use multiple links for failover only.
• HighThreshold
• LowThreshold
• NumberOfLinksInThisBundle
• internetGatewayDevice.WanSharingBundle.{i}.link.{i}
• Enable
• LinkStatus
  
• LivenessMethod : None, Ping, DNS
• DomainName : If ping or DNS is chosen as liveness method.
• Liveness interval
• FailureCount
• WanInterface : This is Full Qualified WANIPConnection or WANPPPConnection instance under WANDevice.
• CostOfLink
  
• NumberOfProtocolBindings
  
• internetGatewayDevice.WanSharingBundle.{i}.link.{i}.ProtocolBinding.{i}
• Enable
• MinSourceIPAddress
• MaxSourceIPAddress
  
• MinDestIPAddress
• MaxDestIPAddress
• Protocol
• MinPort
• MaxPort

These are my high level thoughts. More thinking should go in to make it more generic. I hope that it provided decent introduction for further work.