RCUs - Eliminate reference counts & increase maintainability

Reference counts in application specific blocks (records) are used for many purposes in Multicore environment. Main reason for reference count variable in records is to ensure that memory block is freed only after every module or core stop using that record. External modules normally increment reference count of the record before storing the pointer in their storage. Also cores using the record also increment the reference count, do processing and decrement once the pointer is no longer required in its processing. This will ensure that any core deleting the record will not free the record to heap prematurely and record gets freed only when its reference count becomes zero. Book keeping of record with reference count has many problems associated with it.

Let us take some dummy packet processing module 'dmod' running in a Multicore SMP operating system. It maintains its records in a hash list - drecList. Its record is called 'drec'. Let us also assume that there is one external module to this module called 'emod'. 'emod' has its own records of type 'erec' maintained in erecList. For discussion sake, let us assume that every time 'drec' is created, it calls a function in 'emod' which inturn stores 'drec' instance in its 'erec' instance. There are two paths for the packets. 'dmod' gets hold of packets from the link layer and also it gets packets from the 'emod'. 'emod' calls the 'dmod' packet processing function with the paacket and 'drec' pointer that was stored in its record.

Traditionally with spinlocks, it would be implemented as follows:

dmod:

'drec' structure contains two variable delete_flag, 'refCount' information. If both delete_flag is true and refCount is 0, then the record gets freed.

CreateDrecInstance()
{

• Allocate memory for drec.
• write_lock_bh(drecListLock);
• Add the drec to the drecList
• Set the RefCount to 1 and Delete_flag to 0;
  
• IncDrecInstance(drec) because it is going to be referred beyond write_lock_bh
  
• write_unlock_bh(drecListLock);
• CreateERecInstance( drec );
• return 'drec' instance to the caller.
  

}


DeleteDrecInstance( drec )
{

• write_lock_bh(drecListLock);
• Remove from linked list
• DecRefCount(drec) as it was set to 1 as part of adding to the linked list as part of 'CreateDrecInstance' function.
• write_unlock_bh(drecListLock);
• DeleteDrec(drec);
  

}


SearchDrec(parameters)
{

• read_lock_bh(drecListLock);
• Search with parameters for matching drec;
• if found, IncRefCountIfNotDeleted(drec) - since it is used by caller, reference count needs to be incremented here. If the record is already set for deletion, it is treated as record unmatched.
  
• read_unlock_bh(drecListLock);
• return matching 'drec' instance pointer.
  

}

IncRefCount(drec)
{

• spin_lock_bh(drecRefCntLock);
• increment drec->refCount
• spin_unlock_bh(drecRefCntLock);

}


IncRefCountIfNotDeleted(drec)
{

• spin_lock_bh(drecRefCntLock);
• if ( drec->delete_flag is true) drec = null;
• increment drec->refCount;
• spin_unlock_bh(drecRefCntLock);
• return drec;

}

DecRefCount(drec)
{

• freeflag = false;
• spin_lock_bh(drecRefCntLock);
• decrement drec->refCount;
• if ( drec->delete is true and drec->refCount is 0 ) freeflag = true;
• spin_unlock_bh(drecRefCntLock);
• if ( freeflag is true) Free drec memory to heap.

}

DeleteDrec(drec)
{

• freeflag = false;
• Call EmodDereference(drec); Call external module to remove reference to this. Note that this may not remove reference immediately in this function call. This function might generate an event and come out. That event eventually in some other thread context might get processed and reference to drec would be removed.
• spin_lock_bh(drecRefCntLock);
• set drec->delete to true;
• if (drec->delete is true and drec->refCount is 0 ) freeflag = true;
• spin_unlock_bh(drecRefCntLock);
• if ( freeflag is true) Free drec memory to heap;

}


DmodPacketProcess(pkt)
{

• Get search parameters from 'pkt'.
• SearchForDrec(parameters);
• if ( drec is null) CreateDrecInstance(parameters);
• DoPktProcessing(drec, pkt);
• DecRefCount(drec);

}

It is expected that 'emod' function while storing the 'drec' instance pointer does with its own spin_lock and read by incremeting 'drec' instance pointer.

There are two spin locks are taken on per packet basis - One list lock and another to increment reference count. As you might have known, locks are expensive even if there is no contention. Number of instructions that get executed as part of lock/unlock functions are significant.

RCUs alone will work fine and can be used to eliminate list locks and also to eliminate the need for reference counts as long as no external modules store the records by reference. In earlier example, there is 'emod' which stores the drec pointer. Since external module may not remove the reference to drec in the current processing cycle, reference counts are required. Using arrays with magic numbers for external module purposes will eliminate the need for reference counts.

Reference count mechanisms for one major reason of many bugs in Multicore environment. Elimination of need for having reference count would be welcome by many software developers. It is not a good practice to let external module store the reference (pointer) to the module records. Rest of this article describes one technique.

RCUs with Array level indirection would eliminate the reference counts in records. Let me take 'dmod' to explain this concept. In addition to maintaining the hash list it also needs to have one array of size 'number of records' that are possible in this module. Each array element has 'drec pointer', 'magic number'. External modules are not expected to store the pointer to drec at any time. They are expected to store index to the array and the magic number. External modules are expected to check the validity of array element by matching the magic number stored with the magic number in the array element. If both match, then it can assume that 'drec' pointer is valid in the array and can use the pointer for rest of processing.

struct RCU_indirection {
unsigned int magic; -- Value 0 means it is unused, any value indicates rec is valid.
void *rec;
} ;

dmod:

struct RCU_indirection drecArray[MAX_DRECS];

'drec' need not contain 'refCount' and 'delete' flag. But one variable 'index' is required.

Have global magic count : drecMagic = 1;

CreateDrecInstance()
{

• Allocate memory for drec.
• write_lock_bh(drecListLock);
• Add the drec to the drecList
• Add the drec and drecMagic in a free array element.
• Increment drecMagic
• Put the index into array element in 'drec'
  
• write_unlock_bh(drecListLock);
• CreateERecInstance( index, magic );
• return 'drec' instance to the caller.
  

}


DeleteDrecInstance( drec )
{

• write_lock_bh(drecListLock);
• Remove from linked list
• Set the magic to zero in the array element pointed by index in drec.
• Set the drec to NULL in the array element.
• wmb();
  
• write_unlock_bh(drecListLock)
• call_rcu ( ) : Callback function associated with it will free the drec.
  

}


SearchDrec(parameters)
{

• Search with parameters for matching drec
• return matching 'drec' instance pointer.
  

}

DmodPacketProcess(pkt)
{

• rcu_read_lock_bh();
  
• Get search parameters from 'pkt'.
• SearchForDrec(parameters);
• if ( drec is null) CreateDrecInstance(parameters);
• DoPktProcessing(drec, pkt);
• rcu_read_unlock_bh();
  

}

When 'emod' calls 'dmod' packet processing function, it should do following:

{

• rcu_read_lock_bh()
• if ( drecArray[erec->drecIndex].magic == erec->magic) drec=drecArrayp[erec->drecIndex].drec
  
• DoPktProcessing(drec, pkt);
  
• rcu_read_unlock_bh();
  

}

As you see it is simple and eliminate two locks on per packet basis. Even if there are more external modules, this scheme works. If record pointers are required to be stored in any other data structures within the module, then too there is no need for any reference counts as long as those data structures also manipulated within the same spinlock ( 'listlock' in example above).

Read-copy-update (RCU) - What are they and how they can be used

Very good explanation about RCUs at wikepidia with heading Read-Copy-Update.

RCU locks are replacement for Read-write spin locks. But they are very fast for read operations. It has similar overhead as write locks for add operations and little bit additional overhead in case of 'delete' operations. Networking applications typically maintain multiple context blocks (records) in data structure such as hash lists, linked lists, arrays etc.. . In many cases, these data structure are searched to find out the matching record on per packet basis. Records are created and removed infrequently. RCUs are best way to protect these kinds of data structures in Multicore SMP environments.

Read operation in RW locks in SMP Environments is heavy due to high number of instructions in these functions. Hence these functions take more CPU cycles even when there is no lock contention. That is why, RCUs are preferred.

There are mainly five API functions in RCU:

1. rcu_read_lock
2. rcu_read_unlock
3. rcu_assign_ptr
4. rcu_dereference
5. call_rcu/synchronize_rcu

rcu_read_lock and rcu_read_unlock are used while searching for a record in any data structure (linked list, arrays etc..). While searching for a record, yet time you may need to go through the list via next/prev pointers in linked list. These pointers must be dereferenced using rcu_dereference. Typically the pointers which are modified with rcu_assign_ptr in add and delete operations need to be dereferenced while traversing the list. These two functions are mainly to take care of weak order processors which execute instructions out-of-order and also write into the memory locations out-of-order. If you really look at body of these functions, they execute memory barrier instructions. Many RCU implementation also don't have many instructions in rcu_read_lock and rcu_read_unlock. In Linux, RCU read lock mostly disables preemption (If preemption is enabled). Writer task is expected to use normal spin locks to protect the data structure from other writers. Write task can involve either addition or deletion of record in and out of data structures. If readers can be in softirq, it is necessary that writes use 'bh' version of spin locks.

RCUs read operation depends on the fact that any word assignment to a memory location is atomic which is true in all processors. Reason for having memory barrier operation in rcu_assign_ptr is to ensure that value assigned appears on the cache/memory before any dereferencing happens by other cores.

Whenever writer removes the records from application data structure, it needs to call synchronize_rcu or call_rcu. call_rcu is asynchronous mechanism and is used by application when it needs to get control after all cores complete their current scheduling cycle, which means that the record is not being accessed by any core. call_rcu takes a function pointer and a callback argument. This callback function is called after all cores complete its current scheduling cycle. Application callback can take further action such as freeing the record. If the record is being referred by pointers by some other application records, then this callback function can't free the memory until all references are removed. In those cases, reference count need to be maintained by records in addition to RCUs.

Application Delivery Controllers versus Server Load balancing devices

Server Load Balancing (SLB):

Let us first revisit the "Server Load balancing" concept and its features. Server Load balancing was invented during initial stages of web in late 1990 and early 2000s. At that time the pages served were mostly static. So simple Layer4 load balancing was good enough to do balancing of TCP connections across multiple servers in the server farm. Load balancing device, typically, is assigned with Public IP address. When client connection lands on this public IP address, this device figures out the best server that can serve the connection and hands over the connection to the server by doing "Destination NAT", that is, destination IP address of remote client generated packets gets translated with IP address of server selected. To ensure that packets going to client from server are accepted by the client, response packets from the server would undergo source IP address translation. By this clients only see the public IP address in response packets. Now, let us discuss some SLB features.

• MAC Level redirection : When servers in the server farm have public IP address, this option can be used instead of doing "Destination NAT". Server Load balancing (SLB) device should be placed in the traffic path to ensure this mode works. In this mode, SLB device selects the server in the farm and modifies the destination MAC address with the server MAC address. Optionally it can also change the Source MAC with the SLB device MAC.
  
• Additional Source NAT : This mode is useful in assymetric routing topologies. When destination NAT is done on the client packets, it is required to do source IP translation on server to client traffic. For this to happen, server to client traffic must pass through the same SLB device that did the DNAT on the client traffic. Source NAT is done on the client traffic with the SLB device IP address to ensure that the traffic come to the SLB device even if the servers' routing tables are configured with some other default gateway.
  
• Multiple different algorithms to select the server: Based on SLB deployment and need, several different ways of scheduling connections to server can be chosen. Following are some of the methods.
  

• Round Robin : Each server is treated same from capability perspective and servers are chosen on round robin fashion. If there are 2 servers in the server farm, then alternate connections are given to each server.
  
• Weighted Round Robin : Typically each server is given a weight 1 to 10, 10 is assigned to server that can serve 10 connections for every 1 connection served by server that has weight 1. Until weight number of consecutive connections are assigned to a server, next server is not chosen.
  
• Least Connections : In both Round robin and Weigthted round robin methods, it is assumed that the all connections are almost equal with respect to duration of connection, amount of traffic that is sent on each connection and processing power used by each connection on the server. But in reality it is not true. In 'Least Connection' method new connections is given to server which is serving least number of connections at that time. SLB device maintains the count of active connections with each server to facilitate this scheduling.
  
• Weighted Least Connections : All servers can't be assumed to be equally powerful and this feature allows deploying servers of different processing power capabilities. Each server can be given a weight. If 2 and 5 weights are given to server 1 and server2, it means that server2 is powerful enough to serve 5/2 times the active connections simultaneously than server1. SLB device keeps this weight in mind in deciding the best server for new connections.
  
• Fastest Response time : Though 'Least connections' and 'Weighted Least connections' do their best to distribute the connections, there can be situations where many high processing connections are given to one server. Any new connections going that server can overwhelm that server and due to that connections may not be served well. Any connection that does deep data mining function could overwhelm the server. To take care of these situations, it is better to monitor the processing power left in each server for new connections. What is the better way than monitoring the response time of servers. Though it is not 100% way of finding the least utilized server with response time, it is close as long as monitoring for response time is happening continuously. Some SLB devices monitor the response time as part of health checking. That is not good enough as health checks happen once in a while. Response time measurements must be done on response to real client traffic.
  
• Fill-and-go-next : Green is the buzz word now - Save power and save on cooling costs. Round Robin and other methods above distribute the traffic to all servers. Even when there is less load across servers, they are all used and thereby not giving chance to any server to go to any power saving modes. When this mode is chosen, SLB device uses one server to full utilization before selecting next server. Typical configuration include the number of connections a server can handle and when to start warming the next server. For example if 10000 and 8000 numbers are given, after 8000 active connections on existing server, next connection is given to next server to bring that server up to the full power and both servers are used until 18000 active connections and then another server is warmed up and so on. This mode can be combined with Round Robin method where this is used to balance the traffic across all warmed up servers.
  

• Syn Flood Detection and Prevention (DDoS Prevention): This feature is common in servers from last few years. But detection and prevention at central place in SLB allows administrators not worry about feature availability in all servers or enabling this feature in servers. This feature prevents attackers from sending large number of SYN packets without completing the TCP connection establishment phase. SLB devices using this feature detect the SYN flood attack without consuming its resources using SYN-Cookie mechanism. Only when three way handshake is complete, connection is awarded to server in the server farm.
• Traffic Anomaly detection and Traffic Policing: DDoS attacks such as Syn flood detection is really thing of a past. Newer kind of attacks complete TCP connections and issue requests that consume CPU power there by creating Denial of Service for genuine users. Since these attacks complete the connection, it is possible to track the users by their IP addresses. Traffic anomaly detection feature allows administrators to configure the baseline traffic characteristics based on normal day activity. SLB devices based on this configuration detect any anomaly and can take corrective actions such as:

Throttling the traffic coming from identified IP addresses : Throttling the traffic can be on "Simultaneous active connections", "Connection Rate", "Packet/Byte Rate".


Blocking the traffic from identified IP addresses for certain time.


• Health Checking: SLBs are expected to use servers that are online. To ensure that, SLBs typically do health checks on servers to ensure that they are alive. Some types of the health checks that can be configured are:

ICMP Ping : Using this check SLB knows that the server machine is active.


TCP Connection Checks : These checks allow SLBs know that a given TCP Service is running. Note that it is possible that Machine is alive, but the user process listing for TCP connections is dead. This check allows SLBs to check the TCP Server liveness.


Application Health Checks: These checks allow SLB to know whether the application is running well on the TCP Service. Note that TCP Server may be running in the server, but application that acts on the application data could be different user process than the process that is listening for TCP connections. Hence it is important to have these health checks enabled in the SLB devices.


• Persistence: For some end applications, it is necessary that all connections coming from one client go to the same server. This is necessary if the application maintains some state across connections coming from the client. It necessitates temporary and dynamic binding state in SLB devices. When there is no matching bind state, one server is chosen for the connection and immediately binds the client IP address with the chosen server IP address. Any new connection from this client is given to same server without server selection process. To ensure that these bindings don't consume large amount of memory on SLB devices, these bindings are removed after certain amount of inactivity. As you can see later, binding are implemented using cookies in case of HTTP connections without having need to store binding state in the SLB. Cookie received in each request would indicate the bound server.

Application Delivery Controllers (ADC):

Server Load balancing based on L3-L4 ( Source IP, Destination IP and Ports) does not solve all issues with different kinds of applications running on the servers. It requires L7 intelligence in the SLB devices to meet the load balancing requirements of newer kind of applications. Since they have L7 intelligence in making load balancing decision, I guess these devices are being called 'Application Delivery Controllers". In short they are called ADCs.

First generation of ADCs were trying to solve challenges associated with E-commerce and Session-persisting HTTP based applications. One challenge is to ensure all connections go the server which was chosen for the first connection from the client. This is necessary due to the fact the applications on the server maintain user state in the memory and this state is updated with the user selections until user logs out. This challenge has another twist where these connections are SSL encrypted.

First generation ADCs solve these challenges using HTTP protocol intelligence in ADC devices.

• Persistence using HTTP Cookies : Though IP based persistence works decently, it fails when a given user IP address changes across connections. This can happen if the user router gets new IP address in the life of user session. I believe another possibility is due to different NAT addresses usage by some ISPs. That is, ISP might use one NAT IP address for one connections and another NAT IP address for other connections of same user. Due to these issues IP address persistence does not work in all cases . In addition, IP address persistence occupies some memory in SLB devices. Alternative for HTTP based applications is to use HTTP Cookies. HTTP Cookies allow servers to relate the connections belonging to the same session. Same mechanism is used by ADCs to relate user connections. ADCs define their own cookie for storing the selected server in encrypted/encoded cookie data. When new HTTP connection is received, it checks whether it has its cookie and value. If not, it assumes that this is first connection of session and adds a Cookie in the response going to client with encoded/encrypted selected server information. Any further connections coming from the same user to this site would have this new cookie and this cookie value is used by SLB to map the connection to the server. Since this cookie belongs to the SLB, SLB devices remove the cookie from the request before sending it to the server.
• SSL Termination : E-Commerce web sites require SSL/TLS Security and also session persistence. To facilitate this, SLBs provide SSL termination capability. This capability in addition also offloads CPU intensive SSL processing to ADCs and thereby saving CPU cycles on server for application processing.
  

Second generation ADCs extend above functionality with HTTP Optimization, L7 protocol data based server selection, threat security features and multiple virtual instances.

HTTP Optimization features:

• HTTP Connection pooling : HTTP protocol allows multiple transactions in one TCP connection. This feature in ADCs multiplex multiple client connections into few connections to the server. Without this feature, there are as many TCP connections to the Server from SLB as number of TCP connections it is terminating. With this feature, there are only few TCP connections towards server, there by saving some CPU cycles and memory on the servers. Though the amount of saving one can have with this feature is debatable, this is one feature many ADCs support.
  
• HTTP Compression: HTTP compression using gzip and other algorithms can save bandwidth by compressing the response data. ADC devices offload this capability from the Servers there by making server CPU cycles available for application processing.
  
• HTTP Caching & Delta Encoding: HTTP Caching feature between clients and servers avoids duplicate file download if content was not changed on the server. But there are many instances where file/data content change but not significantly. This feature allows only difference (delta) being sent to the clients rather than complete content there by saving bandwidth usage. Since delta encoding requires significant CPU cycles, ADCs offload this functionality from HTTP Servers.
  

Programmatic L7 Protocol content based Server Selection:

SLBs traditionally don't look at the application protocol data to make server selection. ADCs augment the criteria for selecting the server with protocol data such as HTTP data. Some examples where protocol intelligence is required:

• Multiple server farms with each each farm having different content.
  
• Different server farms for Mobile users and Normal users.
• Different response on errors based on browser type.
• Prioritization/throttling/shaping of requests to specific URL resources upon congestion.
• Addition/Modification of request/responses based on the type of applications running on the server farm.
• Combination of above.

Since each deployment has its own requirements, one configuration solution for all is not sufficient. Hence ADCs provide flexible mechanism using 'rules' similar to IPS devices which provide rules for detecting vulnerabilities. Rules are created by administrators using the application protocol (eg. HTTP) keywords exposed by ADCs. Typically each protocol element is exposed as keyword. Multiple rules can be created with each rule having keywords and associated values. ADCs internally match the connection data with the rules. First match of rule with respect to values of keywords is used to take actions as per the rule. Actions can be 'Selection of Server farm', traffic policing etc..

Threat Security:

Since ADCs are becoming a central location for all requests going to the data center servers, this is place to enforce threat security and thereby offloading threat security functionality from the servers. Some of the threat security functions ADCs support are:

• Traffic Policing based on protocol intelligence: ADCs with protocol intelligence can do better job of DDOS prevention than SLBs. Baseline traffic can be much beyond typical 5 tuples. It can include URL pattern, User-Agent, Cookie values and beyond.
  
• Web Application Firewall: There are many attacks that are being exploited by attackers on web server applications including SQL injections, XSS (Cross Site Scripting), LFI (Local File include) and RFI (Remote File Include). ADCs are increasingly offloading this functionality from the servers. It not only improves server performance, but also administrators can configure signatures at one place i.e ADC.

Virtual Instances:

This feature is not common in ADCs today. But I believe this is going to become critical feature moving forward to ensure that multiple server farms belonging to multiple domains corresponding to different customers are supported by one or few ADCs. One ADC device should be able to support multiple customer server farms in public data centers. There are two types of virtual instances possible - In first type. One executable image supports multiple instances. Here configuration data and run time data are stored separatetly and even provide UI/CLI with role based access to configuration belonging to virtual instances. Second type of Virtual instance is to have multiple images with each image for one virtual instance. In this case, even if there is any issue with one virtual instance, other instances are not affected, there by providing good isolation. First type can provide large number of instances where second type is limited to few tens. I personally think that Linux container approach is better for second type as this is lean and uses common Operating system and TCP/IP stack image. Only user processes are instantiated multiple times as number of containers.

Third generation ADCs:

What are the features one would expect from third generation ADCs. I can think of few based on where data center market is heading.

• Cloud Computing : Traditionally ADCs are configured with all servers in a given server farm. It works well when manual intervention is required to setup/take-off servers from server farm. Administrators are used to configure the ADCs accordingly. In Cloud computing, servers are no longer physical. They are virtual. They can be added and removed dynamically and programmatically. They also can be disabled and enabled dynamically without manual intervention. Some use cases - virtual instances are brought up and down based on the load on the servers or based on time of day, holidays etc.. Since they are coming up and going down dynamically, it is expected that ADCs also would get this information dynamically.
• Beyond HTTP Servers (SIP, CIFS, SMTP, IMAP etc..): HTTP had been and is dominant in data centers today. It is changing with cloud computing. Many organizations are expected to host their internal servers too in the cloud. They include mail servers, File Servers etc.. Third generation ADCs are expected to balance the traffic across Email Servers, File Servers etc.. In addition it is expected to have optimization features to save network bandwidth. File Servers today dominantly support CIFS. ADCs are expected to have CIFS proxy that can save bandwidth by doing delta encoding, caching and de-duplication etc..

Advice to Network Device testers - Simulate Capacity/Stress related faults

Capacity in network devices such as UTM is specified with respect to simultaneous connections in case of firewall, ALGs, intrusion Prevention functionality, tunnels in case of IPsec VPN, number of sessions in case of Anti Virus and Spam functions and many more related to different smaller functions. All the functions are not normally used at the same time. Even if all functions are used at the same time, all sessions may not be going through all functions. Due to this, network device vendors typically oversubscribe memory. That is, the memory needed for all functions for the specif ed capacity would be lot more than the memory available in the devices.

This could pose interesting problem in the field. If there is a deployment where multiple functions are used by large number of connections, there could be memory shortage and other resources shortage. This leads to error being returned when the resource is being allocated. If error detection, propagation and recovery is not taken care well by the software, this could lead to instability, leaks, crashes and lockups. It is tester job to ensure that these kinds of problems do not happen in the field. Typically testers simulate different conditions and ensure that system is stable. Yet times, it is not possible for testers to test all different combinations or simulate different conditions.

I believe testers should be able to simulate all possible combinations by simulating all kinds of error conditions. As part of it, testers should ask development team to provide facilities to inject the faults. In particular, testers should ask for facilities to inject faults for following.

• Memory allocation failures: Almost all functions in software would allocate memory either at the time of connection establishment or on packet basis or to queue the packets and control data etc... Testers should have ammunition to inject the memory fault for specific functions.
  
• Socket/File open failures
• Semaphore creation failures
• Thread/Tasklet creation failures
• Fault simulation of any other OS resource that gets allocated after software is completely initialized.

Testers should go at testing in methodical way:

• Keep list of all functions and OS resource allocations they do.
• For each one of them, create a test case.
• Before running the test case, configure to inject fault.
• Run the test and ensure that system works as expected.
• Run the test without fault and ensure that system is stable.
  

I believe that this kind of testing should happen for every release - feature or maintenance releases. If these tests are done manually, it takes very long time. My suggestion is to automate them.

Firewall and NAT ALG Testing Recommendations

Overview:

Stateful inspection firewalls open temporary holes to allow data connections based on information it reads from the control connection. Some protocols such as FTP, SIP, RSTP, H.323, MGCP open a connection and exchange IP address, port information to peer end point for data transfer. Ports that are exchanged in control connections are not well known ports and they are ephemeral. Due to this, administrators can't configure firewall rules to allow these connections without allowing everything. Application Level Gateways (ALGs) are software modules within firewall interpret the protocol packets by extracting the ephemeral port information and open temporary holes to allow data connections to pass through the firewall between protocol end points. Since each protocol is different, multiple ALG modules are required - one for each protocol.

ALGs also do the address and port translation in the protocol data if firewall supports NAT functionality. If the IP address or ports are specified in ASCII form, there is a big possibility where the data length of the packet changes after translation. In case of TCP based ALG, this results into sequence number modifications in TCP header. Firewalls typically take care of maintaining the delta sequence numbers and modify further packets with this delta in both "Sequence number" and "Ack number" fields to ensure the integrity is maintained with client and sever end points of the connection. It is also important to note that ALGs modify different packets during the life of session and firewall software is expected to keep updating the delta sequence numbers appropriately. It is also to be noted that firewalls need to keep the history of delta numbers with respect to original sequence numbers to apply appropriate delta in case of retransmitted packets which are older.

To apply translation on the data, it is required that the ALG has complete PDU. In some protocols such as H.323 and SIP, this can be large. If there is congestion in the network, the end point does not send the PDU in one TCP packet and requires acknowledgment to send rest of PDU. Due to this, newer generation of firewalls send the acknowledgment to make the end point send rest of protocol data.

Many routers change the TCP MSS value of SYN and SYN+ACK packets of transit traffic to lower value whenever there is multimedia traffic to ensure that VOIP packets do not get stuck. As we all know, Voice traffic is delay sensitive and it should be transmitted as soon as possible. If routers has slow link then it takes significant time to transmit 1500 byte packets. If link bandwidth is 256kbps, it takes around 45msec to transmit 1500 byte packet. If WAN controller of the router chooses 1500 byte packet and if VOIP packet comes right after that, then VOIP packet may need to wait upto 45msec there by increasing the delay of real time traffic. By lowering the TCP MSS value, the size of TCP packets generated by end points can be controlled. Broadband routers setting the value of MSS value of transit TCP packets to 256 bytes are quite common. In these cases, the protocol data of complex protocols requiring ALG comes in many packets. Firewall and ALGs ensure to extract the relevant data for opening holes and translation even protocol data is coming in multiple TCP packets.

As discussed before, ALGs open temporary holes - pin holes. If ALGs are not implemented well, attackers can make control connection and send PDUs with data such a way that pin holes are created to access internal critical services. Also attacker can DoS the firewall by sending large number of PDUs which creates large number of pin holes there by causing service disruption to genuine users/connections.

ALG implementation can become very complex. Vulnerabilities increase with complexity. Buffer overflows, boundary conditions are typical problems associated with complex protocol implementation. That is one place validation should concentrate on.

Many protocols specifications (standards) don't specify maximum length of protocol messages - especially text based protocols such as SIP, HTTP etc.. Protocol implementations (end points and ALGs) typically assume the typical sizes while allocating buffers to buffer the data and don't allow the traffic if it exceeds this limit. Since these sizes are not universally adopted by different implementations, this could pose interoperability problems if ALG implementation assumption of size is different from end point implementations. This is one area validation should concentrate on. In my view ALG implementations should not assume any size restrictions for the PDUs which are not interpreted for its operation. For PDUs that are needed to be buffered, this size restriction should be as maximum as it can be. Some times the protocol messages is prepended with size information. In those cases, ALG implementations must allocate the buffer based on this size information. Validation should concentrate to ensure that ALG does not impose any problems in functionality.

As said before, ALGs also do the translations in the protocol data. ALG implementation should ensure that the translations happen for all three cases - Source NAT, Destination NAT and Source & Destination NAT. Many times validation Engineers concentrate on testing using one session. Many times problems related to NAT can't be found if only one session is used. Validation Engineers should test the ALG based firewall implementations with multiple sessions.


Recommendations:

As you can see, validation testing of ALGs is not as simple as running standard applications on both ends of firewall device. For example, running standrad FTP client and Server applications on two sides of firewall device and ensuring the file transfer succeeds is necessary, but not enough validation of FTP ALG. I recommend validation engineers to consider following for each ALG before certifying.

Functional testing:

• Test with standard applications: Make a list of popular applications. Ensure that different combination of applications as client and server succeed in following cases. Configure firewall to allow control connections (initial connections) only.
  
• Without NAT
  
• Source NAT : With clients behind internal network and servers in external network.
  
• with NAT IP address whose length in dotted decimal form is more than that of source IP address.
• with NAT IP address whose length in dotted decimal form is equal to length of source IP address.
• With NAT IP address whose length in dotted decimal form is less than that of source IP address.
• Destination NAT: With servers in internal network and clients in external network.
• DNAT IP address in dotted decimal is equal to length of Destination IP address in dotted decimal form.
• DNAT IP address is more in length than that of destination IP address in dotted decimal form.
• DNAT IP address is less in length than that of destination IP adderss in dotted decimal form.
• Source NAT and destination NAT together: With servers in internal network and clients in external network.
• Explore all options of applications and ensure that all options work with above NAT combination.
• Ensure that Private IP address (client IP address in case of SNAT, Destination IP address in case of DNAT) does not appear on the packets after translation. This requires capturing the packets and searching for IP address in both binary and dotted decimal form.
  
• Understand the protocol and get familiar with messages and fields and their lengths. If size is not mentioned in the protocol specification, get familiar with realistic maximum length of messages and fields. Test to ensure that ALG does not drop messages when messages with maximum sized fields are sent.
  
• Ensure that ALGs perform well even when there is temporary packet loss. FragRoute tool can be used to drop the packets. Ensure to test with all NAT combinations.
• Ensure that ALGs perform well when the TCP packet sizes are smaller than protocol messages. FragRoute tool can be used to change the TCP packet sizes. Ensure to test with all NAT combinations.
• Ensure that ALGs perform well when TCP packets with smaller size and reordered. FragRoute tool can be used to reorder the TCP segments. Ensure to test with all NAT combinations.

Negative Testing: This testing is required to ensure that systems don't crash when invalid packets are sent.

• Ensure that ALGs don't misbehave (crash or lockup) when protocol messages and fields of different lengths and values are sent. Make a note of all messages in the protocol specifications. Ensure to send messages of different length by writing your own client and server protocol or instrument the existing client and server implementation. I prefer later to reduce the effort required to do this kind of testing. It is relatively simple to generate messages and fields with different lengths and values for the first message of protocol. But for messages that are deep down the protocol require successful initial protocol phase. Hence I prefer going with instrumenting the existing open source client/server protocol implementations.
• Go through some of the common vulnerabilities found in client and server implementations by searching through the CERT repository. Since ALG is also interpreting the protocol messages, ensure that these kinds of vulnerabilities are not present in the ALG.
  

Stress testing: This is one important step to ensure that system under test can cope up with capacity specified. Also, it is important to ensure that system performs as per performance requirements.

• Use IXIA/SmartBits kind of tools to simulate large number of client and servers to ensure that system works as specified with respect to capacity.
• Use IXIA/SmartBits to test the connection rate and ensure it satisfies the specifications of the box.
• Use IXIA/SmartBits to test the throughput requirements.
• Use IXIA/SmartBits to test throughput and connection rate combination requirements.
• Repeat above test cases for 12 hours to ensure that the system is stable.

Avoiding Network pipeline programming problems

Pipeline programming divides the task into multiple stages and links these stages. Each stage is typically implemented in different execution context such as tasklets, threads or even processors. In network pipe lining, packets go through multiple stages with each stage doing some application specific task before packet goes out or consumed. When a particular stage is completed, packet gets enqueued to the next stage. Next stage picks up the packet from its input queue, processes and enqueues to the next stage and goes on until all stages are completed.

If care is not taken, pipeline programming can introduce major bottlenecks. Inter stage queues can become longer when there is asymmetric processing with respect to CPU cycles among processing stages. This gives rise to memory depletion and dropping packets. Jitter of packets can increase if further stages don't pickup the packet soon enough. One would see major problems if control information is also being passed via queues to the next stage. Unlike packets, if control events are dropped, either due to memory failures or due to queue size constraints, it results to functional issues.

Yet times, many software applications require additional information to be sent to the next stage along with the packet. Some implementation allocate new memory blocks to hold meta information along with packet buffer. If control data and packets allocate from the same memory pool, there is a chance of memory failures for control data when large number of packets are received.

My recommendation is to avoid pipeline programming as much as possible and go for run-to-completion model. Yet times pipeline programming is not avoidable and one instance of this programming is required due to stack size restrictions in Linux kernel. In may Linux distributions, stack size is limited to 4K. Some complex applications having multiple logical modules require more than 4K stack if run-to-completion model is adopted. Pipeline programming with multiple logical stages is one solution to avoid stack size problems. Another instances is where asynchronous based hardware accelerators are used. To take full advantage of hardware accelerators such as cryptography and pattern matching accelerators, applications running in cores handover the job at right time to accelerator hardware and work on some thing else such as processing next packet. When hardware accelerator completes its job, it indicates the result asynchronously (mostly through interrupts). Application software picks up the result and does rest of the required processing on the packets. Basically, application processing is divided into pre-acceleration stage, acceleration stage (which is done in some hardware) and post-acceleration stage, thus pipeline model. As in pipeline model, there will be input queue to the hardware engine and output queue to the processors.

Recommendations

One of the main issues that come due to pipeline model is bloating of inter-stage queues. It eats up memory to hold the nodes in the queues, drops partially processed packets thus wasting the CPU cycles if the next stage tasklet does not get chance to run for a long time if incoming packets are coming in very high speed. As in run-to-completion model, better way to avoid is to stop reading new packets until read packets are processed and sent. As much as possible, simulate run-to-completion model even in pipeline programming. This can be achieved by making cores to process the queues before new packet is read. Based on type of system architecture chosen, either packets are read from the hardware in poll mode or in interrupt mode. Many multicore processors in the market today provide poll model. Until packet is read, packets are kept in the hardware. In interrupt mode, packets are read without application knowledge by the operating system and packets are handed over to the applications.

In poll mode case, my suggestion is to force processing of queues of all pipeline stages before dequeuing the next packet. In case of interrupt mode, my suggestion is to process the pipeline stage queues before processing the new packet. To make this happen, pipelining should not involve dedicating the cores to the specific stages. The core receiving the packets should be able to execute any stage of application and any core should be able to receive the new packet. In essence, go for SMP model in case of multi core processors. In case of multi core processors, my suggestion is to divide the sessions across multiple cores to avoid too many contentions, but ensure to run all application code in all cores.

To avoid contentions, it is advisable to use as many queues as number of cores for any given stage. Stage that is outputting the packet to next stage enqueues to the current core specific queue. As many tasklets as number of cores are required to process the queues. Each tasklet takes the queue node from its queue and does the processing. Tasklets are scheduled by the operating system normally. In addition to processes queues via these tasklets, my earlier recommendation is to force the queue processing before new packet is processed. Since in SMP model, there is a specific need for processing certain packets in specific cores, forced processing should process only current core specific queues of pipeline stages. One might think that forced processing is good enough and there is no need for tasklets. Remember that packets need to be processed by all stages even when there are no new packets. Since forced processing function is called for every new packet, it is necessary that it takes very few cycles especially when there are no nodes to process.

Another recommendation I have is to avoid memory allocations to hold meta information along with the packet while queuing the information to the next stage processing. Typically, each packet buffer has its own meta information in its buffer header. In case of linux, skbuff holds buffer specific information. My advice is to provide additional space in buffer header to hold queue node specific meta information. By avoiding memory allocations, performance goes up, memory fragmentation is reduced and overall system performance improves.

Techniques to defend aganist DNS Cache Poisoning attacks

This subject is covered very well in many forums, IETF drafts and RFCs. My purpose of this article is to provide some idea on how network security devices can play a role in defending the DNS cache poisoning attacks. Before that, let me give some background.

Many DNS attacks, discussed recently are on DNS Caching Servers, DNS resolvers and there are less number of attacks on Zone Authority DNS Servers. Attacker sends DNS response packets to DNS Caching Servers before authoritative DNS Servers respond to queries sent by DNS Caching Servers, there by poisoning the cache with IP addresses for domain names of their choosing. Typically DNS Caching servers are located in company premesis and ISPs. These DNS Servers cache the responses until TTL expires. When query is raised to these caching serves, they reply immediately with response if it corresponding entry is present in its cache. If not, it sends the query to some other pre-defined uplink DNS Servers or to the authorative DNS Servers corresponding to the domain in question.

Attackers send the DNS response as if it was from uplink DNS Server or authoratative DNS Servers. Since DNS works on UDP, it is quite easy for attacker to spoof the DNS response. The difficulty in making the attack successful by attacker lies in making the response acceptable by the Caching Server. DNS Caching Servers, typically accept the response only:

• they have sent the query.
• response contains the same transaction ID as the query it had sent.
• response contains the destination port same as the source port it used to send the query.
• response contains the source port same as the destination port it used to send the query.
• response contains the source IP same as the destination IP it used to send the query.

But, the attackers seem to be able to penetrate above defenses. According to recent vulnerabiltiy reports, many DNS Caching servers randomize the transaction ID already, but it seems that it is not good enough. One of the syggestions to defend against this attack is to make source port of the query also random. It helps in making attacker life difficult as it requires more responses and hence time. It appears that many DNS Caching servers and DNS resolvers patched their software and randomizing the source port. That is good news. But, it appears that even this defense is broken. See this article here: http://tservice.net.ru/~s0mbre/blog/2008/08/08. As seen in the blog, it takes more time, but attacker can make the exploit successful.

Many DNS Caching servers implemented some additional defeneses such as:

• Additional corelation checks between Query and responses, such as checking for "Domain name" in Query section matches with Query and Answer section in the response : Many attackers are getting around this defense by sending their own queries with arbitrary domain names as part of queries and sending same in the DNS responses . By sending queries themseleves gives them even more control. That is, they need not wait for subscribers of DNS Caching Server to intiate the query, thus greatly increasing the chance of cache poison. Attackers themselves are sending the query with unknown subdomains which certainly triggers the query from the Caching server. Attacker can time their responses right after sending the query with un-resolvable and randomly created domain names. One might ask, how is Cache is getting poisoned if attackers are sending random subdomain name. At best it is kind of DoS attack. From the recent attack exploit scripts (script1 and script2), one can understand that cache poisoning is not happening with the information present in the answer section of DNS response, but using NS Resource Records (Athority section) and Additional Resource records of DNS response. Attackers are ensuring that the Domain name in Question section in both query and response is same. To understand more about this kind of attack, read 2.3 section (Name Chaining) of RFC3833.txt
  
• Preventing from Birthday attacks - Again this defense is being overcome by sending random domain names in each query by attackers.
• Ensuring that the NS record entry has some portion of the domain name in the Question section to make sure that arbitrary NS record is not honored: This kind of defense also is being broken by attackers. Attacker are actually creating full domain name in Question section with random string followed by the victim domain name and creating response with victim domain name in Authority section. Thereby, the defense is being bypassed by attacker. For example, if attacker wanted to serve his/her own IP addresses for www.veryimportantsite.com, then the type of queries and responses he/she sends to victim caching server look like this:

-----------------------------------------------------------------------
Query:
QUESTION SECTION:
.www.veryimportantsite.com. IN A

Response:

Question section is same as Query.

AUTHORITY SECTION:
www.veryimportantsite.com. 6000 IN NS attacker.veryimportantsite.com.

;; ADDITIONAL SECTION:
attacker.veryimportantsite.com. 6000 IN A 2.3.4.5

----------------------------------------------------------------------------

As you see above, defenses used by DNS Caching servers are not going to work. In above case, DNS Caching Server is going to use 2.3.4.5 as the Authoritative DNS Server for www.veryimporantsite.com domain name. Any queries to this caching server for domain names www.veryimportant.com and any other subdomains within it, 2.3.4.5 server is contacted. Since this server is hosted by attacker, he/she can choose to provide IP addresses for victim domain names. Basically, in this case, NS List is corrupted with some poisoned entry. Replace veryimportantsite.com with google.com, then it will have devastating effect if the DNS Caching server is one of popular ISPs' servers. Many users behind this ISP will be directed to site of attackers' choosing when they visit www.google.com or any subdomain under www.google.com.

Many people agree that the right solution is getting rid of UDP and use DNS over TCP or use DNSSEC. It is going to take time for everybody to adopt this. Until then DNS Caching Servers need to improve their randomization and security administrators install security devices to prevent the attack or get early signs of attempts.

How can security devices help?

Due to the stateful nature of network security devices, DNS response packet is not accepted by the firewall functionality of devices if 5 tuples of response don't match up with 5 tuples of DNS query. That is, DNS response packet must have its SIP equal to DIP of the query, DIP equal to SIP, SP equal to DP and DP euqal to SP. In addition, many firewall devices also check for matching transaction ID. One might think that Firewall is not adding value here since many DNS Caching servers are already doing these checks. But firewall goes one step further. It only allows DNS response from secuirty zone to which DNS query is sent. Also, it is good practice to set up the rules to allow DNS traffic as needed. Typically in Enterprises, only outbound DNS is allowed. If there are Authoritative DNS Servers in side, DNS requests are allowed only to these particular servers. This will reduce the probability of successful poisoning attacks. Let us analyze.

Assume that, a company has DNS Caching Server in its "Intranet-DMZ" zone and ISP DNS Server, ofcourse, is in untrusted (External) zone. Enterprise administartor creates a rule 'Intranet DMZ Zone' to 'External Zone' on Destination Port 53 Destination IP as ISP DNS Server with "Allow" action. Due to this rule, any DNS queries from the attacker will be dropped. So, the attacker only needs to depend on queries generated by DNS Caching Server due to genuine queries from its local users. That dramatically reduce the success of the attack. If there are internal attackers, they need to send both queries and responses.

Some firewall devices come with local DNS Caching Server (DNS Resolvers) functionality. In these cases, it should take same precautions such as randomizing the source port and transaction ID while sending DNS query to uplink DNS Servers.

I feel that security devices implementing firewall and IPS can add more defensive measures such as:

1. Many DNS resolves and Caching servers are already enhanced and donot send DNS queries on persistent sessions. That is, Each DNS query is sent with different randomized source port. Security devices can remove the session once the corresponding DNS response is received. Then count the number of DNS responses received for which there is no session entry.

As you observe from the attack scripts, for every query, large number of responses are sent by the attacker. By having some logic on counting the number of orphan responses in a particular quantum of time indicates that some attacker is trying to exploit DNS Caching Servers. Security devices can warn the administartors for further analysis.

2. Once the attack attempt is detected, security device can send one more query with the same information as in the original query and match the responses. If the content looks similar, then it can be cached and send response to the original querier. Ofcoruse, this requires some DNS resolution functionlity in the security device. Note that this functionality can also be implemented within the DNS Caching Servers too.

3. Security devices implementing NAT should not make the guessing of ID and source port easier. When NAT port is selected, it must ensure that it is random port.

Data Center Firewall features

What makes firewall a good data center firewall?

Before going further on what features are expected by data center IT/Security professionals, it is good to revisit the data centers. Data center providers are mainly hosting providers. They host their customer applications and machines. Some customers of data centers share a machine resource, some like to host their application in a virtual system and some like to host their applications in a dedicated machine(s)/blades. To provide availability and share the load, application servers are installed in multiple machines with "load balancers" distributing the load across the server farm. As we all know, HTTP/HTTPS servers by far the single most server application in data centers. Most of the times, services provided by hosted servers are meant for general public.

Increasingly, there is a trend by Enterprises offloading hosting of Intranet servers to external data center providers. Intranet servers are typically provide access to Employees and limited access to their partners. For example, many email services, sharepoint and wikis are being offloaded to data center providers by many small and medium Enterprises. Many of these services require user authentication. Enterprises don't like to duplicate the user databases in multiple machines/applications. So, you also see the trend of 'Central Authentication Database' across internal servers and servers hosted outside. Many web applications are providing SAML based authentication for federate identity. Since web services need to talk to outside identity providers, there can be outbound connections. Note that, traditionally, servers in data centers only see inbound connections.

Enterprise administrator also requires facilities to upload the content and do other administrative activities on hosted servers. Typically FTP, SSH are some of the services required by administrators. Some applications might have web interface running on Port 80/443 for administration. To provide added security beyond user authentication, data center providers likes to control admin access from particular network(s), typically Enterprise Networks.

With more and more services (both Intranet and Extranet) being hosted in external data centers, the need for securing them is high. Collaborative services/servers such as wikis, share point, CRMs and other work flow servers are typically used to be part of Enterprise networks and only accessible for local users. They are being hosted in external data centers for reasons such as providing access from anywhere for employees, partners, contractors etc.. and also reduce the administration headache. Since they are exposed to access from anywhere, they are open for attacks from attackers. So, need for detection and prevention of exploits becomes much more than what data centers are used to. Quick look at the vulnerabilities published by NIST indicates (nvd.nist.gov) that SQL/XSS/LFI/RFI injections are on rise. You can also see number of wikis, blogs and other collaborative applications are targets of attackers.

Intranet servers when placed in external hosting providers' network, Enterprises would like the communication channels to be secure to protect data from eaves dropping. HTTP over SSL/TLS is one common method used to achieve data confidentiality on the wire. For security devices, placed outside of these servers, to do better job of access control, intrusion detection and malicious injections, it is necessary for these devices to see the traffic in clear. To achieve this, security devices should have capability to decrypt the SSL and do traffic/data analysis and if required redo the SSL. By the way, Since security devices are expected to be kept right in front of the servers, there may not be any need for redoing SSL. But important take way is that the security device should have capability to terminate the SSL connections.

From last few years, many web applications are using SOA (Service Oriented Architecture) which is built upon XML standards. Traditional ways of plain POST requests, JSON and PHP Objects are fast becoming thing of past. Any security device doing intrusion and data analysis need to move beyond POST, JSON and PHP objects and start interpreting SOAP and XML.

Data center providers provide services to many customers. Each customer requirement from security perspective is different. One generic security policy does not fit in these environments. You could have as many firewalls as number of customers, but that is not scalable from cost, space and cooling perspective. Virtualization in firewall/security devices comes in handy. Virtualization with VMWare/Xen also does not scale well. Old traditional virtualization scales well and suites well for data center providers.

Since security device comes in the way of traffic, things like performance of security devices should be high to support traffic rate that can be processed by servers/services it is securing. Latency, stability, availability and failover capabilities are some more important factors data center providers consider while selecting the security devices.

With above background, it is very easy to map to the features expected by data center providers on security device protecting their application and server infrastructure.

• Access Control : As you see above, access control some times need to go beyond IP addresses and TCP/UDP ports. Some web applications provide administrator and normal user access via same TCP/UDP port. Hence it is not possible to distinguish administrator and normal users from IP addresses and ports. Since many data center providers don't like admin access to be given from any IP address (for providing better security), but from some specific networks, it is required that the access control go beyond to application level information such as URL, Query parameters etc..
  
• Intrusion Detection and Prevention at L3-L7: As explained above, typical traditional intrusion detection systems without web application intelligence will not be able to detect intrusions all the time. There are many evasions being employed by attackers. Some evasions are at the IP and TCP level and more evasions are at the HTTP protocol level. Hence protocol intelligence is required. In addition, with SOA based web services, intrusion detection systems need to have intelligence to extract data from SOAP/XML messages. In addition to web application intelligence, they also need to have intelligence of other common services provided by hosting providers such as DNS, FTP, SIP etc..
  
• SSL Proxy: Network device should be able to terminate the SSL for further analysis on the protocol data.
• Virtualization: One physical hardware box is expected to support multiple virtual instances to reduce number of security devices in the deployment. Each virtual instance would need to have its own security policy configuration. It should be as good as different physical firewall devices. I, personally don't prefer VMware/Xen/KVM based virtualization for these environments. I prefer Traditional virtualization where only configuration data and run time states are instantiated for every context.
• DDOS attack detection and prevention.
• Traffic Anomaly detection and traffic Control.
  
• Performance: To achieve multi gigabit speeds, look for hardware architecture which is scalable.
  
• Stateful failover and high availability
• Logging & Auditing capabilities
  
• Intuitive central Management system

Optional features: Though they are not required, some data centers might find them useful

• Server side NAC: Provide facility for user based access control. NAC does user authentication and provides control access to the different features of an application based on the URL and other fields in the protocol. It also helps in correlating user actions and might be useful in auditing.
  

My intent here is not to go into many details, but provide some ideas on the features security vendors would need to think while providing security device solutions to data center market.

Guidelines for defining data models

TR-106 defines data model guidelines for creating new data models. For interoperability and to provide better clarity, I have defined some more guidelines.

Before you proceed further, please read this.

Guidelines:

Guidelines that are being followed today in defining data models.

• Data type: such as string, base64, integer, unsigned integer, Boolean, char etc..
• Range: In case of integer, unsigned integer values.
• Enumerations: Set of values that this parameter can take. Valid for both integers and strings.
• Min/Max length of string: In case of string and base64 types.
• Read/RW attributes for parameters
• Create/Delete for table nodes.

Additional guidelines:

General:

1. Values of some parameters don't reflect changes done by ACS. These are called action parameters. Each action parameter should have associated result parameter. Action parameters are always strings and it takes value "apply". ACS sets this value to perform the action. These parameters should be defined with attribute called "ActionParameter". This attribute should be set to 1 for action parameters. Associated result parameter fully qualified name should also be set with attribute "AssociatedResultParameterName".
2. For each associated action parameter, there should be result parameter. Result parameters are always strings. "success", "not available" are two values defined by this guidelines. Any other string value is considered as error in performing the action by application. Possible error strings are defined by applications defining the data model. Whenever action parameter value is set, associated result string value should get automatically set to "not available". When the application processing is done successfully, then this parameter value would be set to "success" by device. If application processing returns error, appropriate string value is set to this parameter.
3. Always define default values for all parameters except for mandatory parameters of table objects.

Table objects:

1. Identify the parameter that uniquely identifies the instance in the table object. Indicate that in an attribute "KeyParameter". ACS can check for the uniqueness of this value whenever the instance is created.
2. For each parameter that can't be changed after instance is created, indicate in DM that "DontChangeAfterCreate". Key parameter of the table object typically has this attribute set. There could be other parameters too based on application.
3. Identify all parameters that are mandatory during instance creation. Use "Mandatory" attribute for these parameters. ACS can ensure to send values for all mandatory parameters in "SetParameterValues" method that follows "AddObject" RPC method.
4. Indicate the default value in the data model if the parameter is not mandatory. This information is needed when ACS user asks it to reset the some specific optional parameter to default value.
5. For every table object, one parameter "Number Of Entries" must be present as per TR-106. I suggest to have one parameter "Number off Entries Supported" to represent the number of rows the device can hold. Both these parameters take integer values and both of them are READONLY. Note that names of the parameters could be different for different data models (table nodes). Hence there is need to associate these parameters to appropriate table nodes. To associate these parameters for the table object (table node), these parameter should have attributes associated with it indicating the full qualified table object. I call this attribute as "AssociatedTableNode". In addition to this attribute, two more attributes are needed - "CurrentEntriesIndicator", "MaxNumEntriesIndicator". CurrentEntriesIndicator attribute should be set to parameters indicating the current number of entries and MaxNumEntriesIndicator attribute should be set to parameters indicating the maximum number of entries that device can take for corresponding table.
6. If table is representing the ordered list, then it should have special attribute for the table object called "OrderList". Also it should have associated parameter name that indicate the priority of the ordered list. Call this attribute as "PriorityParameterName". This information is used by ACS as described in previous blog here. Priority parameter name takes integer values. Lower values indicate higher priority instances.
7. If table is an ordered list, it should also have one pair of action and result parameters for revalidation purposes. In addition to attributes it defined earlier for these two parameters, it should have additional parameter representing the table object name with attribute "AssociatedTableNode". ACS uses these attributes to know the parameters names to revalidate the states in the device.
8. As described under section "Nested table objects and special cases" of earlier blog here, there is a need for creating a table object instance with at least one instance of child table. ACS needs to know this special relationship so that it generates the screens such a way that it takes at least one child instance parameters as part of parent instance creation screen. This relationship information also useful for ACS to validate this information before sending configuration to devices. Each child table of this type should have one attribute "OneInstanceNeededByParent" set to 1.

Jericho forum - Is network security device market dead?

In one of the meetings I participated few weeks earlier, one person asked me a very interesting question - Will there be any security devices market in future? When I asked him why that question, he referred me to Jericho forum. Though I have some idea about jericho forum before, it got me interested to know more details about this.

When I first browsed through the forum publications, I thought that the question that was asked was fair. At first glance, it appeared that Jericho forum is proposing to add security along with application and data. But after spending few hours on position papers, Brochure and FAQ, it appears that Forum is not advising people to throw away their firewalls and security devices, but enhance security down to applications, data. Having said that, position papers still confuse readers with some inconsistent statements. I think that Jericho forum did not position their security concerns and resulting architecture very well and hence the confusion and mis-characterization in security industry.

Jericho forum described two main challenges - Business transactions that tunnel over HTTP/HTTPS and exploits/malware escaping traditional firewalls/security devices. I add one more challenge beyond HTTPS, that is, data itself may not be in clear - either it is encoded, encrypted or compressed.

It is true that traditional network address/service level firewalls are not good enough to protect resources from data level attacks and data misuse. Many applications are being developed on top of Port 80/443 (HTTP/HTTPS). Web Services (SOA) architecture is being used to develop multiple applications on a single machine with HTTP/HTTPS as transport. Any application service level filtering is possible only by devices having HTTP/HTTPS and web services intelligence.

It is also true that many newer malwares evade traditional signature detection - either by sending malware executables via HTTPS or constantly morphing themselves to avoid detection. One of the techniques behavioral analysis requires gathering the run time information such as registry entry modifications, listening port, any outbound connections, files being modified etc.. by running the executable on appropriate operating system.

With the challenges described and positioning it is doing, first impression I got was that Jericho forum is advocating adding entire security along with each application in the same machine. It took me a while to get rid of this impression. I guess the term "De-perimeterization' is confusing. I would like to think that Jericho Forum is proposing that security at Enterprise boundary is not good enough and the security is, additionally, needed closer to the applications/resources. So, there are multiple perimeters, with some perimeter having few machines or even one machine or one application. By the way, traditional firewalls and Ipsec VPN devices do very good job of providing access control to desktop systems based on the type of user and provide security connectivity to other branches of organization.

Though adding all security functions along with the application on the same machine provides better security, there are complexities:

There could be multiple machines running same application in cluster mode. In some deployments, it is observed that 100s of machines are used to share the load. In those, it is wise to move security functions such as "L4-L7 access control", "Intrusion Detection functions" to specialized security devices. It saves CPU cycles on application servers. It provides single control for administrators to manage security functions of the applications or set of servers and hence the management becomes easier. Some security functions such as terminating wireless connectivity and mobile device management don't really belong to one specific LOB (Line Of Business) application. They need to be outside of the application servers.

Having said that, some security functions can't be done well outside the LOB machines such as behavioral detection of malwares or when the data is encrypted or compressed with proprietary algorithms. They are better done as in end systems.

There is cost to apply some security functions outside the LOB servers. For example, Many LOB Servers implement security protocols such as SSL, XML Security etc.. Any access control device providing control at the XML field level must terminate the SSL connection, authenticate the user and decrypt and validate the XML documents before doing access control. There is inherent benefit too - It saves CPU cycles on the LOB Servers as it sees clear traffic. But, there may be some concerns in CSOs that some network elements has access to the clear data. If it is micro perimeter, then there may not be any concern. I guess Jericho forum is driving this point where the security perimeter is as close to the applications and data.

Security device vendors would like to make their solutions as generic as possible. They don't like to tie up the device functionality to one or few applications. That is where, standardization helps. I am happy to see that Jericho forum in their COA (Collaboration Oriented Architecture) position paper, chose the SOA and XACML. Both of these architecture heavily dependent on XML messaging. It provides common understanding for network elements outside of LOB servers and there by creating eco-system of vendors comprising security vendors, application vendors.

Having said that, I feel that the LOB applications must have their own security based on the application - Such as authentication, multiple roles, role based access, Auditing etc..

In summary, CSOs need to understand that Enterprise boundary security with traditional network level firewalls is not good enough to protect the data and resources. Application specific security is must. Some security functions can be done outside of LOB servers, but the security device must be as close to the LOB servers as possible. So, I don't see network security device vendor market drying up.

OpenDNS - Domain filtering Cloud computing Service

I recently came across *opendns* service. Visit www.opendns.com to find out more details on this service. "opendns" name is a confusing name given the type of service they are providing. Initially, I thought it is some thing similar to dynamic DNS.

Operation:
OpenDNS mainly provides domain blocking capability. Domains are arranged in multiple categories. It provides facility for users to configure the categories which are to be blocked. It also provides facility for users to create white list and block lists of domain names.

This service is using DNS protocol. It expects the user machines or routers to use their DNS Servers for domain name resolution. As part of DNS resolution, it appears to be extracting the domain name from DNS request packet, search in their local database, get the category and look in user preferences. If category is configured to be blocked or if domain is in the block list, then openDNS server seems to sending DNS response with its own IP address. Due to this, user browser session ends on this IP address. OpenDNS seems to be doing search on the domain name (Host field of HTTP request header) again to determine the category and it shows nice page indicating why it was blocked.

Comments:
This service is good for residential users and even for business users. Residential users get benefited by blocking adult sites for kids and also stop while visiting phishing sites. Businesses also benefit as it stop users going to phishing sites. Having said that, this works fine only when CPE devices work in conjunction with openDNS service. Before going into the capabilities required in CPE devices, let me list down some limitations/issues in using opendns service.

• Privacy issues: Some businesses find it difficult to trust opendns provider due to privacy issues where *openDNS* provider comes to know the sites business users visiting. Business may like to have facility for some users to bypass this service and for some mandate this service. Also, businesses like to have facility to bypass openDNS based Domain name resolutions for some specific domain names.
  
• User or group based white list/block lists/category selection: There are different types of employees in businesses. Also, there are different types of home users - kids, parents, visitors, teens etc.. OpenDNS provides only one profile for all users. This may not be sufficient for many businesses and residential users.
• Evasion: Kids can evade these filters if they use IP address in their browsers.
• Updating Dynamic Public IP address with the opendns account
  

How can CPEs help?

User/Group based lists: User/Group based lists support is only possible if openDNS updates its functionality. One possible way is to have special DNS request with added information such as GroupID. OpenDNS Service can rovide facility in openDNS portal to create category selection/blocklist/whitelist onper group basis. Since one can't expect all PCs to support this special enhancement in the DNS protocol, this kind of support is possible with CPEs implementing DNS proxy to convert DNS requests to add Group ID.

Privacy: CPE devices can help in mitigating privacy issues by providing support to create 'skip' lists - Source skip list and Domain skip list. If the source IP address of the DNS request packet from internal PCs matches the entry in 'Source Skip' list, then it bypasses openDNS based resolution. It can do this by sending the DNS request to one of ISP Domain Name Servers. 'Domain Skip list' is checked for domain names inside DNS request sent by local machines. If there is a match, then it bypasses the openDNS resolution.

Evasion: CPE devices can monitor HTTP requests and check the 'Host' header line. If the 'Host' header line does not have domain name, but IP address, then we can certainly say that domain name is not used while browsing the site. CPE devices can provide configuration on type of action to take. It can provide options like 'Inform' and 'Deny'. 'Inform' action informs parent in case of RG environment or admin in case of business environment. 'Deny' action drops the connection and might even present local HTML error page to the user. Here too, we should 'skip' lists to help scenarios where some sites are only reachable via IP addresses - for example Intranet sites or partner sites etc..

There is another kind of evasion possible too. Local users using their own DNS Server or some public DNS Servers. CPE can check all DNS requests and ensure that only specified DNS Servers are used. It could even do Destination NAT to the required DNS Server IP address.

Dynamic IP address update: Today it is expected that special program is run in the PCs behind CPE routers. It does not work well if we have many machines or machines which do not run the software provided by OpenDNS folks. CPE device can help in those matters where it updates the dynamic public IP in openDNS Servers. CPE devices are already equipped with updating dynamic public IP addresses in DYNDNS servers. They could do additional job of upating in openDNS Server too.

SME IPS and Cloud Computing

Cloud computing providers are betting on small and medium businesses flocking to them. Large number of SME businesses are already using email service provided by cloud computing providers. It appears that this trend is being spread to other services such as File Service, backup service and web application services.

Businesses offloading their intranet and extranet services to the providers would be left with desktops and some minimal servers in their network. I have my own doubts on merits of moving Intranet services to providers, but that discussion belongs to some other topic.

Desktops normally don't provide any services i.e they don't run any servers. May be printers and other networking equipment have some services, but they are limited to internal machines. Hence firewall protection allowing only internal machines is good enough.

Basically, the requirement of server side security function beyond firewall is going to be less in these environments. In addition, many hackers are now moving towards soft targets i.e desktops and applications running on desktops such as browsers, viewers etc..

Many IPS/IDS devices in the market today protect servers better than clients. Due to movement of services to providers and with increase of client side attacks, IDS/IPS vendors must support better client side detection to survive.

IDS/IPS vendors realized this and moving towards this, but not as fast as one would like to see. By Mid-2009, I believe that many IDS/IPS boxes in the market will have sophisticated engines to support client side attack detection and prevention.

Assurance of firewall availability for critical resources : TR-069 support

I guess I have been harping that network security devices are stateful in nature. Let me say that again here :-) They create session entries for 5 tuple connections. DDOS attacks can consume these resources. There are several techniques used to maximize firewall availability. Some of them I discussed before are - Session inactivity timeout functionality, TCP syn flood detection and Syn Cookie mechanism to prevent SYN floods and connection rate limiting.

Above techniques do not guarantee that legitimate connections are not dropped. Rate throttling feature does not distinguish from genuine connections to DDOS connections. But, some resources are very important and access to/from these resources must be made available all the time. That is, some assurance of firewall availability for these critical resources is required.

During DDOS attack and worms outbreak, systems in corporate network should have access to central virus database server to get newer virus updates. Even if some systems in corporate network are compromised and participating in DDOS attacks, other systems should continue to access critical resources while problem is being fixed. Similarly, access to corporate servers should be maximized during DDOS outbreak.

Though all issues can't be solved, enough facilities should be there for assurance of firewall availability for these critical accesses.

Many firewall today support feature called 'Session Reservation and Session Limits'. Using this feature, certain number of sessions can be reserved to individual machines/systems. This feature also limits the number of simultaneous sessions for some non-critical systems/machines.
One use case example: Let us say that a Medium Enterprise has 500 systems. Say that this company bought a UTM firewall with 50000 session entries. Administrator can reserve 20 sessions and limit 100 sessions for each PC. That is, 10000 entries are reserved. Rest of 40000 sessions are free for all. When all 40000 sessions are used up, then reserved sessions are available for PCs. Each PC can use its reserved 20 session entries. Thereby, when there is a DDOS attack, even after 40000 session entries are used, these PC continue to have access 20 more session entries. No other system can occupy these reserved sessions.

Session reservation database is set of rules. Each rule contains following information:

• Rule ID: Identification of the rule.
• Description: description of this record.
  
• IP Address information: IP addresses for which this rule applies. All the action information in this rule is specific to each IP address.
• Connection Direction: Outgoing or incoming. Indicates whether the sessions to be reserved for connections made by machines represented by 'IP addresses' or for connections terminated by these IP addresses. 'outgoing' indicates this rule is applied for connections originated and 'incoming' indicates whether this rule is applied for incoming connections.
• Zone : Indicates the zone ID. If 'Connection Direction' is outbound, then zone indicates the destination zone. If 'Connection Direction' is inbound, then zone indicates the source zone.
  
• ReserveCount: Number of sessions reserved for this rule.
  

Session Limits database also contains set of rules. Each rule contains following information:

• Rule ID: Identification of th rule.
• Description
• IP address Information: IP addresses for which this rule applies.
• Connection Direction: Outgoing or Incoming.
• Zone: Zone ID
• Limit Count: Number of maximum sessions for each of IP addresses.

TR-069 data profile:

• internetGatewayDevice.security.VirtualInstance.{i}.firewall.maxSessionReservationRules: R, unsigned Int
  
• internetGatewayDevice.security.VirtualInstance.{i}.firewall.maxSessionLimitRules R
  
• internetGatewayDevice.security.VirtualInstance.{i}.firewall.sessionReservations.{i} PC
• ruleID: RW, Unsinged Int, Value between 1 and maxSessionReservationRules.
• description: RW, String(128)
• ipAddressType: RW, String(32). It takes values such as 'immediate', 'ipobject'. Immediate indicates that IP addresses are given as values and 'ipobject' indicates the IP address information points to one of the IPObjects.
• ipAddresses: RW, String(64) - f the type is immediate, then it can be single IP address in dotted decimal form, subnet by providing network IP address and prefix in terms of number or range of IP addresses with '-' in between low and high values. If the type is 'ipobject', then it has one of ipobject names from internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.IPValueObject.{i} table or internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.IPFQDNObject.{i} table. 'any' is special value indicating all source IP values. Examples: 10.1.5.10 or 10.1.5.0/24 or 10.1.5.1-10.1.5.254
• connectionDirection: RW, String(16). It takes values 'outgoing', 'incoming'.
• zoneID: String(32), RW - One of the Zone IDs. It takes value of ZoneName from internetGatewayDevice.securityDomains.VirtualInstance.{i}.Zone.{i} table.
• reserveCount: RW, Unsigned Int.
• internetGatewayDevice.security.VirtualInstance.{i}.firewall.sessionLimits.{i} PC
• ruleID: RW, Unsinged Int, Value between 1 and maxSessionLimitRules.
• description: RW, String(128)
• ipAddressType: RW, String(32). It takes values such as 'immediate', 'ipobject'. Immediate indicates that IP addresses are given as values and 'ipobject' indicates the IP address information points to one of the IPObjects.
• ipAddresses: RW, String(64) - f the type is immediate, then it can be single IP address in dotted decimal form, subnet by providing network IP address and prefix in terms of number or range of IP addresses with '-' in between low and high values. If the type is 'ipobject', then it has one of ipobject names from internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.IPValueObject.{i} table or internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.IPFQDNObject.{i} table. 'any' is special value indicating all source IP values. Examples: 10.1.5.10 or 10.1.5.0/24 or 10.1.5.1-10.1.5.254
• connectionDirection: RW, String(16). It takes values 'outgoing', 'incoming'.
• zoneID: String(32), RW - One of the Zone IDs. It takes value of ZoneName from internetGatewayDevice.securityDomains.VirtualInstance.{i}.Zone.{i} table.
• limitCount: RW, Unsigned Int.

Common mistakes by TR-069 CPE developers - Tips for testers

There are some mistakes which can go undetected for a long time. Testers and certification agencies need to watch out for those. I tried to list down some of those mistakes in TR-069 based CPE devices.

Problems associated with Instance numbers: It is expected that instance numbers are persistent across reboots. But, this does not seem to happen always. Typical problems observed are:

• Instance numbers are not stored in persistent memory: If instance numbers are not stored and retrieved from persistent memory, then CPE device may provide duplicate instance number next time when it comes up for AddObject RPC methods. ACS may reject and new records will not get created. On a side note : Surprisingly some ACS systems don't even care to check for duplicate instance numbers.
  
• Instance numbers are stored and retrieved, but the relationship with actual table records is not maintained: That is, ACS might have one view of instance number to the row and CPE devices have different mapping. This will have problems in modification of values. ACS thinks that it is modifying parameters of specific row, but CPE modifies some other record.
• Instance numbers are stored and retrieved for records which were created and populated with values, but not saved for unpopulated records: This scenario occurs when AddObject is successful, but before the record is populated with values, the CPE is restarted or got restarted for some reason. If CPE does not restore these instance numbers, the configuration of that record will not be successful (when CPE comes back) as ACS sends the configuration with instance number returned by CPE before.
  

Problems associated with ParameterKey: ParameterKey is also expected to be stored in persistent memory across reboots. Some ACS systems use this as a configuration checkpoint. ACS systems know the CPE configuration by reading paramterKey value from the CPE. ACSes use this to figure out the 'configuration diff' between what it thinks CPE has versus the latest configuration it has for CPE. Only this difference is typically sent to CPE. Due to this, it is expected that parameterKey is stored in persistent memory along with the rest of configuration. Typical Problems observed are:

• ParameterKey and configuration is not stored in persistent memory: There are two issues due to this. ACS reconfigures the device every time it restarts. This could be a problem as the device is unavailable until ACS configures the box.
  
• Configuration is saved and retrieved, but not ParameterKey: Once the device restarts, ACS thinks that CPE does not have any configuration and reconfigures the device from scratch. Since configuration is actually retrieved, there would be many duplication errors. ACS might get confused and might stop pushing the configuration until admin is intervened. In some cases, I observed some intelligent ACS sets the device to factory defaults and configures by sending entire configuration without any manual intervention from ACS administrator.
  
• Configuration and parameterKey are saved at different times: This is dangerous. Essentially the mapping between configuration and parameterKey is broken. When device comes back, ACS view and device view of configuration is different.
  

Problems associated with Access Control and Notifications: Access control is one feature CPE vendors forget to provide. Typical problems observed are:

• No support provided for Access Control: This is one of the important features for managed service providers. If end user changes the important configuration and makes mistake, debugging may take significant time for Service providers. Due to this , service providers would like to allow change of only specific configuration by subscribers. Without having this support in device makes that intention difficult.
  
• Support provided for Access control and notification only for some variables, but not all.
• CPE sends notifications for unasked variables

Many problems, I believe, are result of configuration parameter mismatch between local configuration methods and TR-069 method. For example, Web interface provided on CPE might have been developed long ago before TR-069 based remote configuration method is implemented. Some CPE developers forget to integrate their web interface backend functionality with TR-069. In some cases, they do poor job of integration.

Cloud computing security is going to catch up.

Please see this article in information week.

Google and IBM are teaming up together to provide cloud services. Google is already providing email and storage services and they want to go beyond that.

One interesting thing that was mentioned in the article is
"With the exception of security requirements, "there's not that much difference between the enterprise cloud and the consumer cloud," Google CEO Eric Schmidt said earlier this month during an appearance in Los Angeles with IBM chief Sam Palmisano."

One more quote from the article:
"The cloud has higher value in business. That's the secret to our collaboration."

Another thing I observed in the article is their planned usage of Xen.

Combining all of them put together:

• Cloud computing requires security. Otherwise, Enterprises may not be able to offload their servers to cloud.
• Cloud computing makes use of Virtualization.

I was giving choices in my earlier blog on *Cloud computing and Security*. Though information week article is not giving enough information on how the security services are going to be offered, but they will start thinking soon.

I am beginning to think that both kinds of models which I suggested earlier would be used.

• Flexibility for Enterprises to put their preferred vendor security products as virtual appliances.
• Providing security using one mega security appliance.

My prediction is that mega security appliance is required to provide typical infrastructure security. Virtual appliance flexibility will be provided for specialized security.

Packet processing applications - Updating to Multicore processors

I described the need for session parallelization and some tips on programming here. There are many network packet processing applications which don't take advantage of SMP systems. I tried to describe the steps required to convert these applications to run on SMP systems with as less modifications as possible.

Target packet processing applications:

• Packet processing application that maintain sessions (flows) such as firewall, IPsec VPN, Intrusion Prevention Software, Traffic Management etc..
• Packet processing applications where significant amount of cycles are spent on each packet. Session Parallelization technique uses some additional checks and queuing (enqueue and dequeue) operations of the packets in the session. Session Parallelization technique only works, if the number of CPU cycles required for packet processing is much more than the CPU cycles it takes for few checks and queue operations. Otherwise, you are better off taking locks during packet processing. For example, IP forwarding packet processing application can be parallelized using session parallelization technique.

Let us examine the typical packet processing application requirements.

• Upon first packet of the session, they create a context - Session context. These session contexts are kept in some kind of run time data structure such as Hash buckets, linked lists, arrays, some container etc.. - I call it as Home base structure.
• Subsequent packets of the session gets the session context from the home base Data structures.
• Session contexts are deleted either by timer or by user actions or due to some special packets.
• Session contexts are typically 'C' structures with multiple members (states)
  
• Some members are set during session context creation time and never changed. I call them "SessionConstants".
• Some members are manipulated during packet processing and those values are needed by subsequent packets. And these are not required by any other function other than packet processing. I call them "Session Packet Variables".
• Some members are manipulated by both packet processing and also in non-packet processing contexts. I call those "Session Mutex Variables".
• Some variables are used as containers by other modules. That is responsibility of storage and retrieval is responsibility of those modules. I call them "Session Container Variables"
• Any packet processing application might contain multiple modules and each module has its own sessions (control blocks). To improve performance and other reasons, yet times, modules store other modules' control block references in their sessions. Also, current module session might be referred by other modules. when the session is actively used by other modules by having pointer to the session, it is necessary that session is not freed until all active references are dereferenced. At the same time, delete operation can't be postponed until it is deferenced by every other module. To satisfy both the requirements, each control block typically contains two variables - "Reference Count" and "Delete flag". Reference Count is incremented when external modules store the reference of the session. When the session is being deleted (either due to inactivity timer, due to special packets or due to user actions etc.. ) and if the reference is used, then modules set the "Delete flag" of the session. Any search on the session ignores the sessions having "Delete flag" set. Also the module is expected to inform the external modules that the session is being deleted. Upon this notification, external modules are expected to remove the reference and as part of it they decrement the reference count of the session. Session is freed only after all references are removed and Delete flag is set.

Process to follow to make applications SMP aware (Example: Applications running in Kernel space)

• Identify the Session or Control block of the target module.
• Define two additional variables in the control block - Reference Count and Delete flag.
• Identify the home base structure. Since the sessions can be created in multiple CPU contexts, define the lock for this structure. It is better to define Read/Write lock. Read lock can be taken when the session is searched in home base data structure. Write lock needs to be taken while adding or deleting the control block to/from the home base data structure. Ensure to increment the reference count while adding to home base structure. Remove it from the home base structure only if reference count is 1 and Delete flag is set to TRUE.
• Some times home base could be some other module's control block. In this case, it is responsibility of other module (Container module) responsibility to store, retrive and reset this control block reference atomically.
• Always ensure to initialize the control block completely before adding it to the home base data structure.
• Ensure that control block reference count is incremented within the home base lock for both add and search operations.
• Identify session constants. There is nothing that need to be done for SMP.
• Identify Session Packet Variables. There is no need to lock these variables if "Session Parallelization" technique is employed.
• Identify Session Mutex variables. Further identify the logic groupings of these variables. For each of these logical groupings:
• Define a lock. This lock can be global lock or session lock. Session locks are good for performance reasons, but it requires more memory.
• Ensure to define MACROS/inline/functions to manipulate and access variables of the group.
• Identify container variables. If this control block is home base structure for other module control blocks, then define MACRO/inline/functions to add/retrieve/reset. These macros would be used by other modules.
• Define this set of MACROS for each container variable.
• Define a session lock for each container variable. Above MACROS use this lock to protect the integrity of the container value.
• Since ADD macro is keeping the reference of foreign module control block, ensure to increment the reference count of foreign control block. To allow this, ADD macro should expect the fucntion pointer being passed to it by the foreign module. It is expected that the function ponited by function pointer is used to increment the reference count.
• Expect the "Increment Reference Count" function pointer passed to the RETRIEVE macro. Since RETRIEVE macro returns the pointer of foreign module, it is necessary to increment the reference count to ensure that pointer is not freed prematurely.
• Ensure that complete functionality of ADD/RETRIEVE/RESET MACROS work under the lock defined for each container variable.
• Some guidelines on Reference Count and Delete Flag variables.
• Define "IncRefCount", "DefRefCount" and "SetDelete" and "GetDelete" macros.
• Define a lock to manipulate and access "Reference Count" and "Delete flag" variables.
• Use this lock only in above macros/inline functions.
• Use above macros for manipulating and accessing these variables.
• Do first level of cleanup of control block when "SetDelete" macro is called. This cleanup involves informing external module of its intention of going down and also removing any foreign module references. Foreign references are ones this module stored as part of its packet/session processing. Removing foreign references involves decrementing the reference count of foreign reference and invaliding the reference in the control block.
• Do final cleanup such as freeing up memory for control block only when "Reference count" is 1 and "Delete flag" is set to TRUE. Before freeing up the memory, remove it from the home base structure. This condition typically happens as part of "Decrement Reference Count" macro.
• In non-SMP scenarios, the reference counts are incremented when external modules store the reference of the session in their modules. In SMP scnearios, while one processor processing the packets of the session, another processor can delete the session. Due to this, it is expected that reference count is incremented even while packet processing is happening. That is, whenever "Search" for session is done or when the session is being retried from container module, the reference count must be incremented. As indicated above, retrival of the session, including incrementing the reference count must be done either in data structure/container lock. Since incrementing the reference count happens within its own lock, you would see lock within lock in this scenario. That is, reference count lock is taken with data structure or container lock taken. This is Ok as data structure or container lock is never taken under reference count lock even in cases where session is added or removed from the data structure/conatiner.

DDOS Mitigation functionality - Tips for Admin

There is some discussion going on in focus-ids mailing list on DDOS attack mitigation. That discussion prompted me to write this article. ISIC, UDPSIC, TCPSIC, ICMPSIC are some tools used to measure the network security device effectiveness of detection and mitigation of DDOS attacks. As we all know one of the main intentions of DDOS attacks is to make the service, network or target unavailable. By looking at the packets, you can't see the difference between normal genuine packets and packets generated by DDOS attacks. This makes it difficult to stop these attacks based on signature based methods.

One of the properties of many of DDOS attacks is that they try to make the discovery of source of attack difficult to find. There are two types of DDOS attacks that are common.

• Spoofing of source IP address in the packets: DDOS attacks are generated by spoofing the source IP address of the packet. ISIC, UDPSIC, TCPSIC and ICMPSIC tools simulate these kinds of attacks. Any packet that is sent back to the source does not reach the attacker. Due to this, TCP based sessions don't get established. Note that non-TCP sessions don't have connection establishment phase.
  
• Botnets : The attacker instructs the agents which were installed on compromised hosts across the globe to bombard the target. Attacker keeps changing the hosts that attack the target. Thereby, in effect making the source discovery ineffective.
  

Now, in addition to Botnets, there is a third kind of DDOS attack. Recent DDOS attack on cnn.com by Chinese hackers is one example. Here too, sources are known, but there are many. Note that here the attack on cnn.com is not generated by botnets, but supposedly by out of patriotism. Some in China felt that CNN is biased in its reporting on Olympic torch and its linkage to Tibet religious freedom. As I understand, it was simple attack, where it connects to cnn.com website and accesses some URL for every 2 seconds. This attack executable was distributed and many home users in China, out of patriotism, executed it.

DDOS attack incident detection may be easier, but mitigation is difficult. If the intention of the attack is to consume the bandwidth of target site, there is nothing much the target network administrator can do. Target company/organization needs to depend on its ISP to block the flood of packets. Gathering as much information as possible and providing that information to ISP is one of the things the administrators can do.

The current trend of DDOS attacks go beyond consuming the link bandwidth. With less number of hosts participating in the DDOS attack, these attacks consume the CPU, memory bandwidth of target networks/servers. I feel the network security appliances providing DDOS attack mitigation functionality can help in this scenario. It can not only provide detection, but can stop bombardment of servers.

There are multiple products *DDOS mitigators* in the market claiming to solve some of above problems. Many IPS boxes also support this feature.

If you are hosting some servers, you can be a victim. As an administrator, I look for following features from these appliances.

DDOS attack consumes 1Mbps link by making 512 connections/sec (approximately) . Any DDOS mitigator, ideally should be able to process 512 connections in every second for 1Mbps link. If the connection is maintained for 20 seconds (which is typical), then the connection capacity needs to be 10K. For 100Mbps link, DDOS attack mitigation appliance needs to support 51200 connections/sec and should have 1M session capacity. With this capacity and connection rate, it can do better job of protecting internal networks/servers/other stateful security devices without itself getting bogged down.

DDOS mitigators are expected to limit the amount of traffic that goes to the internal servers/machines/networks etc.. Each resource in the network would have some limitations on how much traffic, connections, connection/sec it can take. Adminis, once they make a list of resources and their limitations, should be able to configure DDOS mitigators. DDOS mitigators must ensure that the resources are not flooded and it should shape the traffic accordingly. DDOS mitigators need to provide features like:

• Ability to configure
• connections/sec
  
• Packets/sec
• Bytes/sec
• On per resource basis - Server/machine basis, Network basis
• From a given source with respect to IP address range, Subnet.
• Ability configure to filter traffic on combination of 5 tuples.

As you have observed, admins not only would like to shape the traffic to internal resources with respect to connections/sec, maximum number of connections and throughput, but also would like to have these limits from particular source(s). Yet times, I also observe that there is a requirement to limit the amount of traffic within each 5 tuple connection, between any IP address combination. Mitigators need to provide this flexibility without expecting admins to create many rules. Many times, it is not possible to create rules with all combinations of IP addresses. DDOS mitigators need to provide flexibility of creating rules with ranges, subnets along with provision to configure granularity to apply the specified traffic rates. For example, admin should be able to configure ANY to 'Internal HTTP Server IP addresses' with 10 connections/sec for every combination of source IP and destination IP. If there are 100 different sources are trying to access internal HTTP Servers, DDOS functionality should be able to rate limit the number of connections to 10/sec for each of sources independently.

As with any security device, it must also support multiple zones and provide flexibility with respect to zones. In case of hosting environments, provider may be servicing multiple customers. So, virtual instance, with each instance belonging to a customer is needed. In case of Enterprise environments, normally only one virtual instance would be used.

Flexibility is expected to be provided to disable limiting of traffic for some source networks. These networks could be networks belonging to remote offices. This feature is called white listing.

Ofcourse, it is expected that DDOS mitigators provides facilities to stop half open connections by providing TCP syn flood protection, UDP based session exhaust protection facilities, facilities to configure service inactivity timeouts for interactive protocols etc..

UDP Broadcast Relay : TR-069 Support

UDP broadcast relay functionality became very popular due to NetBIOS. Broadcast packets are used by NetBIOS for name resolution. Windows Network neighborhood is one functionality that makes use of NetBIOS name service. Due to broadcast functionality, NetBIOS name service works within subnet. If there are multiple subnets, then WINS Server is required. Broadcast relay functionality in routers separating subnets eliminates the need for WINS Servers. Name resolution using UDP broadcast relay function can even be extended to networks in remote offices by relaying broadcast packets over VPN tunnels.

UDP broadcast relay functionality in routers receives broadcast packets and send to other subnets by replacing destination IP of original packet with destination subnet broadcast address.

Since firewall/VPN gateways are also routers, this functionality is implemented in many firewall/VPN gateways. These gateways provide control for administrators on type of broadcast packets to relay and destination subnets to relay to. Multiple of these rules can be created for different types of broadcast addresses.

Configuration consists of set of rules. Each rule containing incoming braodcast IP address/interface and relay subnets/interfaces. Rules can be created and deleted by administrator.

TR-069 profile:

• internetGatewayDevice.security.VirtualInstance.{i}.UDPBroadcastRelay.{i} PC
• name: String(32), RW, Mandatory - Identification of broadcast relay rule. Once rule is created, this value can't be changed.
  
• description: String(128), RW, Optional
• enable: Boolean, RW, Mandatory: Value 1 indicates the record is enabled and 0 is used to disable.
• incomingBroadcastAddressType: String(16), RW, Mandatory - Indicates whether the broadcast address is represented as an IP address or Interface identifier. Takes one of the values "ipaddress", "interface".
• incomingbroadcastAddress: String(128), RW, Mandatory - Either dotted IP address or Fully qualified TR-069 instance of VLAN, LANDevice, WANPPPConnection or WANIPConnection etc.
• incomingbroadcastPort: Integer, RW, Mandatory - Destination Port of incoming broadcast packet.
  
• internetGatewayDevice.security.VirtualInstance.{i}.UDPBroadcastRelay.{i}.relayTo.{i} PC
• relayBroadcastAddressType: String(16), RW, Mandatory - Indiacates whether the relayTo broadcast address is specified as IP address or interface - Takes one of the values "ipaddress", "interafce".
• relayBroadcastAddress: String(128), RW, Mandatory - Either dotted IP address or fully qualified instance of interafaces from VLAN, LANDevice, WANPPPConnection or WANIPConnection.

In case of remote subnets, relayBroadcast is specified as remote subnet broadcast IP address. If the subnets are directly attached to the router, then interface names can be used in relayBroadcastAddress field.

Configuration Synchronization between TR-069 devices and ACS

Most of the times, configuration for TR-069 based devices is done at the provider end (ACS end). ACS pushes the configuration when device connects to it. TR-069 also provides facility for subscribers to change the configuration locally, but providers have control over which functions of device can be changed by subscriber. 'SetParameterAttributes' RPC method defined by TR-069 specification carries the 'Access control' attributes of parameters.

When allowed to change the confguration by subscribers, if no caution is taken, the configuration view can be vastly different between device and ACS over time. That might result to wrong conclusions by people administring the device. Fortunately, TR-069 provides a way for ACS admin to notifiy the changes to ACS when changes done by local subscribers. Like in 'Access Control' , notification attributes are also sent using SetParameterAttributes RPC method. If Synchronization of configuration is required, it is necessary that ACS sets notification attributes on parameters which are allowed to be changed by local subscriber.

It appears that notification of configuration change works smoothly between devices and ACS on modifications of existing parameters. But, it runs into rough weather when local subscriber adds a record in table objects or deletes a record from table object. Based on my not-so-much experience on different devices and ACS systems, it seems that this was not taken care well and TR-069 also not helping in this regard.

Let us examine different approaches that are possible.

• Approach 1: Definition of one parameter 'Number Of Entries' for every table object. ACS can set the notification attribute on this parameter. Whenever local subscriber adds/deletes an instance from the table object, this parameter gets incremented/decremented. Since notification is set on this, ACS gets informed. It has performance disadvantage. When 'Number of Entries' parameter is changed, ACS gets notified with new value of this parameter. Now it is responsibility of ACS to find the difference between its configuration and configuration in the device. That might require walking through the table in device and making modifications to its database. Another disadvantage is that, this approach does not work if there is any existing data profile that has table objects without this special parameter 'Number of Entries'.
  
• Approach 2: Setting the notification attributes on 0th instance: It is observed that many devices don't use 0th instance for records in table objects. This instance can be used to set the notification. If notification is set on 0th instance, it can be treated to indicate Add/Deletion of records to/from the table. Whenever new record is added/deleted by subscriber, device needs to check the notifications on 0th instance. If it is set, it can send instance number of newly created record or instance number of deleted record as parameter and value 1 for 'Add' and 2 for 'Delete' as part of 'ParamterList' of Inform method. By this, ACS knows instance number of newly added record or deleted record. In case of newly added record, it can issue 'Get' operation on specific instance number to read values of parameters. In case of deleted record, it can remove the record from its database.

Approach1 does not require any changes to the TR-069 specification, but has disadvantage of performance issues if the number of instances in a given table are high. Approach 2 is clean, but requires addition to TR-069 specification or clarifications in TR-069 specification.

If anybody choose to use TR-069 based device management for Enterprise devices, I strongly suggest to go with Approach 2.

TR-069 Protocol and applicability in Network security devices - Opinion

Recently a security software developer asked me a question. He wanted to know whether TR-069 protocol is suitable for managing network security devices. Further he wanted to know what enhancements I would like to see in TR-069 protocol, if any.

I am sure that quite a bit of thought had gone into defining the RPC methods and their usage. It is certainly easier to create new data models and software associated within device and ACS implementations when you compare with SNMP based management.

TR-069 is certainly suitable for network security application configuration. But there are some points developers should be aware of.

AddObject method:
This is one thing I have difficulty in understanding the intentions behind in defining 'AddObject' method in TR-069 protocol. 'AddObject' method does not take any parameter values. To add a row in a table, it requires two methods from ACS - AddObject followed by SetParameterValues method. Due to this, it has some complications and additional logic in TR-069 implementations in security devices.

Configuration of many security functions such as firewall, IPsec, AntiX and IPS consists of multiple rules (rows). Each row has its identification and value of these identification parameter (Let us call it as Instance Key) are typically configured by administrators. This identification parameter must be unique within the table. Some configuration databases identify the rows by integers and some identify by strings. Once the record is created, security software does not allow changes to identification values. Of course, other parameters of the rule (row) are allowed to be changed.

TR-069 defines its own identification for each row in the table called instance number. Instance number is of integer type. Instance number is returned in AddObject response. This can't be used to identify the record in a table as far as security applications are concerned. I guess this instance number is mostly for TR-069 protocol. Since record name/ID (Key) is not part of AddObject, record can't be created in the security applications upon reception of AddObject by the device. It needs to wait until SetParameterValues method is sent by ACS with the record identification value.

Though this is not a major hurdle of using TR-069 in configuring security applications, but it could have been avoided if AddObject allows setting up of parameter values.

I don't see any reason why separate instance number is required as it is defined in TR-069. By avoiding instance number and replace with application Instance Key, it provides many advantages:

• Device does not need to maintain the state between "Add Object" and "Set Parameter Values" RPC methods to create the instance (row) in the applications.
• Device complexity increases as it needs to maintain the mapping between instance numbers and application instance key values even across reboots to maintain consistency between devices and ACS.
• ACS does not need to maintain the mapping of instances with each device instance numbers. Since ACS is expected to manage thousands of devices, it needs to maintain this run time mapping information which limits the ACS scalability.

I like to see the enhancement the TR-069 protocol where it avoids instance number approach for the rows. As a matter of fact, local management engines don't have special instance numbers for each row of tables. One of the parameters itself used as the key to identify the instances in the table. With that in mind, I like to see following approach.

• One of the parameters in each table object in data model can be identified as the instance key. It requires only one parameter to identify the instance at that level due to tree structure of the data model and nested table objects. Table object is already identified uniquely due to upper (parent) table objects. Due to this, one parameter is good enough to identify the instance within the table object. By having one of data model parameters as the key parameter, ACS also can identify the row by this parameter value.
• Today configuration transaction is limited to one SetParameterValues method. Adding a row is outside the transaction today. To make the adding instance also as part of transaction, row creation along with its parameters and values should be made part of SetParameterValues RPC method and eliminate AddObject RPC method.
• Any modifications to the instance at later time need to send the key value along with the parameter names. Existing instance number position in the parameter name can be replaced with the key value.
  

Mandatory parameters in one RPC method:

Security application makes some configuration parameters 'must to have'. Some of these configuration parameters can have default values, but not all. My experience shows that many mandatory parameters don't have any default values. Due to this, security application software typically expect all the mandatory parameters values as part of its record creation. As discussed before, actual object creation in security software module is done when the first 'SetParameterValue' RPC method is received (after AddObject). So, it is expected that ACS sends SetParameterValues method with all mandatory parameters and its values. TR-069 protocol does not specify any rules in regards to this. Due to this, ACS systems have a choice of sending these parameters in multiple RPC methods. It complicates the device implementation, where it needs to wait until all mandatory parameters are received. This is not practical as device does not know when to give up waiting for the mandatory parameters. I believe, there is an understanding that ACS systems always send most of the parameters' (including mandatory parameters) values in one RPC method.

Since this information is not documented, you might often hear from your QA guys that it is not working as expected. I wish that TR-069 specification clarifies this clearly. By the way, if the approach as indicated in above section (under AddObject method section) is implemented, then this is a mute point.

Default Values for Parameters

TR-069 data model guidelines don't clearly specify that all optional parameters of table objects and normal parameters must have default values.

Many security devices and I am sure other devices also have requirement of resetting to factory defaults upon user action. In addition, some devices provide option for administrators/users to set all/some optional parameters to default values on existing configuration. To facilitate this, I think TR-069 data model definition must mandate setting up the default values for non-mandatory parameters.

Factory Defaults

Many device applications are shipped with default configuration. Many devices also support resetting the device to factory defaults via both hardware and software means. TR-069/TR-104 does not specify the ways to define the factory defaults. Due to this, when device resets its configuration to factory defaults, ACS does not know the configuration of device until it reads the configuration from the device by traversing through the data model template tree.

An additional benefit of defining factory defaults in a standard fashion also helps device to do the factory reset from central place (TR-069 client in device) without letting each application to do its own factory reset.

Validations

TR-069/TR-104 data model definition facilitates the parameter value validations such as data type, possible enumeration values, min and maximum length in case of string and base64 data types, range of values in case of integers. But, there are no validation definitions on number of instances that can be created. Many applications limit the number of instances on per table object basis. Having this number known to ACS facilitates the validation at the ACS itself. Note that this limit might be different from one device type to another type. Some generic software application tune this number based on amount of memory available on the device it is being installed. Some times, this limit is also configurable by the local administrator.

TR-104 mandates the definition of "Number of Instances" parameter for each table object. In addition, mandating one more parameter "Max Instances Supported" for each table object would sole above problem.

Nested Table Objects and Transactions

TR-069 protocol dictates that all parameters within the "Set Parameter Values" method either set in entirety or reject every thing. For all practical purposes, a configuration transaction is limited to one instance of the RPC method. Device implementations ensures this by checking with each application using "Test" functions of applications first for all configuration parameters and then followed by setting the configuration in the applications if all "test" functions are successful - basically tests followed by sets as in SNMP world. Applications' "test" functions also has its own limitations, especially if nested table object instance parameters are being checked if parent table object instance is not yet available in the applications, even though it is available in the same transaction. Since "Set" functions are expected to be called only after all "test" functions are successful, parent table instance would not have been created in the applications. Hence "test" function of nested table object instance would return FAILURE. To avoid complicated "test" functions, I feel that TR-069/TR-104 should put guideline for ACS developers to avoid setting parameters of nested table object instances along with parent table object instances in the same RPC method if it is followed by appropraite "AddObject" methods.

Nested Table Objects and Special cases

Even though it is true for all cases that the nested table instance can't exist without parent table object instance, in some cases it is required that instances of some table objects are created with atleast one instance of child table object. This is true specifically in security rules such as firewall ACL rules and IPSec SPD rules. ACL and IPSec SPD can be represented as table objects in the data model. These rules contain 5-tuple selectors with Source IP, destination IP represented by multiple sets of network objects. That is, source IP (and destination IP) field of the ACL rule is a table object to represent many networks. Administrator would be allowed to add more networks to the "source IP" and "destination IP" tables at later time once ACL rule is created. But when ACL rule is created, security application expect at least one network in "source IP" and "destination IP" fields of ACL rule object.

Today data model does not allow this kind of relationship. Due to this, ACS developers don't know about this dependency. If not taken care of by ACS then the rule creation will be rejected by device continuously.

It is necessary that data model definition allows specifying this special relationship. One simple proposal is to indicate for each table object on whether at least one instance of this is needed to create instance of immediate parent object.

Ordered Lists

Rules in many security applications such as firewall, IPsec VPN, Web application firewall etc.. are ordered. That is, the rules are searched from top rule to bottom rule for matching in the table object and hence the rules must be ordered in the list with top rule being the highest priority rule and bottom rule being the lowest priority rule. Administrators, normally in the course of administration need to add rules in between the list, top of the list, bottom of the list or change the priority of the rules in the list. TR-069 does not provide any RPC methods to change the priority of the rules. Due to this, the data models defining ordered list are expected to have 'priority' parameter. Administartor changes value of this parameter to change the priority of the rules. Devices are expected to form the ordered list based on the value of priority parameter acorss multiple instances in the table object. Though it works fine in cases where the configuration changed rarely, it poses performance penatlies on the device. Let us get into this details.

• Security applications revalidates run time states every time there is a change in the rule list. Revalidation can be costly in systems where millions of states are created. For example, firewall application in Enterprise/Carrier market segments support more than million sessions where each session corresponds to TCP/UDP/IP connection. If the rules are modified or their order is changed, firewall application is expected to revalidate the sessions and remove sessions that are no longer valid with respect to new rule base.

If a rule needs to be added in between, then the priority value of rules below the current rule need to be changed by at least 1 value less than the current values to accomodate the rule. That is, if there are 500 rules and if one rule is added at position 250, then bottom 250 rules would undergo change. This will raise 250 more additional parameters being modified. This results to 250 changes in the rule base in device. So, there would be 250 revalidations of Millions of sessions. To avoid this performance penatly, it is necessary that each ordere list (table object) has one parameter outside the table object "Revalidate now". ACS sets this value at the end of all priority parameter values in SetParameterValues method. When this parameter is set to 1, device is expected to initiate revalidate process. It avoids revaldation logic in the device for each rule change. This parameter value is not expected to be persistent. Its purpose is to initiate revalidation action. Reading of this parameter should always give value 0 even after it is set.

ACS also needs to know the objects that are ordered in nature. ACS also needs to know the parameter name used to order the list. If ACS does not know this information, then the administartor is forced to change the priority value of all 250 rules in above example manually. That is not some thing which administrators (users) would enjoy. ACS, by knowing this informaton, can change the priority values itself based simple user actions internally and communicate with device appropriately. ACS at the end of any action on the ordered list should send set the "Revalidate Now" parameter. To faciliate this, data model defintion should identify the ordered lists. My proposal is to introduce an attribute "Ordered list" to the table objects. This attribute will not be present for non ordered table objects.

Above guidelines on data model work fine for existing TR-069 protocol. Future TR-069 enhancements needs to think of differnet approach for ACS scaling by introducing new commands in "SetParameterValues" RPC method. SetParameterValues RPC method as explained above also need to take 'Add' command in addition to implicit 'Modify' command supported today. My proposal is to introduce "Move" command for changing the order of instances in the table. "Move" command takes the identifcation value of target row and command attributes such as "first", "last", "before", "after". If the command attribute value is one of "before" or "after", then it also takes the relative row identificatin value. "First" value indicates to put the target row in the beginning of the list. "Last" value put the target row at the end of the list. "Before" keeps the target row immediately before the relative row and "After" keeps the row immediately after the relative row. This proposal eliminates the need for having "priority" parameter. It also reduces the need for sending multiple parameters when the row is added in between. Note that "Revalidate Now" parameter need to be still present if more than one row is being added or more than one "moves" are initiated by administrator.

Log Export

One of main features of network security devices is logging. Detection/Protection of internal resources as important as notifying administrators of any violations. When central management is used to configure network security devices, it is also expected that central management console provides facilities to show alerts, notifications, analyze logs and generate reports. TR-069 protocol does not specify any protocol elements to send the log messages to ACS.

You may need to device your own protocol (such as syslog over IPsec) for sending logs. Also, you need to work with ACS vendors to corelate the logs with configuration.

Firewall session inactivity configuration - TR-069 Support

Firewalls maintain session entries for each 5 tuple connection. Session entries are created upon first packet of the connection. TCP Sessions are removed whenever TCP RST packet is observed or ACKs for TCP FINs are observed in both directions. Sessions are also removed when there is no traffic for some period of time. This period of time is called inactivity timeout period. Non-TCP sessions such as UDP, ICMP sessions are removed only due to inactivity as they don't have any connection boundaries.

There are multiple application protocols (services) running on TCP and UDP transports such as FTP, Telnet, SSH, HTTP, LDAP, RADIUS, SMTP, POP3, IMAP etc. Some application protocols are interactive applications and many are non-interactive applications. Telnet, SSH, FTP are interactive applications. HTTP, HTTPS and many others are non-interactive. In non-interactive applications, once the connection is made to the server, there is no user input in between until the connection is terminated. Entire user input is taken before making the connection. In interactive applications, user input is taken after connection is established. Inactivity timeout period for non-interactive protocols can be in terms of 10s of seconds. Since interactive applications wait for user input, less inactivity timeout value may remove session if user does not feed any input data for a longer duration. So, interactive application protocols require longer inactivity timeout. If longer inactivity timeout is configured for non-interactive application protocols, there is a danger of keeping stale sessions for a longer time and that may result in session entries exhaustion problem. So, there is need for providing different inactivity timeout values for different protocols.

Please refer document on maximizing firewall availability. One of the techniques suggested there was to provide 'INIT Flow Timer optimization'. This approach suggests to have separate inactivity timeout period during connection establishment phase (3 way TCP handshake) . This inactivity timeout value can be way less than the inactivity timeout needed after connection is established.

Keeping both of above points in mind, session inactivity configuration includes following:

• TCP Pre Connection inactivity timeout : Inactivity timeout during connection establishment phase.
• UDP Pre Connection inactivity timeout: UDP does not have any connection establishment phase. For this discussion, UDP connection establishment phase considered complete when it receives at least one packet in both directions of the connection (client to server and server to client).
• TCP Inactivity timeout: Inactivity timeout value after TCP connection is established.
• UDP Inactivity timeout: Inactivity tiemout value after UDP session is established.
• Generic IP inactivity timeout: Inactivity timeout value for non-TCP and non-UDP sessions.
• TCP FIN timeout: Inactivity timeout after TCP FINs are observed in both directions.
• Application protocol specific timeout records. Each record containing
• Application protocol information : Protocol and Port
• Inactivity timeout value: This inactivity timeout value is used after connection is established. If there is no matching application protocol specific inactivity timeout record, then TCP, UDP or generic IP inactivity timeout value is used.

TR-069 based configuration:

• internetGatewayDevice.security.VirtualInstance.{i}.firewall.serviceInactivityTimeout P
• tcpPreConnTimeOut: RW, Unsigned Int, Default : 10 seconds - Value in seconds.
• udpPreConnTimeOut: RW, Unsigned int, Default: 10 seconds - Value in seconds.
• tcpTimeOut: RW, Unsigned Int, Default: 60 seconds - Value in seconds.
• tcpFinTimeOut: RW, unsigned int, default: 10 seconds - Value in seconds.
  
• udpTimeOut: RW, Unsigned Int, Default: 60 seconds - Value in seconds.
• IPTimeOut: RW, Unsigned Int, Default: 60 seconds - Value in seconds.
• internetGatewayDevice.security.VirtualInstance.{i}.firewall.serviceInactivityTimeout.applicationTimeout.{i} PC
• name: String(32), RW, Name of the record. Once the record is created, this can't be changed.
• Description: String(128), RW, Optional - Description of the record.
• protocol: String(8), RW, Mandatory - Takes values "tcp", "udp"
• port: String(8), RW, Mandatory - Takes port value
  
• inactivityTImeout: Unsigned Int, RW, Mandatory - Inactivity timeout in seconds.

Network security computing with Session parallelization - Development tips

Until 2005, performance of network security was a function of processor power. Processors were just keep getting better and faster and thereby the network security performance. In recent past, processors are not getting better, but more processors (cores) are being added to the chip. I guess physics limited the processor technology getting faster. Manufacturers are concentrating on increasing gate count in the chips than increasing the clock rate. Chip vendors are adding more processing cores in one die. These chips are called 'Multi Core Processors' (MCP).

Before any application takes advantage of MCPs, operating systems should support Multiple cores. Linux OS and other operating systems have been supporting MCPs for last few years. Linux OS calls this feature as 'SMP' (Symmetric Multi Processing). In this mode, all cores share the same memory i.e code and data segments are same across all cores. Since any core can access any part of the code and data, it is necessary to protect the critical data by serializing the access to the code segment accessing critical data.

Applications are expected to take advantage of multiple cores in the system. Now the mantra for applications is parallelization. There is no telling that parallelization of software is not easy. It is time consuming and takes significant investment.

One of the serialization technique is to stop cores from processing the critical code while other processor is executing that code. This is typically done in Linux Kernel space using spin locks. Too much of serialization reduces the performance and the application performance does not scale well with the number of cores. At the same time, critical data and data structures (such as binary trees, linked lists etc..) must be protected by serialization.

Other Difficulties in parallelization of code are:

• Parallelization is subject to many errors such deadlocks, races conditions and more importantly it is difficult to debug.
• Reproducibility of problems are difficult. Hence problem identification takes longer development cycles.
  
• Maintainability of the code: First time the code might have been written keeping all parallelization problems in mind. As time progresses, different developers work on the code and maintain the code. They may not be aware of parallelization problems and make mistakes.
• Single CPU based test coverage is not sufficient. Updated test suite should be able to find as many race conditions as possible.

Any parallelization approach should make it simple to develop, maintain code and yet efficient. 'Session parallelization' approach is one method that makes this task simple for networking applications running in the kernel space.

Network security primarily consists of firewall, IPsec VPN, Intrusion Prevention, URL filtering and Anti Virus/Spam/Spyware functions. To improve the performance of firewall, IPsec VPN and in some cases IPS functions, they are typically run in Linux kernel space. Each security function has notion of session. In case of firewall and IPS, sessions are nothing but 5 tuple flows and in case of IPsec VPN, session is 'SA Bundle'. Network security functions maintain state information within the sessions. State information, some times, changes on per packet basis and new packet processing depends on the current state. There are two ways to ensure the packet synchronization is ensured with respect to states. One way is serialize the code path that updates and checks the state information. If there are multiple instances where states are checked and modified in the packet path, then there are multiple serializations. Depending on number of serializations, performance impact is smaller or higher.

Another way is 'Session parallelization'. In session Parallelization, packet synchronization happens at the session level. At any time, only one core owns the session. If any other core receives the packet, then it is queued in the session for later processing by the core owning the session. If no core owns the session, then the core that received the packet starts processing the packet after stamping its ownership. Once the core processes the packet, it checks whether there are any packets pending to be processed. If so, it processes them and if not it disowns the session. Using this method, upon session identification, rest of packet processing does not need to take any locks for serialization. It not only improves the performance, but also less error prone. Having said that, locking of code can't be avoided in some cases such as session establishment, inter session state maintenance.

During session establishment, access to the configuration information is required. Configuration update can happen in one core context, while session establishment happens in some other core context. Data integrity must be maintained to ensure that core does not get the wrong configuration information. Also, data structure integrity must be maintained. Think of cases where configuration engine is removing a configuration record from a linked list and session establishment code is traversing the linked list. If care is not taken, then session establishment code might access invalid pointer resulting null pointer exception. Since, session establishment phase required read only access to the configuration structures, locks can be taken in read only mode. In read only mode, multiple cores can do session establishment for different sessions and thereby it does not effect the session establishment performance. Since configuration updates are less common, read only locks usage does not degrade the performance.

Another area where locks to be taken during packet processing is in situations where inter session state is maintained such as 'statistics counter update', session rate control state etc.. Since Many cores work on the different sessions simultaneously, it is necessary that inter session state information is protected using write locks around the code that manipulate the state information. It is very important that amount of code that is executed under locks is as small as possible as possible for better performance.

Though there are cases where locks are used such as inter session state and configuration data access, these instances are less in number. Session state updates and accesses are more common. As long as session parallelization is done, the SMP problem is manageable. Ofcourse, one could argue that a given session performance is limited by power of one core. But, in typical environment, there are many sessions and hence the system throughput will be proportional to number of cores.

Cloud Computing and security

Cloud computing has become popular term in recent past. Cloud computing providers have large number of cloud servers interconnected. They provide services to end users - Renting virtual server with CPU power required, Storage and some specialized services such as PHP, Java, Ruby-on-rail based servers etc..

Since these servers are outside of offices, it is required that you have very good internet connectivity. Cheap bandwidth and reliable connectivity favors the cloud computing model. From cloud computing provider perspective, this is becoming possible with very high speed, high density multi core processors and virtualisation with its inherent facility to provide isolation and running multiple services on a physical hardware.

Advantages of cloud computing for users (Enterprises) are same advantages you get with data centers such as

• Reduce system and network infrastructure administration burden.
• Save on Electricity cost by selecting data center with lower cost of electricity.
• Save on real estate.

Cloud computing provides additional advantages such as

• Handle peak loads by provisioning computing power with a click of a button.
• Isolation of application servers from physical machines.

There are some concerns which are not yet fully matured.

• Who is going to take care of security aspects of user applications? Is this cloud computing provider or is it the responsibility of users?
• Who monitors the vulnerabilities of different applications and takes care of patchoing them?
  
• Will there by any visibility provided of exploits and attacks to the user?
• Who takes responsibility of provisioning security infrastructure? Who takes responsibility of tuning IPS/IDS signatures?
• Who takes responsibility of complaint requirements such as PCIDSS etc..?
  
• Who takes responsibility of auditing systems, application etc..?
• If you have remote users that need access to these services, what kind of security on the wire required and who provides VPN Connectivity?
  

When cloud computing provider provides specialized services such as Email etc.., I feel that it is responsibility of cloud computing provider to check for vulnerabilities, hardening, patching, checking for spams and preventing from phishing attacks etc.. Do they do that today? What kind of guarantees provided?

When cloud computing providers provide generic services such as renting out virtual server, I have a feeling that responsibility of security them may fall on user s's shoulders. Now the questions arise such as:

- Do Cloud computing SPs provide *Cloud Security* service?
- Do SPs give flexibility for users to select their own security vendor?
- Do SPs expect security appliance is provisioned as Virtual service? If so, what kind of virtualization technology SPs provide?
- Do SPs provide network visibility for user to link the security service with application servers.


It is not possible for cloud computing providers to provide security for applications which they don't know. Many security problems are specific to each application. Typically Enterprises have their own applications in addition to standard applications. As you see in the questions, there is lot of tuning on security applications, such as adding new signatures in IPS, that happen over time. So, it makes sense for cloud computing providers to provide flexibility for users to create their own security environment. Enterprises also typically provide remote security connectivity for their employees to access critical services. Securing the Enterprise services not only involve exploit detection, tuning, hardening and patching, but also providing VPN service to employees.

I have a feeling that, Like the way computing services are provided in the cloud, security services will also be provided by cloud computing providers. Cloud Security Service provisioning not only involves security application, but also connectivity between security service and application servers. Even to provide complete security, it may involve multiple security services provisioning such as VPN Service, IPS Service, Firewall Service, Web application firewall service or it could be one UTM service.

If Service providers are going to provide flexibility for end users to provision their choice of security application, then SPs would provide choice of running Virtual security appliances.

Yet times, Service provider may not like to provide flexibility of security application and they may provide security as specialized service from them. In those case, SPs may go for mega security appliances supporting multiple instances with instance provisioned for one customer.

Let us see how this market turn out to be.

But, in both cases, need of computation power for security services is very high. Multi core processors are going to fill this gap.

Central Management Systems - Critical Missing features

Central Management systems /Network Management Systems are used to configure and monitor multiple network elements from a central location. Some of the features that are supported by many CMS solutions are:

• Ability to configure multiple network elements.
• Ability to collect log information.
• Ability to analyze log and generation of periodic reports.
• Ability to monitor critical events.
• Ability to issue diagnostic commands to network elements and getting the results.
• Ability to allow multiple administrators to use CMS solutions - Role based Management.
  
• Ability to view the Audits - Who changed what and when.
  

CMS solutions architecturally contain

• UI console: Allows users to configure network elements and also allows users to view the reports/analyze logs.
  
• Policy Server with Policy repository: Stores the configuration for each network element.
  
• Element Adapter: Convert configuration information to device understandable format. Sends the configuration information via protocol supported by devices.
• Log Collector and Report Generator: Collects logs from network elements and also can generate reports.
  

To provide scalability i.e to support large number of network elements, multiple element adapters and log collectors are used which each of them supporting fixed number of network elements. Typically one policy server is used, but many UI consoles are used at any time.

There are some pieces which CMS vendors tend to ignore, but in my view they are very important. One of them is to do with 'Configuration session'. Traditionally, each administrative change in the configuration results to a command for the device. That is, if administrator changes the configuration X number of times, then X different commands are prepared for the device. It is observed, if the administrator changes a rule (let us say firewall rule) and un-modifies the the changes, then two commands are generated. These commands are eventually sent in order of the changes to the device. Many times, this is not a problem. But this could be a problem in some instances where:

• First modification when applied changes the state of the device such as removing some run time states etc.. Second command which unmodifies the previous changes does not bring back the run time states that were destroyed.
• Some times, the first modification might even stop the traffic in network elements.

Administrators should be given chance to make configuration changes (additions, modifications or deletions), review them and commit them. Only when it is committed, it should send the consolidated changes to the device. Configuration modifications and commands must be de-linked. When the commit is issued, it should generate commands on the configuration differences. Some examples would help in understanding the feature better.

• Administrator added a rule and changes his mind and deleted the rule before committing. In this case, no command should be generated for this rule.
  
• Administrator deleted a rule and changed his mind and revoked the configuration change. In this case, no command should be generated for this rule.
  
• Administrator changed few parameters of a rule and again changed to newer values. Only new values should go to the device.

This feature requires following support from CMS solutions:

• Configuration session should have start and End. End can be complete revoke or commit.
  
• At any time, administrator can check for errors during configuration sessions. CMS solution is expected to provide 'Validate'.
• Checking for duplicate configuration session. At any time, only one configuration session is allowed. Note that multiple user can view the committed configuration. If new configuration session is started, then CMS solution should return warning to the user that configuration session from 'user' at 'date&time' from 'ipaddress' is already started. It can give options such as 'take over the existing configuration session' or 'start from scratch by revoking previous session'.
  
• At the time of 'commit', CMS solution can take new version string, which is prepended by date&time.
• At the time of 'commit', CMS solution should generate commands to be sent to the device. It should do this by reading the information from configuration session, but not by the sequence of actions user did.
• New configuration session should be allowed even if commands generated out of previous configuration session were not synchronized with the device.
• Clearly show in the UI of the configuration which is part of current configuration session.
  

In addition, CMS solution should support:

• Listing down the configuration versions of a particular network element.
• Facility to migrate the network elements to previous versions.
• Facility to remove very old configuration versions to preserve the space in database.

Botnets using fast flux and double flux techniques - IPS devices

Access to the botnet servers, malware servers and servers serving other objectionable content from corporate networks is being thwarted by IP black lists by IPS/IDS and UTM devices. Cyber criminals started using single flux and double flux DNS techniques to make this kind of blacklisting ineffective. These techniques change the IP address of malicious servers very frequently. In some cases IP addresses are changed every 5 minutes. Please check this link to get more information about fast flux and double flux techniques.

Criminals take advantage of compromised servers to act as redirection servers. Domain name of criminal servers get resolved to these compromised servers. When innocent users connect to this domain name (through social engineering attacks), the HTTP request lands on redirection servers. Redirection servers get the content from original malicious server (Honeynet white paper calls it Mother ship server) and serve to the innocent users. The list of compromised servers to be given in a DNS instance is determined by many factors such as whether the compromised redirector is online, bandwidth of the link that internet link of the compromised redirector etc.. Since the cyber criminals run the DNS server along with malicious content server, they have control over which IP addresses to send as part of DNS response. This technique is called fast flux as the IP addresses of the domain name registered by criminal changes very often.

From the attack description provided in honeynet link, Cyber criminals rent botnet for redirection servers. Botnets owners would have compromised unhardened victim machines for their nefarious activities. It appears that some botnets have thousand of compromised systems. Many of home users PCs are typically infected with botnets.

Since the IP addresses of domain name keeps changing, traditional blacklisting technique that uses IP addresses is ineffective. Also, it becomes difficult to identify the mothership servers. To thwart this kind of attack, security developers also started creating blacklists for DNS Servers. This thwarting technique also depends on IP addresses.

Now attackers started using double flux technique, where DNS Server IP addresses also change very frequently. This requires change of Name Server IP addresses in DNS registrars or resellers. As some registrars making this facility available through programming interfaces (web based interface), this is being automated by cyber criminals (I need to verify this statement). Some service providers are lax and don't follow guidelines of checking credentials while registering the domain names or while changing the name server IP addresses.

Since two kinds of IP addresses are changed - Name server IP address change, DNS resolution IP addresses, this technique is called double flux. This technique can't be solved by IP address blacklisting.

Mitigation:

IP address based blacklisting is some what effective when used with single flux. Double flux technique makes that ineffective. It appears now that domain name is fixed in both kinds of techniques. Mitigation is possible if domain names are checked.

DNS domain name blacklists are required to thwart this attack. www.malwaredomains.com
provides the list of domain names hosting malware content. IDS/IPS devices should have intelligent DNS application engine to extract domain names from DNS query and check against this list.


Some characteristics of DNS replies when these techniques were used. They are:

- TTL is around 5 minutes to 30 minutes.
- Multiple IP addresses in Answer section.

Since this kind of DNS replies are possible in some normal cases, this information can't be used to stop the DNS traffic by IPS devices. But it provides valuable information for offline analysis and to track the ultimate malware server with the help of service providers.

IPS/IDS Buyer - Deployment recommendations

CSO and security professionals goal is to ensure network security of their business networks and resources without any discontinuity in business operations. IPS/IDS devices solve one of network security problems - that is Intrusion detection and protection. Selection of IPS/IDS device is complex. I am trying to address considerations with respect to deployment while selecting IPS/IDS devices for your network.

Network Deployment modes:

Tap mode (IDS mode) : In this mode, IPS device is used mainly to detect intrusions/attacks. It does not block the attack traffic. IPS/IDS device is typically connected to hub or SPAN port in managed switches so that this device gets entire traffic in that network. Since it is not passing the traffic, network performance is not impacted.

Though tap mode is least intrusive as it does not come in the way of normal traffic, it may not detect all attacks, if traffic is not received by it, either due to congestion at the SPAN port or due to the processing ability of device. Many recent IDS/IPS devices are stateful in nature. If packets are lost or not processed, newer packets or out-of-state packets either don't get processed or IDS device may generate false positives. Inline IDS deployment mode takes care of some of these problems.

Inline IDS mode: In this mode, all packets pass through the device. In this mode, IDS/IPS device does not drop packets or terminate sessions upon attack detection. IDS/IPS devices are installed behind Enterprise core routers or perimeter routers. Since it is inline of traffic, it observes entire traffic before sending it out. Unless traffic is analyzed, it is not sent out.

Since it analyzes entire traffic that is passing through the device, the detection rate of attacks is only limited by IPS/IDS device functional capability. If the traffic is faster than it processes, then the excessive traffic gets dropped.

Though detection rate is going to be high, there is some impact on traffic -

• Traffic may get dropped due to processing capability of IDS/IPS device.
• Packet latency increases.
• Packet jitter also may be impacted.

Inline IPS mode: This is similar to Inline IDS mode, except that it can be configured selectively to stop the attack traffic. This mode inherits all advantages and disadvantages of Inline IDS mode.

As long as only attack traffic gets dropped, then it is perfectly fine for security professionals. IPS/IDS technology, though came long way, still continue to have problems such as false positives and false negatives. Significant technology of IPS/IDS devices depend on signatures or rules. Signatures are of two types. Many signatures are developed by IPS vendors to stop known attacks. Another type of signatures detect protocol, data and traffic violations. With sophistication of attacks, yet times, signatures created to detect attacks may result into false positives. One of the difficult problems in IDS/IPS world is to detect client side attacks without any false positives. Look for IPS/IDS functional capabilities that detect attacks with less or zero false positives.

Based on your requirements , you should find out the different deployment modes you require and look for devices supporting the required modes.

If you decide on inline mode, you should look for following capabilities when deploying IDS/IPS device.

• Granularity of blocking action: Look for this selection on protocol category basis and also on per rule (signature) basis.
  
• Ensure that Inline IPS/IDS mode work transparently without any changes to the network addressing of existing network.
  
• Traffic continuity when session resources get exhausted: Many IPS/IDS devices are stateful devices. They maintain sessions entries for connections. These session entries are removed only upon inactivity or due to TCP RSTs/FINs. When there is session exhaust DDOS attack targeting IPS/IDS device it could stop legitimate traffic, there by disrupting business operations.
  
• Look for traffic throttling functionality so that IPS/IDS devices don't exhaust its resources.
• Look for session timeout configuration functionality so that you can configure different session timeouts for different applications.
• Look for session timeout configuration during session establishment (Pre-Connection timeout)
• Also look for control on behavior of new traffic when sessions indeed get exhausted. Fail Open and fail close upon resource exhaust are two options you should check for. Fail open option lets the new traffic pass through without inspection. Fail close selection drops the packets.
• Mode Change capability: Look for provision to change mode from Inline IDS to Inline IPS. Each network is different. They have different types of traffic, servers, desktops and mobiles. Security professionals first need to get confidence of effectiveness of IPS/IDS device in their network. Though their ultimate use of IPS device is to stop the attacks immediately, to get confidence and understand its usage, as a security professional, you may like to deploy it in inline IDS mode first and then change the mode in Inline IPS.
• Control on CPU utilization: In Inline modes, traffic gets dropped due to non-availability of CPU. Look for controls that limit the CPU utilization. In particular, check for following capabilities.
• Signature selection capability: More CPU power is used to when the rules are higher in number. Look for facilities to disable specific signatures by deselecting the family of signatures and individual signatures.
  
• Control on quantum of data to inspect: Some detections are very expensive - such as malware detection. Since these are not protocol related vulnerabilities, this detection requires data inspection. Typically, these signatures have very complex patterns and hence it takes significant number of CPU cycles. It appears that many attacks can be detected within 16K bytes of connections. This observation can be used to limit the traffic inspection, thereby saving CPU cycles. Look for capability in IPS/IDS devices where administrator can control the amount of data to be inspected on per protocol basis and also across protocols. One word of advice is to start with inspection of all data, analyze and tune/configure this configuration item based on type of applications and traffic in your network.
• Latency, Jitter and throughput: Figure out the type of applications for your business in your network and note down your requirements of throughput, latency and jitter tolerance and ensure that IPS/IDS deployment does not disturb these parameters beyond the limits you set.
  
• Behavior of IDS/IPS device on unrecognized traffic: IDS/IPS devices may not have capability to inspect all kinds of traffic. Many IDS/IPS device limit themselves to inspect IP traffic. If your network has traffic such as multicast, IPv6, IPX, Apple Talk or proprietary protocol traffic, then check the behavior of IPS/IDS device. At the minimum, you should expect these devices to pass this traffic, even though they don't inspect the traffic.
• Network monitoring: Many Enterprises use common SNMP based monitoring tools. IDS/IPS device becomes one of the network elements. If this is important for you, then ensure that IDS/IPS device you select support SNMPv3 agent supporting MIB-II.
  

High Availability:
Availability of IPS/IDS device functionality is very important in inline modes. Your IDS/IPS device can become a failure point in your network. Hence ensure that IDS/IPS device can support high availability functionality.

LAN bypass functionality: This functionality short circuits all Ethernet ports, basically making it as Ethernet hub' when there is any failure in the software/hardware of IPS/IDS device. This functionality is typically implemented in hardware. In recent times, this also can be implemented using virtualization. Please see this link: http://network-virtualization.blogspot.com/2008/03/lan-bypass-using-xenvmware-kind-of.html. Look for this function if this is good enough for your network.

Redundant devices: LAN bypass function ensures that connectivity is not lost, but it bypasses security inspection. That may not be a choice for some business environments. In those cases, look for IDS/IPS devices supporting redundancy. Two or more devices (typically two are good enough) can be installed in parallel. When one device goes down, another device starts processing packets. Optionally, some IDS/IPS devices even support takeover of existing sessions. With that capability, existing connections will continue to work even after other devices takes over. In addition, Some of the items, administrator should look for are:

• Amount of time it takes a new device to become active. It should be in terms of seconds in single digit.
  
• Amount of it takes for Central Management System to understand the switch.
• Ensure that Central Management System populates all backup devices with signatures even before it becomes active.

Disaster Recovery:

High availability will not help upon major disaster. It may require procuring new devices. Security professionals would have spent significant effort over time to tune the IPS/IDS devices. If this work is lost, then it takes significant time to re-tune the devices. That is where disaster recovery functionality provided by IPS/IDS devices is very important.

Security professionals should look for facility in IDS/IPS devices to store the configuration and restore it whenever it is required.

Central Administration:

Large Enterprises require more than one IDS/IPS sensor devices. They are placed at different places in the network. Central Management reduces the configuration burden. It also provides corelation of events and logs. If your network requires many sensors, look for Central Management system. Typical features one should look for:

• Multiple administrator accounts.
• Role based management.
• Multiple UI consoles.
• Corelation of logs and events.
• Traffic Reports
• Attack Reports.
• Alerts.
• Audit Reports.
  

There are many choices of IDS/IPS devices in the market. Selection of device depends not only on its functionality, accuracy of detection but also how easy for you to deploy and monitor. I try to address some common items to be considered in your buying decision.

ALGs - Firewall/NAT Travrersal Control and PortMap and TR-069 support

Firewall/NAT Traversal Control:
Application Layer Gateway modules (ALGs) in firewall and NAT devices interpret the protocol data, transform IP addresses based on NAT configuration and open pinholes in firewall to allow new connections. For example, FTP ALG function is expected to interpret 'PORT', 'EPRT' and 'PASV reply' messages, modify IP addresses if required and open the pin holes to allow FTP data connections. Many protocols require this kind of ALG functions for firewall and NAT traversal. Some of the protocols requiring ALGs : SIP, H.323, MGCP, some gaming applications, NetBIOS, SUNRPC, MSRPC, L2TP, PPTP, IPsec VPN etc..

Newer versions of protocols are designed such a way that they traverse through firewall/NAT devices even if they don't support ALGs. For example, SIP has some extensions where there is no need for ALG function in firewalls between SIP UA and SIP proxy. IPsec VPN working group added NAT-T extensions to IKE and IPsec and it does not require any ALG function in firewall and NAT devices between IPsec peers. But, they introduced newer problems. Some of these NAT-T extensions in newer versions of protocols don't work well with firewall/NAT devices which support ALG function already. Hence, it is required that firewall/NAT devices provide ability for administrators to control the ALG function for different protocols. One simple control that is expected at the minimum is boolean control ie. Enable/Disable. Ideal control of configuration would take end point IP addresses into consideration. Imagine cases where some end points support new NAT-T extensions and some not. But, for this discussion, I am taking simpler configuration i.e ALG function enable/disable for each protocol.


ALG port map:

Yet times, companies install server application on non-standard ports. Though 5060 is standardized for SIP, yet times, SIP servers are run on non-standard ports. In these cases, the ALG functions in the firewall that is protecting these SIP servers should know about these ports for its operation. Port map record functionality of firewall/NAT devices facilitates the administrators to feed this information. For example, if SIP server is run on port 5061, administrator can create a port map record with Port 5061 and map it to SIP ALG function.

Both of above functionalities and their configuration require definition of ALG names. I propose following names for ALGs.

"ftp", "tftp", "oracleDbNet", "sunRpc", "msRpc", "udpDns", "tcpDns", "netbios", "udpSip", "tcpSip", "h323", h323GateKeeper", "rtsp", "udpNet2Phone", "tcpNet2Phone", "mgcpCallAgent", "mgcpGW", "msnIM", " microsoftILS", "aolIM", "irc", "pptp", "l2tp", "ikev1", "mszone", "quake", "udpMicrosftGames", "tcpMicrosoftGames'.

TR-069 representation of above configuration:

• internetGatewayDevice.security.VirtualInstance.{i}.ALGTraversalControl.{i} P : New entries can't be added by ACS. ACS can only change the 'featureControl' parameter.
  
• name : String(32), Read Only - Name of the ALG. It takes one of values mentioned above.
• featureControl: Boolean, RW - Take 1 (Enable) or 0 (Disable). Default value is 1.
• internetGatewayDevice.security.VirtualInstance.{i}.ALGPortMap.{i} PC
• name: String(32), RW - Name of the port map record. Once the record is created, it can't be changed.
  
• description: String(128), RW - Description of the record. Optional parameter.
• algName: String(32) RW, Mandatory parameter- Name of the ALG function. It must be one of the values mentioned above.
• mappingProtocol: String(4), RW, Mandatory parameter - Protocol value. Either "tcp" or "udp".
• mappingPort: String(8), RW, Mandatory parameter - Port number.
  

Penetration testing directory

I saw this email in pen-test mailing list. www.penetrationtesting.com site is trying to be a directory for all pen testers. It seems to be promising. One consolidated place to see all topics, tools related to penetration testing.

From: listbounce@securityfocus.com [mailto:listbounce@securityfocus.com]
On Behalf Of Victor DaViking
Sent: Saturday, April 05, 2008 2:43 PM
To: pen-test@securityfocus.com
Subject: Update on the penetration testing directory project

Hi list,

Quick update. I wanted to thank everyone who's been
helping out with the pentest directory project
(www.penetrationtests.com) during these 5-6 months in
terms of submitting new links, providing feedback,
suggesting categories, etc.

As of today, we already have 230 different
pentest-related links manually organized in the
following categories:

- http://www.penetrationtests.com/Blogs/
Blogs related to security (Company blogs, Personal
blogs, Group blogs)


- http://www.penetrationtests.com/Business/
Business links (Scoping, SoWs, Invoicing,
Deliverables)

- http://www.penetrationtests.com/Companies/
Security Companies (1-99 employees, 100-500, 501-more,
unknown size)

- http://www.penetrationtests.com/Documents/
Security Documents/papers (Database servers, webapp
testing, secure programming)

- http://www.penetrationtests.com/Frameworks/
Penetration testing Frameworks (for a price, free)

- http://www.penetrationtests.com/Mailing-Lists/
Mailing lists (Penetration testing)

- http://www.penetrationtests.com/Methodology/
Methodology (Links related to methodology definitions)

-
http://www.penetrationtests.com/Security-conferences/
Security conferences

- http://www.penetrationtests.com/Security-standards/
Security standards (PCI, SOx Act)

- http://www.penetrationtests.com/Tools-Software/
Tools & Software (hundreds organized in sub
categories)

- http://www.penetrationtests.com/Websites/
Websites (Project sites, Proxys)

IPsec VPN load balancing - technical bit

Many Enterprises are having multiple WAN links for sharing the WAN load and also to provide redundancy. Enterprises are increasingly going for WAN links from different providers for continuous connectivity even when there are failures at service provider end. There are many solutions in the market which take advantage of multiple links - by bonding these WAN links. Since these WAN links belong to different service providers, load sharing is done typically at the connection level. That is, all packets of a 5 tuple connections go through the same WAN link, but packet belonging to different connections go via different links.

IPsec VPN tunnel between two offices is considered as one connection by these solutions. Due to this, even if both offices have multiple links, the bandwidth for IPsec traffic is limited to maximum bandwidth provided by selected link. Actually, it would be the minimum of office 1 link and office 2 link bandwidth. In many situations, the traffic among offices (branch offices to head office) is mainly IPsec traffic.

IPsec VPN load balancing functionality is expected to solve above problem and fully utilizes the maximum bandwidth provided by multiple links.

Let us examine different deployment scenarios:

• Sceneario 1: Branch office having one WAN link and Head office having multiple WAN links. And entire VPN traffic in each side serviced by one VPN router (or UTM box)
  
• Scenario 2: Both offices having multiple WAN links and all the VPN traffic in each side serviced by one VPN router (UTM box)
• Scenario 3: Branch office having multiple low bandwidth links and head office having one high bandwidth link.
  
• Scenario 3: Both offices having multiple WAN links and as many UTM VPN routers as number of links.

IPsec VPN functionality is typically used in tunnel mode. In tunnel mode, IP packets traversing between the sites are encapsulated. As part of encapsulation, newer IP header is added and it is called outer IP header. Outer IP header is mainly used for routing of packets across Internet to reach the right VPN router. The IP addresses of outer IP header, hence must be public IP addresses. These IP addresses are called 'gateway' IP addresses. Local gateway IP address is used as 'source IP' and remtoe gateway IP address is used as 'Destination IP' in the outer IP header. These two gateway IP addresses are called 'gateway pair'. WAN link load balancing function uses these IP addresses (mainly source IP address, as both WAN links have default routes) to route the packets on appropriate WAN link. Hence, it is required that VPN router create multiple tunnels with different local gateway IP addresses and balance the outbound traffic across these tunnels to use the bandwidth of local WAN links efficiently. It is expected that the remote router also does same thing on its outbound traffic.

In all scenarios, the IPsec VPN functionality of UTM box should take multiple gateway pairs configuration in appropriate SPD policy records. In theory, there could be M * N pairs with M being number of links in site 1 and N being number of links in site2 2. But, in reality, number of gateway pairs one would configure is some where between minimum of M and N to M*N. For example, In scenario3, to take advantage of multiple WAN links in branch office, both head office router and branch office router can be configured with two gateway pairs. On head office side, local gateway IP address is same across these two gateway pairs and in branch office side remote gateway IP address is same across gateway pairs.

How to load balance traffic among multiple tunnels between two sites:

Many VPN routers today have stateful security functions such as firewall, IPS etc.. UTM boxes when used in place of VPN routers have many stateful security functions. If entire VPN traffic is handled by one device, packet based load balancing works fine. But, if the WAN links are handled by two different VPN routers having stateful functionality, then packet based load balancing does not work. Even connection based load balancing can have problems for applications having multiple connections. Hence, I suggest not to use load balancing, but use manual load balancing by creating multiple SPD policy records between two sites, with SPD policy record having different gateway pairs and with different selectors.

In summary: Packet based load balancing across different tunnels of one SPD policy record is applicable if VPN routers don't have stateful security function or if the VPN router is terminating all tunnels.

Access Control list in firewalls : TR-069 support

Access control lists are heart of firewall. ACLs control the traffic among different zones of organizations. Typical firewall implements multiple ACLs with each ACL implementing multiple access control rules. Each rule is defined with L3 & L4 protocol information and in some cases with L7 protocol fields - L3 fields are typically source IP, destination IP and Protocol and L4 fields are typically TCP ports, UDP ports and ICMP type/code values. Some of examples of L7 protocols are HTTP, SMTP, NNTP etc..

Access rule typically contains 'Selectors' and 'Actions'. First packet of every session is matched against the rules. 'Selector' fields are checked as part of matching operation. 'Actions' of matching rule is applied. If there is no match, then the packet is dropped or rejected based on whether stealth mode is enabled or disabled. Note: If stealth mode is enabled, then the packet is dropped. It generates TCP resets to both end points in case stealth mode is disabled. Since the matching operation is terminated upon first match, the access control list is organized as ordered list with first entry being higher priority rule and last entry being lowest priority rule.

'Selectors' fields are typically categorized into 'primary selectors' and 'secondary selectors'. Both kinds of selectors are checked against the packet values and other values in matching operation. If primary selectors are changed during the course of sessions, then the existing sessions are revalidated. In case of secondary selectors, this revalidation does not happen. Zone information (From zone and To Zone) and 5 tuple values in the rules are primary selectors. 'Time window' or 'time schedule' typically considered as a secondary selector. 'Time window' in rules is used to allow/deny connections during some period of week. For example, some connections may not be allowed during day time, but allowed in night time.

'Actions' in the rule decides the connection traversal through the firewall. 'Allow' action lets the connection through the firewall. 'Drop' action drops the packets and 'Reject' action sends the TCP reset (in case of TCP connection) to the client. Allow, Deny and Reject actions are mutually exclusive and these are primary actions. More sub actions are also can be defined.

• 'Log' is one sub-action. This indicates whether the connection is to be logged with logging system. If this option is selected, firewalls typically send 'Connection Start' and 'Connection End' messages to the logging system in case of allowed connections. In case of 'Drop/Reject' actions, log is sent to indicate that the connection was not allowed.
  
• Packet mangling: TOS (Type of Service) or DSCP parameter - This takes new TOS value. Packets are updated with this TOS value if it is configured. This is typically used to increase or reduce the priority of the packet for traffic management purposes. This feature is specifically provided where VOIP/Video devices behind the firewall don't differentiate between data and real time traffic. Another packet mangling parameter that is supported is setting of MSS value in TCP packets having 'SYN' flag. If this parameter is set, then all TCP connections falling on this rule would be changed to this MSS value (if this MSS value is less than the MSS value that is being negotiated in SYN packets). This setting is specifically useful when the WAN uplink bandwidth is is less than 256kbytes/sec. When this value is low, the TCP packets sent by both end points are small in size and hence the packet transmission time is less. Once the packet is submitted to the hardware to transmit, it can't be preempted and that is any new packet has to wait until this packet is sent out. Due to this, VOIP packets also need to wait and this might give rise to lateny and thereby jitter. To reduce the latency and jitter, it is necessary that the data packets which are queued to hardware are small enough. MSS value helps in ensuring that data packets are small.
  
• Rate control: Another sub-action that is supported in rules are controlling the rate in terms of packets, bytes, connections. In addition, even the connection limits also can be controlled by the administrator. If this is configured in the rule, if the traffic falling on this rule exceeds these parameter values, then packets or connections are dropped. If packet or byte rate exceeds across all sessions of this rule is exceeded, then the packets are dropped. If connection rate is exceeded, then the connection establishment does not succeed. Similarly, if number of existing connection due to this rule exceed the maximum connections allowed by this rule, then the connection establishment does not succeed.
• Application protocol command filtering: I am not really fan of keeping the application command filtering as part of each rule. With IPS becoming part of many security devices, this kind of filtering can be achieved through IPS rules/signatures.
• IPS Signature based Intrusion Detection function Control: Many firewall devices are adding Intrusion Prevention feature. As we all know, intrusion detection is CPU consuming function. To reduce the load on the CPU, I feel that firewall function needs to provide flexibility for administrators to disable Intrusion detection on per rule basis.
  
• IPS Traffic Anomaly detection & throttling function control: IPS function provide multiple detection methods. Traffic anomaly is one detection method supported. This function also takes significant CPU cycles and also takes memory to maintain traffic states - some times on per connection basis. Due to this, having control at firewall policy rule helps in tuning the system for performance as well as for the deployments.
• Inactivity timeout: Each session created from this rule inherits this inactivity timeout. If there are no packets within this inactivity timeout period, then the session is deleted. This value is in seconds. If this is not configured i.e if the value is 0, then it session inactivity timeout period determined from other configuration database information.
  

Multiple Access Control Lists:

Firewalls have come long way. Initially, firewall used to implement one ACL. Now, firewalls provide multiple lists to cater to different requirements.

• Normal ACL: This is traditional list. It contains rules for traffic going across zones.
• Dynamic ACL: This list is populated by other services and applications. That is, this list is not generated by the administrator. Rules are typically created in this list when configuration of some other service happens. uPNP and MIDCOM kind of applications create dynamic rules. These dynamic rules go to Dynamic ACL.
• User Group specific ACL: Normal ACL and Dynamic ACL rules are applied to entire traffic by default. Yet times, Enterprises require to provide user specific access control. That is, some privileged users might need to be given access to some important resources, which are prohibited for general users. Similarly, some users might need to be give access only to some particular resources and nothing else. To provide this flexibility, firewalls typically authenticate the user first and activate user specific rules. Providing and creating user specific ACL is big burden for administrators, if the organization has more than 10 users. Many a times, it is possible to categorize users into small number of groups. Administrator only needs to create as many ACLs as number of groups. I am calling these ACLs as 'User Group ACLs'.

Ordering of rule search: Firewall searches dynamic ACL, user group specific ACL and finally generic ACL. If no match, then packet gets dropped.

TR-069 and ordered lists: TR-069 does not have any specific RPC methods for ACS to move the position of rules in devices. Due to this, it is required that data models for any ordered lists have their own parameters to represent the priority. My suggestion is to have 'position' parameter for ordered lists. Lower the position number, higher the priority. ACS in its user interface need not provide 'position' as a configurable parameter. It can rather have intuitive drag and place UI for changing the relative position of the records with respect to others. Internally, ACS can change the 'postition' values of affected records and send them to the devices. Based on movement of records, 'position' value of many records may change though.


With above background, time window and firewall ACL representation in TR-069 can be represented in following way.

• internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.timeWindowObject.{i} PC
  
• name : String(32), RW - Name of the object. Once this variable is set, this can't be changed.
  
• Description: String(128) - RW - Value describing the object.
• Day1Begin: String, RW - Staring day of the week - Takes values 'Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday' and 'Sunday'.
  
• Day2Begin: String, RW
• Time1Begin: String, RW: Starting time in hour and minutes. 1:00PM is represented as 13:00 and 2:00PM is represented as 14:00PM and son on.
• Time2Begin: String, RW
• Day2Begin: Same as Day1Begin.
• Day2End: Same as Day1End.
• Time2Begin: Same as Time1Begin.
• Time2End: Same as Time1End.
• Day3Begin:Same as Day1Begin.
• Day3End:Same as Day1Begin.
• Time3Begin: Same as Time1Begin.
• Time3End: Same as Time1End.
• Day4Begin: Same as Day1Begin.
• Day4End:Same as Day1Begin.
• Time4Begin: Same as Time1Begin.
• Time4End: Same as Time1End.
• internetGatewayDevice.security.VirtualInstance.{i}.firewall P
• MaximumNumberOfRules: Read Only. Unsigned Int - This determines the rule ID in each ACL. Rule ID can't exceed this number.
  
• internetGatewayDevice.security.VirtualInstance.{i}.firewall.generalACLRules.{i} PC
• RuleID: Unsigned Int, RW - Identification to the rule. Its value can't exceed 'NumberOfRulesPerACL'. Once reord is created and ruleID is set, this can't be changed. This should be unique within the ACL.
  
• Description: String(128), RW - Description about this rule.
  
• Position: Unsigned Int, RW - This indicates the position of this rule in this list. Note that position values need not be consecutive. Lower the position number, higher the priority of the rule.
  
• Enable : Boolean, RW: 0 or 1 - Indicates whether this rule is enabled or disabled.
• FromZone: String(32), RW - One of the Zone IDs. It takes value of ZoneName from internetGatewayDevice.securityDomains.VirtualInstance.{i}.Zone.{i} table.
• ToZone: String(32), RW - One of the Zone IDs. It takes value of ZoneName from internetGatewayDevice.securityDomains.VirtualInstance.{i}.Zone.{i} table.
• SourceIPType: String(32), RW - It represents the source IP of the selector. It takes values such as 'immediate', 'ipobject'. Immediate indicaets that IP addresses are given as values and 'ipobject' indicates the IP address information points to one of the IPObjects.
• SourceIPValue: String(64), RW - If the type is immediate, then it can be single IP address in dotted decimal form, subnet by providing network IP address and prefix in terms of number or range of IP addresses with '-' in between low and high values. If the type is 'ipobject', then it has one of ipobject names from internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.IPValueObject.{i} table or internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.IPFQDNObject.{i} table. 'any' is special value indicating all source IP values. Examples: 10.1.5.10 or 10.1.5.0/24 or 10.1.5.1-10.1.5.254
• DestinationIPType: Same as 'Source IP Type'. This represents destination IP selector information.
• DestinationIPValue: Same as 'SourceIPValue'.
• ServiceType : String(64), RW - Represents the Protocol, source port and destination Port part of selectors of the rule. It takes values 'immediate' or 'serviceobject'. In case of 'immediate' type, protocol, source port and destination port values are part of the rule.
  
• ServiceObject: String(32), RW - One of the values of Service Records from the same virtual instance. This parameter is valid only if 'ServiceType' has value 'serviceobject'. 'any' is special value.
  
• Protocol : String(16), RW - It takes values such as 'udp', 'tcp', 'udptcp', 'icmp', 'esp', 'ah', 'ospf', 'ipinip' and integer value in string format representing the protocol value. This parameter is valid only if 'ServiceType' is 'immediate'.
• SourcePort: String(16), RW - It takes a single value or range of port values. Examples: 1214 or 1214-1230. This parameter is valid only if 'ServiceType' is 'immediate'.
• DestinationPort: String(16), RW - It takes a single value or range of port values. Examples: 1214 or 1214-1230. This parameter is valid only if 'ServiceType' is 'immediate'.
• TimeWindow: String(32), RW - It takes 'name' value from timewindow object table. 'none' indicates no timewindow.
• Action : String(16), RW - Action to be taken on the connection matching this rule. It takes values 'allow', 'drop', 'reject'.
• EnableLog: Boolean, RW - If the value 1, it generates logs upon session creation and session termination. Takes value 1 or 0.
  
• EnableTOSMangling: Boolean, RW - If value is 1, then firewall sets the TOS value in the IP header with the value of 'TOS' parameter.
• TOS: unsigned int, RW - Value can't exceed 255. Applicable only if 'EnableTOSMangling' is set to 1.
• EnabelMSSMangling: Boolean, RW - Take values 1 or 0. If set to 1, TCP Option MSS is set with the minimum of value given in 'MSS' parameter and the value in TCP packet.
• MSS: Unsigned int, RW.
• EnableBandwidthRateControl: Boolean, RW - Takes value 1 or 0. If set to 1, 'ByteRate' parameters is valid.
• ByteRate : String(32), RW - It takes form of X/Y - X being number of Kbytes and Y being number of seconds. Example: 10/5 means limit the traffic falling to this policy to 10Kbytes for 5 seconds. This parameter is valid only if 'EnableBandwidthRateControl' is set to 1.
• EnableConnectionRateControl: Boolean, RW - Takes values 1 or 0.
  
• ConnectionRate: String(32), RW - It also takes form of X/Y - X being number of connections and Y being number of seconds. Example: 1000/3600 limits number of connection establishments to 1000 per hour. This parameter is valid only if 'EnableConnectionRateControl' is set to 1.
  
• EnableMaxConnectionsControl: Boolean, RW - takes values 1 or 0.
• MaxConnections: Unsigned Int, RW - Maximum number of connections allowed at any time. Example: 1000 indicates that number of connections falling in this policy rule will not exceed 1000.
• EnableSigBasedIntrusionDetection: Boolean, RW - Takes values 1 or 0. Value 1 enables intrusion analysis.
  
• EnableTrafficAnomalyDetection: Boolean, RW - Takes values 1 or 0. Value 1 enables traffic anomaly detection.
• inactivityTimeout: Unsigned Int, RW - Default is 0. Value represented in seconds.
  
• internetGatewayDevice.security.VirtualInstance.{i}.firewall.dynamicACLRules.{i} P : It is repetition of above table, except that all values are read only.
  
• RuleID: Unsigned Int, Read Only.
  
• Description: String(128), Read Only.
  
• Position: Unsigned Int, Read Only.
  
• Enable : Boolean, Read Only
  
• FromZone: String(32), Read Only.
  
• ToZone: String(32), Read Only.
  
• SourceIPType: String(32), Read Only
  
• SourceIPValue: String(64), Read Only.
  
• DestinationIPType: String(32), Read Only.
  
• DestinationIPValue: String(64), Read Only.
• ServiceType : String(64), Read Only
  
• ServiceObject: String(32), Read Only.
  
• Protocol : String(16), Read Only
  
• SourcePort: String(16), Read Only
  
• DestinationPort: String(16), Read Only
  
• TimeWindow: String(32), Read Only
  
• Action : String(16), Read Only
  
• EnableLog: Boolean, Read Only
  
• EnableTOSMangling: Boolean, Read Only
  
• TOS: unsigned int, Read Only
  
• EnabelMSSMangling: Boolean, Read Only
  
• MSS: Unsigned int, Read Only
  
• EnableBandwidthRateControl: Boolean, Read Only
  
• ByteRate : String(32), Read Only.
  
• EnableConnectionRateControl: Boolean, Read Only.
  
• EnableMaxConnectionsControl: Boolean, Read Only.
  
• MaxConnections: Unsigned Int, Read Only.
  
• EnableSigBasedIntrusionDetection: Boolean, Read Only.
  
• EnableTrafficAnomalyDetection: Boolean, Read Only.
• inactivityTimeout: Unsigned Int, Read Only.
  
• internetGatewayDevice.security.VirtualInstance.{i}.UserGroups.{i} PC
• Name: String(32), RW - Name of the user group. Once this is set, this can't be changed.
• Enable : Boolean, RW - Takes value 1 or 0.
  
• internetGatewayDevice.security.VirtualInstance.{i}.firewall.ACLRules.{i} PC - Following section is repetition of genericACLRules.
  
• RuleID: Unsigned Int, RW - Identification to the rule. Its value can't exceed 'NumberOfRulesPerACL'. Once reord is created and ruleID is set, this can't be changed. This should be unique within the ACL.
  
• Description: String(128), RW - Description about this rule.
  
• Position: Unsigned Int, RW - This indicates the position of this rule in this list. Note that position values need not be consecutive.
  
• Enable : Boolean, RW: 0 or 1 - Indicates whether this rule is enabled or disabled.
• FromZone: String(32), RW - One of the Zone IDs. It takes value of ZoneName from internetGatewayDevice.securityDomains.VirtualInstance.{i}.Zone.{i} table.
• ToZone: String(32), RW - One of the Zone IDs. It takes value of ZoneName from internetGatewayDevice.securityDomains.VirtualInstance.{i}.Zone.{i} table.
• SourceIPType: String(32), RW - It represents the source IP of the selector. It takes values such as 'immediate', 'ipobject'. Immediate indicaets that IP addresses are given as values and 'ipobject' indicates the IP address information points to one of the IPObjects.
• SourceIPValue: String(64), RW - If the type is immediate, then it can be single IP address in dotted decimal form, subnet by providing network IP address and prefix in terms of number or range of IP addresses with '-' in between low and high values. If the type is 'ipobject', then it has one of ipobject names from internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.IPValueObject.{i} table or internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.IPFQDNObject.{i} table. 'any' is special value indicating all source IP values. Examples: 10.1.5.10 or 10.1.5.0/24 or 10.1.5.1-10.1.5.254
• DestinationIPType: Same as 'Source IP Type'. This represents destination IP selector information.
• DestinationIPValue: Same as 'SourceIPValue'.
• ServiceType : String(64), RW - Represents the Protocol, source port and destination Port part of selectors of the rule. It takes values 'immediate' or 'serviceobject'. In case of 'immediate' type, protocol, source port and destination port values are part of the rule.
  
• ServiceObject: String(32), RW - One of the values of Service Records from the same virtual instance. This parameter is valid only if 'ServiceType' has value 'serviceobject'. 'any' is special value.
  
• Protocol : String(16), RW - It takes values such as 'udp', 'tcp', 'udptcp', 'icmp', 'esp', 'ah', 'ospf', 'ipinip' and integer value in string format representing the protocol value. This parameter is valid only if 'ServiceType' is 'immediate'.
• SourcePort: String(16), RW - It takes a single value or range of port values. Examples: 1214 or 1214-1230. This parameter is valid only if 'ServiceType' is 'immediate'.
• DestinationPort: String(16), RW - It takes a single value or range of port values. Examples: 1214 or 1214-1230. This parameter is valid only if 'ServiceType' is 'immediate'.
• TimeWindow: String(32), RW - It takes 'name' value from timewindow object table. 'none' indicates no timewindow.
• Action : String(16), RW - Action to be taken on the connection matching this rule. It takes values 'allow', 'drop', 'reject'.
• EnableLog: Boolean, RW - If the value 1, it generates logs upon session creation and session termination. Takes value 1 or 0.
  
• EnableTOSMangling: Boolean, RW - If value is 1, then firewall sets the TOS value in the IP header with the value of 'TOS' parameter.
• TOS: unsigned int, RW - Value can't exceed 255. Applicable only if 'EnableTOSMangling' is set to 1.
• EnabelMSSMangling: Boolean, RW - Take values 1 or 0. If set to 1, TCP Option MSS is set with the minimum of value given in 'MSS' parameter and the value in TCP packet.
• MSS: Unsigned int, RW.
• EnableBandwidthRateControl: Boolean, RW - Takes value 1 or 0. If set to 1, 'ByteRate' parameters is valid.
• ByteRate : String(32), RW - It takes form of X/Y - X being number of Kbytes and Y being number of seconds. Example: 10/5 means limit the traffic falling to this policy to 10Kbytes for 5 seconds. This parameter is valid only if 'EnableBandwidthRateControl' is set to 1.
• EnableConnectionRateControl: Boolean, RW - Takes values 1 or 0.
  
• ConnectionRate: String(32), RW - It also takes form of X/Y - X being number of connections and Y being number of seconds. Example: 1000/3600 limits number of connection establishments to 1000 per hour. This parameter is valid only if 'EnableConnectionRateControl' is set to 1.
  
• EnableMaxConnectionsControl: Boolean, RW - takes values 1 or 0.
• MaxConnections: Unsigned Int, RW - Maximum number of connections allowed at any time. Example: 1000 indicates that number of connections falling in this policy rule will not exceed 1000.
• EnableSigBasedIntrusionDetection: Boolean, RW - Takes values 1 or 0. Value 1 enables intrusion analysis.
  
• EnableTrafficAnomalyDetection: Boolean, RW - Takes values 1 or 0. Value 1 enables traffic anomaly detection.
• inactivityTimeout: Unsigned Int, RW - Default is 0. Value represented in seconds.

UTM Message logging capabilities

UTM devices are multi function security devices. These functions include firewall, IPsec VPN, SSLVPN, IPS, Anti Virus and Anti Spam. Some devices even include web application firewall function. Each security function is different from others. So, one would expect the information in logs is different across different security functions and even sub functions within a security function. Logs are generated for many reasons. Logs are generated not only to indicate the policy violations, intrusion detections, virus or spam detections, but also generated to indicate session information, configuration changes, login failures, system errors, system warnings etc.. Type of information in each of these types of log messages is different from each other. To facilitate analysis of logs by external log analyzers, each family of log messages should have its own format. That is, each family should have its own keywords to represent the values of different parameters. There could be many message families.

Each message family contains multiple different types of logs. For example, there could be multiple types of logs in intrusion message family. Typically, each signature in IPS is one type of log. In signature based IPS, there are as many logs as number of signatures. Each type of log message is typically represented by message IDs.

UTM devices generate logs during its operation. There could be large number of incidents which generate similar type of log multiple times. Messaging system component of UTM devices provide multiple controls for administrator to control logs. Some of the controls are:

• Message ID level control on enable/disable: If a message ID is disabled, any log incidents of that message ID are not processed, that is, they are not stored or exported to external log collectors.
• Message ID based Log frequency control: It allows logs, but controls the number of logs generated for processing. Typically it takes two parameters - Log threshold count and log threshold time. At most one log is processed for every 'log threshold count' logs or within the 'log threshold time'. That is, if log threshold count is 100 and log threshold time is 5 minutes, then if number of logs generated are more than 200, but less than 300 within 5 minutes, then it emits three logs - 1st log, 101st log and 201st log.
  
• 5 Tuple based Log frequency control: Message ID based log frequency control is good for non-network based logs such as system errors, warnings etc.. But, for connections, this kind of control is not good enough. If there is same intrusion detected in traffic going to multiple victims, then administrator would like to know at least one instance of intrusion going to each victim. With message ID based log frequency control, all victims will not be reported if the intrusions happen within 'log time threshold'. To avoid this, 5 tuple based log frequency control is required. 'Source IP', 'Destination IP' , 'Protocol', 'Source Port' and 'Destination Port' constitute 5 tuple. Each item in tuple can be enabled/disabled. 'Log threshold count' and 'log threshold time' described above is individually valid for each combination of 5-tuple items enabled. Let us say that, if administrator enabled 'source IP' and 'destination IP' addresses in the 5 tuple for a given message ID, then for logs generated for this combination of network connections are processed as per 'log threshold count' and 'log threshold time'. Using this, log systems don't miss attacks and other events happening on individual victim machines and also don't miss reporting 'attackers'.

In multi function security devices like UTM, there could be many message IDs. It is no surprise, even the number of message IDs are in the tune of 1000. Controlling individual message ID would be night mare for administrators. UTM devices need to provide control of logs at higher level than the control on message ID basis. That is where, sub-family of message IDs come in handy. Sub-family is nothing but group of message IDs. This grouping is mainly for log control. This grouping does not define the message format. The controls 'Enable/Disable', '5 tuple based log frequency control' can be specified on per sub-family basis. All message IDs inherit these controls. Of course, administrators are provided control on message ID basis too, when he/she thinks that some message IDs can't inherit the controls from their sub-families.

Though many aspects of this article is based on Intoto iGateway product family concepts, these are generic concepts and valid for any security products. The typical flow of logs is:

• Logs are generated by applications such as firewall, IPS, AV/AS, WAF etc..
• Log throttling system discards logs and processes only some logs based on controls configured by administrator.
• Logs are then stored, exported.
• Log analyzers (local or external) analyzes logs and even create reports. Also, they provide 'search' functions based on different criteria.

Logs are exported by either using 'syslog' or 'email'. iGateway UTM also has facility to store logs locally in a database (postGres).

Exported logs are sent in a format for log analyzers to easily extract values for different fields. WELF is one format that became quite popular. Though iGateway product family uses WELF syntax, but it does not use keywords as specified by WELF as they are incomplete. Some of the rules it follows are:

• Each log forms one line with each line containing multiple fields.
  
• Each field is formed as keyword=value. Keywords are defined by corresponding message family. Each keyword and value pair is separated by one or more spaces.
  
• keyword and values should not have any spaces. They must not have any quotes and '=' sign characters. If the value needs to have spaces, then the value string must be enclosed in double quotes.
• Mandatory keywords across all message families
• time: Date and time in double quotes.
  
• priority: Priority of the message. One of values from 1 to 7.
  
• id : Identity of the device sending the logs. It is configured by administrator on each device.
• mtype: Message family.
• mid: Message ID
  
• Generic keywords: There are some generic keywords which are valid across multiple message families. Though these are not mandatory, these generic keywords must be used wherever they are needed.
  
• vsg : Virtual instance.
  
• fromzone: Zone in which the connection is originated.
• tozone: Zone in which the connection is terminated.
• userid : User name, if this information is known.
• usergroup: User group.
  
• sip : Source IP address in dotted decimal form.
• dip: Destination IP address in dotted decimal form.
• protocol: Protocol of the connection as integer.
• sport: Source port
• dport: Destination port.
• Other keywords are specific to each message family.

Firewall and NAT using same rule base Vs. different rule bases?

Many SMB security devices combined firewall and NAT in one rule base. That is, firewall rule itself can be configured with NAT configuration. I feel this is fine with when you have small number of rules. In Enterprise environment, having two rules bases provide very good flexibility.

NAT and firewalls are two different functions. Firewall rule base is mainly meant for providing access control for different networks, machines and services. SNAT is mainly meant for providing internet access for multiple computers with less number of public IP addresses and also to hide internal IP addressing to the outside world and DNAT is meant for providing access to servers in private network from Internet.

Granularity of firewall rules is very high. Yet times, firewall rules are created with single IP address or service. Also firewall rules are activated upon user login. If NAT configuration is part of firewall, then the NAT configuration needs to be duplicated many times in the firewall rule base.

Granularity of NAT rules is very low. Typically, the granularity is at the network level. So, the number of rules for NAT would be small and independently manageable.

There is another advantage in having independent rule bases. That is to do with role based management. In many Enterprises, NAT configuration is treated as network function, not security function. Firewall function is considered as security function. Organizations having different personnel for security and network administration find it convenient.

Of course there is small disadvantage of having two rule bases. The connection rate performance is typically less than the single rule base. Since the number of rules in NAT rule base will be small in number, this would not be a big disadvantage.

If I were the administrator, I prefer to go with security devices implementing these two function in two different rule bases.

Traffic shaping and Real time traffic - Tips

Don't assume that traffic shaping provided by many Residential and SMB gateways is enough for clear VOIP qulaity. It certainly improves the voice quality, but traffic shaping alone is not sufficient in many scenarios. Try downloading or uploading a file from popular sites in Internet, your voice quality suffers. Why is this? Hopefully this technical tip encourages the device vendors to start thinking towards providing better traffic management in their next firmware versions.

Introduction:
Many existing Residential and SMB gateways are based on SoCs (System On Chip). SoCs combine processor, Ethernet MACs, in some cases wireless MAC, Memory controller, Crypto accelerator and other peripheral buses into one single chip. These processor speed is in the tune of 400 to 500 MIPS. Recent generation of SoCs include gigabit MACs and DDR2 memory controller and also improving performance using fast path acceleration technologies. Though main processing speed has been going up, but not the extent of your Desktop PC processors. Many of these gateways typically drive upto 10Mbps links and hence it is thought this processing power is good enough to saturate bandwidth on WAN links. And it is fair assumption.

These gateways typically installed behind DSL/Cable/T1 modems via Ethernet. Though Ethernet speed is in terms of 100Mbps, but the actual speed of the WAN link is limited to few Mbps. Traffic shaping functionality of RG and SMB gateways actually consider effective bandwidth while shaping the outbound traffic. Within this bandwidth, these devices prioritize the outbound traffic based on multiple conditions. More often or not, it is based on TOS (Type of Service) field of IP header. Many devices also provide functionality and configuration to set the TOS value based on 5 tuple rules. This helps in cases where VOIP TA or VOIP phone behind the gateway does not set the higher values in TOS for voice traffic (RTP traffic) by providing option for administrator to set higher TOS value for the traffic coming from known phones and adapters. Many devices also support setting TOS values on RTP traffic dynamically.

Traffic shaping with this priority based scheduling works in many cases. But, it may not work in cases

• where CPU power is limited: Many routers are coming with 1Gbps ports on LAN side. If somebody pumps the traffic at very high rate, CPU is busy in processing these packets and they get dropped at the WAN link due to traffic shaping. Since CPU is busy, it might not process some real time traffic.
  
• where incoming bandwidth of WAN link is used up: Traffic shaping helps in shaping and scheduling outbound traffic, but it has no control over the packets coming from WAN link. If one of PCs locally is downloading a big file or movie, it might affect the VOIP traffic and you may not hear the remote party well.
  

That is why, it is very important to have Traffic policing in addition to Traffic shaping. Traffic policing functionality typically throttles the traffic at ingress side. Traffic policing can limit the bandwidth usage of incoming traffic and prioritize the traffic. The bandwidth of the traffic that needs to be allowed should depend on CPU power. This policing should happen as soon as packet enters into the system to save CPU cycles. On the WAN link too, traffic policing should be done to make the sender send the traffic little slowly. This may not work for non-TCP traffic, but again, majority of traffic is TCP, so it should work fine as a system.

For developers, I suggest to go with simple token bucket algorithm to detect and throttle traffic. Similar to traffic shaping, multiple rules can exists with each rule having set of throttling parameters, with each rule identifying the traffic by 5 tuple selectors and TOS values.

Multiple router web server vulnerabilities - Recommendations

Recently I came across a project in GNUCitizen called 'Router Hacking Challenge'. It was a project to gather vulnerabilities in home routers. Many interesting problems were found and mainly related to Embedded web servers in the routers. I did some search and found that these problems are not confined to small routers, but they are equally applicable to Enterprise networking devices too. Any security administrator get alarmed by this. It appears that embedded webserver is the weak part of network devices. In this article, I summarize kinds of vulnerabilities found and some recommendations to developers to overcome these problems.

There are many entries in 'Router Hacking Challenge' project. But at high level they fall onto:

• Authentication Bypass
  
• IP based session Management
  
• CSRF (Cross Site Request Forgeries)with default passwords.
• CSRF with Cookies
• XSS (Cross Site Scripting) attacks - Mainly persistent XSS attacks.
  
• Configuration files in document root.
  
• DOS attacks

There is a misconception that as long as web interface is not exposed to the outside world (untrusted network), the device is safe. It is not so. Administrators when lured to visit attacker sites, the malicious page could take control of the device. Attacker site may be exposing a image or hyperlink which internally connects to the local security device of administrator and changes the configuration. When the administrator inadvertantly clicks on these links, then the damage is done. Since, this link is connected from administrator browser, attack is successful. These links and associated java script can do things like opening remote administration, creation of new administrator and changing the policies.

Let me go ahead and give brief description of kinds of attacks.

Authentication Bypass

Normal browsing of network devices' web interface appear to be secure as it forces the user to enter login credentials. But it appears that, in some cases, if user knows internal URLs of web interfaces, he/she can access the interface without going through authentication process. In some cases, it appears that HTTP GET requests are authenticated, but not POST methods. Once the attacker knows the POST action URL and POST variable definition for various configuration items, the attacker can easily take control over the device or change any settings in the device.

IP Address based Session Management

Some devices seem to implement IP address based session management. Once the user is authenticated by the web server, web servers allows the HTTP requests from this IP address without any other validation.

• If there is a NAT router doing SNAT in between the administrator PC and the device, then any other user behind the same NAT router can access the device without signing in.
  
• If some body in organization knows the IP address of administrator PC, he/she can assign the same IP address to his/her PC when administrator is not present and access the device configuration. Of course timing is important in this case, that is, it is possible to access device only after administrator has successfully logged into the device and before the device removes the session due to inactivity.

CSRF with default passwords:

Many administrators don't change the default passwords that come with firmware. Also many home users don't even change IP address of the devices. They assume that devices are safe as long as remote administration of the device is turned off. When any user in the Enterprise or home are tricked into visit an attacker site, the malicious page could change the configuration of the device. Malicious page could have javascript which generates the HTTP request, logs in with default user name and password and send POST method with variables. It becomes easy for an attacker to create the malicious pages with default passwords and IP addresses. Of course the attacker should know the vendor and model of the device to frame the page with appropriate POST method and variables. But, this is not that difficult as some vendors are very popular and they use same software across multiple models.

CSRF with Cookies:

Even when the administrators change the default passwords, there is still possibility of changing the configuration of the devices. This can happen when the administrator is tricked into visiting attacker site when login session with the device is active. When the login session is active, the session cookie, which is used to validate the HTTP requests for authenticity, is still valid. When administrator visits the malicious page with javascript, then the script could make a connection and send HTTP request with details attackers wants. Since it is connecting to the device, browser sends the session cookie along with the HTTP request. HTTP Server does not suspect anything and honors the request.

XSS attacks:

When administrators visits any device page which was XSSed before, then XSS script gets executed. XSS injection could have happened before due to :

• CSRF vulnerability: The attacker site could introduce javascript or any other script in one of the configuration values or logs etc..
• Logs: Some web servers when presented with unknown URI might store URI in one of the log messages. Attacker can induce XSS as part of invalid URI. When administrator visits these log messages at later time, XSS script gets executed.
  

XSS scripting can be used to steal information such as passwords and device configuration. XSS scripts can be written to post this information to attacker websites, public forums etc..

XSS injection problem typically occurs when the input fields are not sanitized by the web servers.

Configuration files and Clear passwords:

It appears that some devices are storing the configuration in the same or nested directories as HTML pages. Due to this anybody having access to the device both privileged and non-privileged users get access to the configuration files. When devices store the passwords in clear, every user comes to know about admin passwords, PPPoE passwords etc.. These passwords can be used for malicious purposes.

DOS attacks:

It appears that webserver implementation of some devices is week. Different techniques are used to crash the web servers. Most of them seem to be buffer overflow attacks. Some of them are:

• Large URI in GET and POST requests.
• Large data as part of HTTP request header name
  
• Large data as part of HTTP request header value.
• Large data as part of HTTP Post Variable name.
• Large data as part of HTTP POST variable value.
• Unknown header name
  
• Sending duplicate headers.

Recommendations for developers:

• Always use challenge based authentication for HTTP access: Challenge must be generated from random data. Every new login session should be presented with different challenge value.
  
• It is advisable to use MD5 on challenge and password entered by user before sending to the web server. Don't send the password in clear. This can be achieved using javascript. Javascript code should be present in the page that takes user name and password from the user.
• Always use session cookies: Once the authentication is successful, webserver should set the session Cookie. This session cookie also must be generated from random data. Don't use source IP address as part of session cookie. Don't use persistent cookies, that is, If browser is closed, this cookie should disappear on the PC.
  
• Validate all requests against session cookie: All methods (whether they are GET or POST) must have session cookie that was sent as part of login session. Since it is not general purpose web server, there is no need entertain any URI without any authentication. If session cookie is not received in the request, then web server should send 'login' page to authenticate the user.
  
• Main page should predominantly show if any default passwords are still present in the device. It helps as a reminder to administrator to change the passwords.
• To avoid from the CSRF with cookies, it is advisable that web server sends a random token along with page having 'forms'. Javascript on the page should always send back the token along with submitted values. Web server validates the token before processing the request. This token value should not be sent to the browser using Cookie (Set-Cookie) as it defeats the purpose. This should be sent separately along with requested page. Web server should keep refreshing these token values periodically or even can change for every page. Web server should keep track of these tokens on per login session.
• To avoid from XSS attacks, web server and logic associated with functionality must validate all input values.
  Ensure that the values are not stored anywhere without validating them.
  
• Web Server should not send the user input values or URL in its response page. Also, it should not store these values in any log messages.
• To avoid from DoS attacks, penetration testing must be done. There are many tools available to inject large data in HTTP request header names and values. But, to test the application specific HTTP POST variable names and values OR XML fields and values, one must create application specific tools to send large amount of data in field names and values. Also, these application specific tools should also have capability to fuzz the data and ensure that system withstands and also ensure that system does not reflect these invalid values on the screen (to avoid reflective XSS sttacks). Many times, it is my observation that application specific tools are not created by developers to ensure that there are no buffer overflow attacks. That makes the web server and its associated logic a weak link.

Port Triggering in firewall - TR069 support

Port triggering feature allows firewall to create pin holes on a connection trigger. These pin holes allow connections that are not explicitly opened by the administrator. Trigger connection details and pin hole details are configured by the administrator. This feature allows pin hole creation in the same direction of the trigger connection or in reverse direction or in both directions. This is specifically useful in cases where applications make new dynamic connections - Client initiated or server initiated connections. TCP/UDP ports used by these dynamic connections are known information. One can say that if this information is known, why can't administrator create the ACL rules? What is the need for port triggering? If ACL rules are created, they are permanent, that is, access on these ports are allowed even in cases where application is not run. In cases where local machines are getting the IP addresses from DHCP Server, it is difficult and some times not possible to create the ACL rules and administrator may be forced to open the door wider.

Examples of port triggering:

IRC: IRC client when it makes connections to IRC Server on one of the ports between 6660 to 76670, IRC server makes connection back to IRC client on port TCP 113 (IDENT protocol). Administartor does not need to open access 113 to all machines inside permanently. Admin is expected to allow only outbound connections on IRC (6660-6670) and configure port triggering functionality to open pin hole to allow reverse IDENT connection. Port triggering functionality, upon trigger connection, opens pin hole to allow IDENT connection coming from IRC Server to IRC client.

Microsoft ActiveSync protocol: Client makes connections to ActiveSync Server on Port 5678. Server makes connection back to on port 990. In this case too, port triggering record with trigger port as 5678 and reverse pin hole creation with port 990, allows inbound connection without having to create permanent ACL rule.

Firewall have capability to even create pin holes when the NAT is involved. There is one issue though when NAT is present. If there are multiple machines behind NAT are making connections to the same server and if server uses same port for reverse or forward dynamic connections, then firewall/NAT functionality will not have enough information to redirect the connections to the right machine. If server is different, there is no issue though.

Many firewalls do support this feature. Configuration consists of a list (non-ordered) containing port triggering rules with each rule having:

• Name : for identification
  
• Description: Description of this record - Mainly for administrators to give some descriptive information.
• Enable/Disable
  
• Triggering information
• Protocol : IP Protocol
• Port Range: Destination Ports of the triggering connection.
• Same direction pin hole details : If the triggering connection is made from 'corporate' to 'external', then the pin holes configured here will allow connections initiated from 'corporate' to 'external'. Many applications work fine with three ranges of pin hole settings.
  
• Protocol
  
• Port Range : Firewall opens the connections to these ports and protocol once the triggger connection is detected.
• Reverse direction pin hole details: Pin holes allow connections in reverse direction of the triggering connection. As per above example, pin holes in this section are created for connections from 'external' to 'corporate' from original server IP to original client IP. Here too, I feel three ranges are good enough for many applications.
• Protocol
• Port Range: Firewall opens the connections to these ports in reverse direction upon detecting the trigger connection.
• Disassociate pin holes from trigger connection: If this is set, then the pin holes exist even if the trigger connection is terminated. If this is not set, then the pin holes are removed along with the trigger connection.
• Inactivity timeout of pin holes: Typically, inactivity timeout is derived from the trigger connection inactivity. Administrator can set this explicitly using this parameter.

TR-069 data model:

• internetGatewayDevice.security.VirtualInstance.{i}.firewall.portTriggerRules.{i} PC
• Name: String(32), RW - Since this is identifier, once this record is created with a name, this can't be changed.
• Description: String9128), RW
• Enable : Boolean, RW - Takes value 1 or 0
• TriggerProtocol: String(16), RW - Takes values such as 'udp', 'tcp', 'udptcp', integer value in string to represent other protocols.
• TriggerPortRange: String(16), RW - Since it is range, the low and high values of the range must be separated by '-'. Example: 1000-2000
• forwardPinHoleProtocol1: String(16), RW - It takes values 'udp', 'tcp', 'udptcp', and any integer value representing protocol.
• forwardPinHolePortRange1: String(16), RW: Destination Ports for forward pin holes.
  
• forwardPinHoleProtocol2: String(16), RW
  
• forwardPinHolePortRange2 : String(16), RW
  
• forwardPinHoleProtocol3: String(16), RW
  
• forwardPinHolePortRange3: String(16), RW
• reversePinHoleProtocol1: String(16), RW - It takes values 'udp', 'tcp', 'udptcp', and any integer value representing protocol.
• reversePinHolePortRange1: String(16), RW: Destination Ports for reverse pin holes.
  
• reversePinHoleProtocol2: String(16), RW
  
• reversePinHolePortRange2 : String(16), RW
  
• reversePinHoleProtocol3: String(16), RW
  
• reversePinHolePortRange3: String(16), RW
• DisassociatePinHolesWithTrigger: Boolean, RW - Takes value 1 or 0. 1 means the pin holes are not removed whenever trigger connections times out or terminated.
  
• PinHoleInactitityTimeout: unsigned int, RW - 0 means that it inherits the inactivity timeout from trigger connection. Any other value provided specific inactivity timeout.

Network Security - Standard Proxies Vs Transparent proxies

Network based security devices provide application firewalls and Virus scanning functionality using proxies. Proxies act as servers for client end points and clients for server end points. They typically terminate the connection (UDP or TCP) from clients and make connections to the original servers. In between they do processing on the data such as access control and scanning and take actions based on the processing results.

Proxies are implemented in two ways - Standard and Transparent. In case of standard proxies, both client and server end points know that there is a proxy in between. Client end points are configured with standard proxy address and server sees the connections from the proxy. That is, destination IP address of the packets from client is proxy IP address and source IP of the packets from proxy to server is proxy IP address. In case of transparent proxies, both client and server end points don't know the existence of the proxy in between. In this mode, both source and destination IPs of the packets correspond to client and server end points. At no time, proxy IP address is seen in the packets on the wire.

Advantages of standard proxies:

• Devices with standard proxies need not be in the line of traffic.
  
• Traffic that need not go through the proxies will not have any latency introduced.
  
• Any problem with security device only effects the proxy traffic and not every thing else.
• Single leg mode: Device only need to have one Ethernet port.
• Standard proxies don't require any special support in TCP/IP stack: In Linux and BSD systems, proxies are typically implemented in user space and they use standard socket library to accept connections, make connections, receive and send data. Since, the packets are destined to local IP address, there is no change required in TCP/IP stack or Linux Kernel for this support. In case of transparent proxies, support is required in Linux kernel to redirect the packets to proxy applications.
  

Advantages of transparent proxies:

• No special configuration required in client end points.
• Servers see traffic from client end points. There is no issue if there is any analysis is required on unique clients connecting to servers from logs as logs indicate original client IP address.
• Any other policy based network devices continue to work as there is no change seen in IP addresses of the packets. Example: Traffic shaping devices doing shaping based on IP addresses.
• It does not have any issues with 'Unknown Domain Resolution' DoS attack. Standard proxies typically get the domain name from the protocol fields and does domain name resolution to get the IP address of server. Attackers can DOS the standard proxy by connecting to proxy with invalid server domain names. If the proxy is implemented to create a thread for each connection, then the proxy can easily be DOSed as these threads block on DNS resolution for a long time. Note that the number of threads that can be created in any given system are limited in number. If the proxy is implemented as FSM (finite state machine) without any blocking calls, then this problem may not arise soon, but the memory can be exhausted due to state maintenance. Transparent proxies don't have this problem as it does not do any domain name resolutions.
  

As indicated above, transparent proxies require special support in Linux/BSD kernels to intercept the packets at the IP layer and redirect them to the proxies. This can be implemented as kernel module. I try to give some brief description on the operations of this module.

• Operation:
• Intercept all packets at the IP layer. In case of Linux, it can be done using 'netfilter' hooks.
• If the packet is first packet of the session, it creates a memory block to store the state information (let us call this as session block). Then it checks whether this packet requires to be redirected to local proxy. If so, it notes down the client IP address (source IP address) and server IP address (destination IP address) in the session block and changes the destination IP of the packets to local IP address. Thereby, rest of the TCP/IP stack sends the packets to the proxy. Note : There are no other changes required in TCP/IP stack.
  
• For further packets, it uses the session state information to redirect the packets.
  
• When proxy makes connection to the server, it uses local IP address which is used as source IP. Since, it is required that server sees original client IP, this module does SNAT on Server to proxy connection. It also creates a session to store any state information.
  
• Proxy interface
• This module can be configured with rules for redirection. Either administrator can configure the rules or proxies themselves can configure the rules. For example, web proxy might configure to redirect all HTTP connections coming from a particular subnet to be redirected to it. It can also configure to redirect HTTP connections going to a particular server (Reverse proxy scenario).
  
• This module provides interface to return original server IP address when proxy requests it. Proxy needs this IP address to make connection to the server end point.
• This module provides interface to link client side session to server side session. It is expected that proxy informs this module before connecting to the server. This module prepares itself by creating server side session and populates the session with state information do to the SNAT.

Direct Download Links/One Click Hosters - Detection by IPS

One click hosting has become very popular in recent past. Unlike P2P protocols such as BitTorrent, one click hosting providers host the entire file (video, audio or data) at one place and provide access to the files using normal browsers using HTTP. Their model is very simple. They let users to upload files. For each uploaded file, users are provided with the 'download link'. Users can put these download links in their website, blogs or forums. When others click on this download link, it takes them to the provider site. Two providers megaupload and rapidshare are very popular one click hosting providers. There are many sites providing this kind of service.

Another kind of sites are video hosting websites such as youtube. These sites allow users to upload videos and view existing videos. Please see this link to see some of video hosting sites. It appears that traditional P2P file sharing popularity is declining with these two kinds of sites.

Network security devices, specifically IPS functionality, have been providing P2P application detection and facility to block these applications or throttle the traffic from these applications, thereby saving precious bandwidth for Enterprise applications' traffic. With One-click-hosting sites and video sharing sites, these controls are becoming less effective. Network security devices need to change with current reality. Due to the usage of HTTP for downloading the video content, it is a challenge for network security devices to detect this traffic. That is where, signature based application detection comes in handy.

As an administrator, you should look for solution that offers:

• Detection of hosting provider with no false positives.
• Detection of hosting provider within first 2K byte of connection.
  
• Ability to set the throttling parameters based on
  
• hosting provider(s).
• machine(s) accessing the hosting site.
• Time of day
  
• Ability to set throttling parameter such as
• Bandwidth (bytes/sec or packets/sec).

IntruPro-IPS from Intoto provides following facilities:

• Signatures to detect hosting providers. Each hosting provider is identified by 'Application ID' using signatures.
• Signatures can be developed with HTTP protocol keywords. Hence detection can happen with zero false positives.
• Traffic Enforcement rules with each rule containing
• Source IP address(es)
• Application ID(s)
• Time of day and Day of week.
• Action : Block Or throttle
  
• Bandwidth in bytes/sec or pkts/sec
• Flag to indicate whether to apply traffic throttling based on each source IP.

PCI DSS - UTM technology requirements

Many security vendors claim that their products can be used to comply with PCI DSS requirements. But, it is difficult to find exactly what is it one should look for in these kinds of products. I try to address this in this blog entry. My focus in this article is on network firewall, IPS, VPN , AntiVirus and web application proxy firewall products.

PCI DSS (Payment Card Industry Data Security Standard) defined 12 requirements. For more information on these requirements, please refer to PCI DSS standards. Those 12 requirements are:

• Requirement 1 - Install and Maintain a firewall configuration to protect cardholder data.
• Requirement2 - Don't use vendor-supplied defaults for system passwords and other security parameters.
• Requirement3 - Protect stored cardholder data
• Requirement4 - Encrypt transmission of cardholder data across open, public networks.
• Requirement 5 - Use and regularly update anti-virus software or programs.
• Requirement 6 - Develop and maintain secure systems and applications.
• Requirement 7 - Restrict strong access to cardholder data by business need-to-know.
• Requirement 8 - Assign a unuqie ID to each person with computer access
• Requirement 9: Restrict physicall access to cardholder data
• Requirement 10: Regularly monitor and test networks
• Requirement 11: Regularly test security systems and processes
• Requirement 12: Maintain an Information Security Policy

Security infrastructure products like firewall, IPS, VPN, Web application firewalls are used to satisfy the PCI DSS requirements.

Expectation from Network based Firewalls:

• Ability to support multiple zones: Cardholder data servers must be separated from the frontend servers such as Web Servers. Thereby, any compromised front server does not give access to cardholder information to attackers. Typically, frontend servers are kept in DMZ of firewall. Cardholder information must be kept in other zones. Firewall should have facility to define new zone and firewall must have facility to setup rules for each zone. All cardholder servers can be kept in the zone separate from 'corporate zone', 'dmz zone' and 'untrusted zone'.
  
• Ability to create access control policy rules: Firewalls must support creation of rules among zones. For example, access to cardholder data servers should only be allowed from 'frontend servers' in 'DMZ' zone. In addition, only specifc ports should be allowed. Firewalls must have capability allowing rule creation with 'From' and 'To' Zones, 'From' and 'To' IP addresses and Service Ports.
• Ability to define rules with named objects: Firewall rules can become pretty complex. For readability and manageability, administrators would like to create rules with named objects (IP address objects and Service Objects).
  
• Ability to generate logs and alerts: Any attempt to card holder data other than the allowed policy rules must be notified. It is also required that firewalls have ability to generate log events for each connection that was allowed. Firewalls should send enough information along with the notification for analysis.
  
• Ability to detect and protect from DOS & DDOS attacks.
• Ability to hide internal IP addresses using NAT capability : Any outbound connection from any zone should hide its internal addresses.
• Factory defaults:
• 5 zones - DMZ for front end servers, PCI for payment systems, CORP for internal users, VISITOR for visitors and MGMT for management traffic.
• ACL rules:
• Allow only DMZ to PCI traffic. Deny PCI to ALL zone traffic.
• Allow only HTTP and HTTS traffic from ANY to DMZ.
• Allow only Enterprise protocol from CORP to ANY and VISITOR to ANY.

Expectation from Network based Intrusion Prevention Systems:

• Ability to protect all services of machines in corporate network, card holder data zone, DMZ zone from attacks exploiting known vulnerabilities: IPS devices are expected to have up-to-date signatures to protect internal resources from exploiting known vulnerabilities.
  
• Ability to differentiate from patched and un-patched services in internal machines: IPS devices are expected to have facility to control generation of logs based on targeted system version.
  
• Ability to detect and stop connections and packets transporting malware to internal network: IPS devices are expected to have up-to-date signatures to detect malware by monitoring the connections and packets.
  
• Ability to detect infected machines in internal zones by monitoring the traffic and connections from the internal machines to other zones: This typically happens if an infected system was brought into the network. IPS devices are expected to detect these machines by monitoring the traffic using up-to-date signatures.
  
• Ability to provide signature additions and editions: Beyond signatures which are auto updated, IPS systems are expected to provide facilities to create custom signature rules. It facilitates the administrator to create rules which are specific to the deployment. Administrator can create rules to detect default passwords of the applications and other characteristics of applications. Also administrator can create rules to detect specific information about cardholders.
  
• Ability to detect protocol anomalies of HTTP.
  
• Ability to detect traffic anomaly: Especially this is required to detect anomaly between frontend servers and cardholder data servers. Administrator can setup normal traffic profile for day and night and check for any anomaly. IPS deviecs are expected to provide this function.
• Ability to detect and prevent from SQL, XSS injections: IPS devices are expected to monitor HTTP POST values, GET values and XML data to detect SQL or XSS injections and check for any blind SQL and XSS injections using its up-to-date signatures.
• Ability to generate logs and alerts: IPS devies are expected to generate logs with comprehensive information for administrators to analyze the issue and take corrective actions.

Expectation from IPSec VPN:

• Ability to create Site-to-Site tunnels: Yet times, frontend servers and cardholder data servers may not be in same location. In these cases, it is expected that Ipsec VPN tunnels are used to transport data among these servers to provide data security on wire.
• Ability for Remote Access VPN: Administrators and other privileged users may need to get to the cardholder data servers remotely. IPSec VPN component of security appliance is expected to allow remote access via VPN tunnel.
• Ability to create new key generation periodically ( as small as 5 minutes).
• Ability to create IKE policies without preshared keys. When preshared keys are used, IPsec VPN should check for the strengh of the preshared keys.
• Atleast one digit.
• Atleast one non-alphabet and digit.
• Minimum length of 6 characters.
  
• Ability to support EAP authentication in remote Access VPN cases.
  

Expectation from Network based AntiVirus:

• Ability to scan email/HTTP traffic for virus and stop virus reaching internal machines: Internal machines are eventually used to access cardholder servers. Network based Virus scanning is one way to stop viruses reaching the internal servers and machines. Note that, network based AV can only do limited job. It is required that host based AV software is installed on all endpoints and servers.

Expectation from Web Application Firewalls:

• Ability to look into SSL traffic: All transactions are based on SSL. Any deep inspection requires decoding SSL traffic.
• Ability to provide HTTP protocol anomaly detection
• Ability to detect exploit traffic
• Ability to detect SQL and XSS injections.
• Ability to provide access control for different services running on HTTP.
• Ability to look deep into XML for SQL, XSS, command injections.
  
• Ability to generate log events and alerts.

Since these security devices are part of merchant network, these also must adhere to generic requirements of PCI DSS. They are:

• Configuration management of these devices should only be allowed by privileged users.
• Auditing of configuration changes: If there are multiple administrators, then each administrators must have unique ID. Each change in the configuration must be logged with information such as 'name of user' , ' change made', 'time & date of change' etc..
• Security devices also can be vulnerable. It must have facilities to upgrade new firmware.
• Mandate change of default passwords first time administrators log in, if local user database is used.
• Check for strong passwords, if local database is used.
• 6 Character length.
• Atleast one digit
• Atleast one non-alphanumeric character.
  
• Security devices predominantly should show alert message in their home page of default passwords.
• Mandate the password change if unchanged for a month or as per company policy.
  

Trace Route Diagnostics - TR-069 profile

Tracing the routing path to a given destination machine is very important diagnostic tool for administrators and service providers. TR-098 Amendment 1defined 'IP PING' diagnostics, but did not define trace route diagnostics.

Trace route utility is provided in many operating systems today. Its main purpose is to find out the route IP packets take to reach a specific destination. It gives indication on the routers in between and round trip time of each probe. This utility sends UDP, TCP or ICMP probes with small TTL and listen for ICMP 'time exceeded' reply. It starts with TTL 1 to find out the first hop, TTL 2 to find out the second hop and so on.

Trace route can happen by sending ICMP, UDP or TCP packets. Trace route utility provides this option for traversing through firewalls. When firewalls don't allow ICMP, trace route can be used with UDP or TCP.

Since, a given host given may have multiple outbound interfaces, trace route utility gives facility to use specific link for its source IP address to ensure that responses come back to this IP address. It also provides option of specifying the gateway to route the packets through a specific link of the host.

Trace route utility provides options such as:

• Destination Host IP address or FQDN.
  
• Use ICMP, UDP or TCP.
• Use IPv4 or IPv6
  
• Packet size : Valid in case of ICMP and UDP. Not applicable if TCP is chosen. TCP probe always goes with SYN flag.
• Port: Port number to use for destination port of the UDP or TCP probe. In case of ICMP, it is used as sequence number.
• TOS: TOS value to use in the IP header of probe packet.
  
• Link interface to use: Utility uses IP address of this link as source IP of the probe packet. if not specified, default is to use link determined by route.
• Gateway IP address: This is mainly to select the right link, if there are multiple links on the host. Default is 'based on route'.
  
• Maximum TTL: Indicates the maximum number of hops to discover. Default : 30
  
• Number of Probes: Number of probe messages to each hop. Default 3.
• Wait time: wait time to wait for the response to probe. Default : 5 seconds.
  
• Send interval: Interval between probe messages to a hop: Default 0 seconds.
  

Results of trace route utility can be represented as:

• Destination Host: For which trace route diagnostics was run.
  
• Number of hops.
  
• Sequence of hops. Each hop consisting of
  
• IP address of router: Display * if no response from that hop.
• FQDN of the router: By doing Reverse DNS lookup
  
• Probe1 round trip time in milli seconds.
• Probe2 round trip time in milli seconds.
• Proble3 round trip time in milli seconds.
• Rest of round trip times in comma separated string (upto 32 bytes).

With this TR-069 profile could be as follows:

• internetGatewayDevice.IPTraceRouteDiagnostics P
• diagnosticsState: RW, String, It takes values of "None", "Requested", "Completed", "Error_HostNameResolutionError", "Error_HopCuntExhausted": Similar to PING diagnostics stage.
  
• destination host : RW, String, 256 bytes max, IP address in dotted decimal form or fully qualified domain name.
• probeProtocolSupported: R, String, comma separated strings. "ICMP", "UDP", "TCP", this is the capability of device.
  
• probeProtocol: RW, String, Takes one of values of "ICMP", "UDP" or "ICMP".
  
• IPv4OrIPv6 : RW, String, Values are "IPv4", "IPv6"
• packetSize: RW, Integer, Not applicable in case of TCP.
• Port: RW, Integer, Destination Port to use. In case of ICMP, it is used as sequence number.
• TOS: RW, Integer
• LinkInterface: RW, String, Fully qualified instance from VLAN, LANDevice, WANPPPConnection or WANIPConnection etc.. IP address of this interface is used as source IP of the probe packet.
  
• gatewayIPAddress: RW, String, Dotted decimal form. It is to select the link.
• maximumHops: RW, Integer
• numberOfProbles: RW, Integer, Number of probes to use for each hop.
• responseWaitTime: RW, Integer, in seconds.
• sendInterval RW, Integer, in seconds.
• internetGatewayDevice.IPTraceRouteDiagnostic.response P
• hostName
• status: "Error_UresolvedHost", "Error_MaxHopCountReached", "Success"
  
• numberOfHopEntriesDiscovered: RW, Integer
• hopIPAddress: RW, String
• hopFQDN: RW, String
  
• probe1RTT: RW, Integer in milliseconds.
  
• proble2RTT: RW, Integer
• proble3RTT: RW, Integer
• OtherProbeRTT: RW, string, comma separated. Upto 32 bytes.

Mobiles - Why Document Management System.

See this article on Risk Management techniques.

Some excerpts from this article:

In years of consulting with Wall Street clients, John Pironti, a board member of ITGI (IT Governance Institute) and chief strategist for Getronics (www
.getronics.com), has learned that some of these security-obsessed financial giants lose as many as 10 BlackBerrys per day. A 2005 survey by CheckPoint Software (www.check
point.com) found more than 25,000 PDAs and laptops left in Chicago taxis over a six-month period. Many have unsecured, confidential data that, if in the wrong hands, could have significant adverse ramifications�a third of CheckPoint’s respondents don’t use passwords or any other security protection on their mobile devices.

25000 PDAs and laptops in one city over 6 months is too high a number. It means that confidential data is not so confidential anymore.

As I mentioned in my previous blog entries, one of best ways is a way where the information does not leave Enterprise. That is where I feel that document management systems should come in very useful. It should provide facilities for access, modify existing documents and create new documents. Of course, this facility is to be provided only if the user is privileged to do the operations. Since mobiles are becoming versatile, a good document management system must provide thick java based application on mobiles for accessing document management system in easy-to-use manner. It also need to provide facility to do off-line work on a specific documents if they are allowed to be downloaded. Also, these downloaded documents must be stored locally in mobiles with strong encryption - Either this could be application based or this can be done using encrypted file systems on mobiles.

If I am IT professional, I look for this minimal feature set in document management systems:

• Easy-to-create work spaces.
• User Group management.
  
• Association of Users to User Groups ( Could be done in LDAP or RADIUS Servers)
• Creation of new documents, Modification of existing documents with history maintained.
• Modification only after checkout with lock.
• Releasing lock upon check-in.
  
• Task Management.
• Work flow management.
• Access control
  
• Read permission on work space and document.
• Write permissions on modifying on per item in workspace.
• Create new documents.
• Allow Synchronization with mobiles.
  
• Mobile segment related features:
• Thick Client on Mobiles.
• User authentication (beyond authentication required to unlock the mobile).
• Easy-to-use UI
  
• Both online and offline editing should be possible.
• Synchronization of document data between Mobile application and backend document management system using OMA-DS.
  
• Encrypted storage.

Referencing - ACS responsibility

TR-069 based data models can become complex with referencing. Some examples of references are:

• internetGatewayDevice.Layer2Bridging.Filter.{i}.FilterInterface takes value of 'AvailableInterfaceKey' of table internetGatewayDevice.Layer2Briding.AvailableInterface.{i}
• internetGatewayDevice.Layer2Bridging.Marking.{i}.MarkingInterface takes value of 'AvailableInterfaceKey' of table internetGatewayDevice.Layer2Briding.AvailableInterface.{i}
• internetGatewayDevice.QueueManagement.Classification.{i}.ClassInterface takes one of the values from WANPPPConnection instances, WANIPConnection instances, Ethernet interface instances of LANDevices, USB interface instances of LANDevice, Wireless LAN interfaces instances of LANDevices or VLAN interface instances.
  

ACS implementation considerations:

• Must to have for interoperability:
  
• ACS must send referred instances first before sending referencing instances to the device. That is, if WANPPPConnection instance is referred by QueueManagement.Classification table, then WANPPPConnection instance creation and configuration must happen before Classification table instance is created and configured. Device may fail to accept the configuration if Classification instance is configured before configuring the referred WANPPPConnection. Hence, ACS must keep this ordering in mind while sending configuration to the devices.
• Note: Data model standard developers must never do the bidirectional cross referencing between two tables.
  
• ACS must not allow administrators to remove instances if they are being referred by others. Going by above example, ACS must not allow removal of WANPPPConnection instance if it is being referred by Classification instance. It should allow this operation only if WANPPPConnection instance reference is removed from the classification instance. Yet times, same instance is being referred from multiple places. Unless all the references are removed, it should not allow removal.
  
• Nice to have features
• ACS may display all possible places where a particular instance object is being referenced. For example, all references of WANPPPConnection instance can be displayed to the administrator so that he/she can remove references.
  

Mobile battery power drain attack - What to do about it?

I recently came across this post http://isc.sans.org/diary.html?storyid=4150&rss

ISC SANS article and paper mentioned in the article named Battery-Based Intrusion Detection describes the intrusion detection using batter power usage in mobiles. It appears that there is significant amount of power consumption on packet processing even though there is no application listening on those packets. I guess this is due to reception and transmission on wireless interface and amount of processing that happens in the TCP/IP stack of mobile before the packets are dropped (power consumption by CPU). Mobiles supporting 802.11x wireless interface are more vulnerable to this attack due

Many mobiles are equipped with power management facilities in both mobiles and 802.11 wireless access points. It works as expected as long as there is no unintended packets are sent to the mobile. Attacks can be mounted by sending flood traffic to the IP address assigned to mobiles, there by draining the mobiles.

What are the precautions wireless infrastructure deployment can take:

• Stop any traffic that was not in response to connections that were made by the mobiles.
• Don't allow any unsolicited connections to the mobiles.
• If there is need for TCP/IP connections to the mobiles, ensure that they are allowed only based on user credentials.
  
• Don't allow broadcast packets by default.
  
• Yet times, mobiles may have VPN connection to the Enterprise security gateways. This security gateway should ensure same precautions.

To achieve above, stateful firewall must be either part of wireless access points or wireless switch. In case of VPN tunnels, VPN server also should have stateful firewall. One should ensure that stateful firewalls also support activation of user based policies upon successful authentication.

Microsoft System Center - Mobile Device Management - Missing Security pieces

Microsoft has well written product reference guide. Please see it here.

It is a comprehensive Mobile Device Manager for Enterprises to manage and control mobiles by their own IT department. It differs from other solutions which are controlled by providers.

It is very obvious that Enterprises would allow mobiles to access their networks for many business applications. Enterprises have to be geared up to provide this access without compromising security. Microsoft solution is taking care of data security on the air. It is not sufficient from threat security perspective. A complete mobile solution require network security processing functions in addition to solution provided by MDM.

MDM seems to be providing following functions using different components:

• Mobile Device enrollment : Simple facility to enroll mobile devices into corporate network.
• Mobile Device Management : Managing mobile devices by providing sub functions such as
• Over-the-Air application management
• Upload new applications.
• Configure applications.
• Inventory Reporting.
• Give complete list of devices that are in pre-enrolled stage, enrolled stage, device characteristics, installed applications, Some application settings etc..
• Role based administration
• Distribution of work among IT professionals to handle large number of mobile devices, applications and its functions.
• Self Help Portal
• For employees to view the list of their mobiles _ note that each employee can have more than one mobile and its inventory of applications etc..
  
• Remote Wipe upon loss of mobile.
• Basically, it helps reduce burden on IT professionals.
• Security Management: Basically, it helps IT professionals and Employees to control the settings on the mobile - Such as Cameras, Blue tooth connectivity, Email settings, SMS/MMS messaging etc..
  
• Mobile VPN Gateway
• To provide secure data access to corporate network from mobiles.
• It provides persistent tunnel capability, Roaming capability.

Additional Functions that are required by Enterprises to provide mobile connectivity to their networks are:

• Granular access controls based on type of user and mobile device.
  
• There are many LOB Servers (Microsoft term for Application servers) and network resources. Some may not be accessible by all employees. Having access control on a Enterprise side provides better control for security professionals.
• If the mobile device is stolen, any traffic coming from that mobile device must be stopped. Again, having control at the Enterprise level provides this capability.
  
• Network traffic scanning for infected mobiles
• Mobiles are now general purpose computers. They also can get infected by attackers as normal laptops. They can create havoc, if these are not detected and stopped immediately. It might infect servers and other network resources. Security professionals would like to have a control on network side to detect and prevent any access upon infection. This kind of security is required even if mobiles have security software installed in them.
• Anti Virus scanning and Anti Spam functions
• These are highly recommended to be on the Network side too.

In summary, mobile deployment by Enterprises must consider network security functions as part of their overall solution.

Network objects in Security infrastructure products

Security policies are complicated. It consists of many rules. Rules typically contain IP address and Service information along with other information related to particular security function. Some security products only support creation of rules with immediate values for IP addresses and services. Thought it is not a problem during creation of rules, it could be a maintenance headache when it comes to modifications due to network changes. Also, it is difficult for new administrators to understand the rules. Good security products support creation of network objects and associate them to the rules. Admins need to modify only network objects if there is any change in the network. Since network objects are named, rules are readable and easy to debug and fix any problems.

Typical network objects are IP address objects and Service Objects.

IP Objects:
IP address objects name the hosts, subnets, range of hosts. For example, all networks in Engineering department can be grouped into one object. Each server can be represented by separate object with descriptive names.

Service Objects name different kinds of services. For example TCP port 80 can be named as 'http' service object.

IMO, IP object definition should take any type of IP address(es) - Subnet, Range, Single IP address, FQDN. It should also take multiple of these in one record. With this in mind, IP object definition can be represented as

• Virtual Instance ID ( See http://srini-addepalli.blogspot.com/2008/03/virtualization-and-zones-in-secuirty.html )
  
• Object name
• Object Description
• IP address Type - Whether this definition for single IP address, IP address range or Subnet
• Multiple IP definitions. With each definition having
• IP address ( if type is single IP address)
• IP address begin (if type is IP address range)
• IP address end (if type is IP address range)
• Network (If IP address is Subnet)
• network mask (If IP address is Subnet)

IPsec policy records when implemented with IKEv1 can't have FQDN as part of selectors. Due to this, IP address objects having FQDN can't be used to represent selectors. Many security software implementations have speciall IP address objects which take only FQDN. These type of IP address objects have following parameters:

• Virtual Instance ID
  
• Name
• Description
• Multiple IP definitions. With each definition having
• FQDN

First kind of IP objects are called IPValueObjects and second kind of IP objects are called IPFQDNObjects.

Service Objects
Service objects represent services. Each Service objects contains following attributes.

• Virtual Instance ID (See http://srini-addepalli.blogspot.com/2008/03/virtualization-and-zones-in-secuirty.html )
  
• Name
• Description
• Multiple definitions of Ports. Each definition having
• Protocol Begin (It takes values of protocol values such as UDP, TCP etc..)
  
• Protocol End
• Port Range Valid (YES or NO) : This is Not Applicable if Protocol range does not have either TCP or UDP)
  
• Port Begin (If TCP or UDP is present in protocol range)
  
• Port End (If TCP or UDP is present in protocol range).
• ICMP type Valid (YES or NO): This is not applicable if protocol range does not have ICMP.
  
• ICMP type begin
• ICMP type end
  
• ICMP Code Begin
  
• ICMP Code End

In addition to objects created by administrators, there could be objects that are created by other applications in the system. These objects are dynamic and can't be deleted by administrators. It is good to show them in UI though.

Based on above explanation, I tried to give possible TR-069 data model:

• internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects P
• internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.IPValueObject.{i} PC
• ObjectName : RW, String of 32 characters. Once the object is created, this can't be changed.
• Enable : 1 - Yes 0 - No
  
• Description: RW, string of 64 characters.
• IPAddressType: RW, Integer Value (Takes values 0 - single IP, 1 - IP address range, 2 - IP subnet )
  
• internetGatewayDevice.security.VirtualInstance.{i} .NetworkObjects.IPValueObject.{i}.IPdefinition.{i} PC
• SingleIPValue: RW, String, IP address in dotted decimal. Applicable only if IPAddressType value is 0.
  
• IPRangeMinValue, IPRangeMaxValue: RW, String. Values are IP addresses in dotted decimal. Applicable only if IPAddressType is 1.
• IPSubnet, IPSubnetMask : RW, string. Values are IP addresses in dotted decimal. Applicable only if IPAddressType is 2.
• internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.DynIPValueObject.{i} P
• ObjectName : Read Only, String of 32 characters.
• Enable : 1 - Yes 0 - No Read Only
  
• Description: Read Only, string of 64 characters.
• IPAddressType: Read Only, Integer Value (Takes values 0 - single IP, 1 - IP address range, 2 - IP subnet )
  
• internetGatewayDevice.security.VirtualInstance.{i} .NetworkObjects.DynIPValueObject.{i}.IPdefinition.{i} P
• SingleIPValue: Read Only, String, IP address in dotted decimal. Applicable only if IPAddressType value is 0.
  
• IPRangeMinValue, IPRangeMaxValue: Read Only, String. Values are IP addresses in dotted decimal. Applicable only if IPAddressType is 1.
• IPSubnet, IPSubnetMask : Read Only, string. Values are IP addresses in dotted decimal. Applicable only if IPAddressType is 2.
• internetGatewayDevice.security.VirtualInstance.{i} .NetworkObjects.IPFQDNObject.{i} PC
• ObjectName : RW, String of 32 characters. Once the object is created, this can't be changed.
• Enable : 1- Yes 0 - No
  
• Description: RW, string of 64 characters.
• internetGatewayDevice.securityDomains.VirtualInstance.{i} .NetworkObjects.IPFQDNObject.{i}.IPdefinition.{i} PC
• IPFQDN: RW, string of max size 256.
• internetGatewayDevice.security.VirtualInstance.{i} .NetworkObjects.DynIPFQDNObject.{i} P
• ObjectName : Read Only, String of 32 characters.
  
• Enable : 1- Yes 0 - No Read Only
  
• Description: Read Only, string of 64 characters.
• internetGatewayDevice.security.VirtualInstance.{i} .NetworkObjects.DynIPFQDNObject.{i}.IPdefinition.{i} P
• IPFQDN: Read Only, string of max size 256.
• internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.ServiceObject.{i} PC
• ObjectName : RW, String (32 bytes). Can't be changed once the object is created with a value.
  
• ObjectDescription: RW, String (64bytes)
• Enable 1 - Yes 0 - No
  
• internetGatewayDevice.security.VirtualInstance.{i} .NetworkObjects.ServiceObject.{i}.definition.{i}
• ProtocolRangeMin: RW, Integer (0-255).
  
• ProtocolRangeMax: RW, Integer (0-255)
• PortRangeValid: 1 - Yes 0 - No Also this is applicable only if TCP or UDP is present in the protocol range.
• PortRangeMin: RW, Integer (0-65535)
  
• PortRangeMax: RW, Integer (0-65535)
• ICMPRangeValid: 1 - yes 0 - No. Also this field is applicable only if protocol range has ICMP.
• ICMPTypeRangeMin: RW, Integer (0-255)
• ICMPTypeRangeMax: RW, Integer (0-255)
• ICMPCodeRangeMin: RW, Integer (0-255)
• ICMPCodeRangeMax: RW, Integer (0-255)
• internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.DynServiceObject.{i} P
• ObjectName : Read Only, String (32 bytes).
  
• ObjectDescription: Read Only, String (64 bytes)
• Enable 1 - Yes 0 - No Read Only
  
• internetGatewayDevice.security.VirtualInstance.{i} .NetworkObjects.DynServiceObject.{i}.definition.{i}
• ProtocolRangeMin: Read Only, Integer (0-255).
  
• ProtocolRangeMax: Read Only, Integer (0-255)
• PortRangeValid: 1 - Yes 0 - No Also this is applicable only if TCP or UDP is present in the protocol range. Read Only
  
• PortRangeMin: Read Only, Integer (0-65535)
  
• PortRangeMax: Read Only, Integer (0-65535)
• ICMPRangeValid: 1 - yes 0 - No. Read Only. Also this field is applicable only if protocol range has ICMP.
• ICMPTypeRangeMin: Read Only, Integer (0-255)
• ICMPTypeRangeMax: Read Only, Integer (0-255)
• ICMPCodeRangeMin: Read Only, Integer (0-255)
• ICMPCodeRangeMax: Read Only, Integer (0-255)

Virtualization and Zones in Secuirty Appliances - A brief

I see this question being asked frequently. There is confusion between virtual instance Versus security zone. This is my small attempt to give some brief on virtualization in security appliances. It is not meant to describe this feature in detail :-).

For a brief on Security Zones, please see this entry:
http://srini-addepalli.blogspot.com/2008/02/mutliple-security-zones-in-enterprise.html

Virtual Instances:
Newer Security appliances have this feature whereby one single hardware box supports multiple security instances. They are called virtual instances. Each virtual instance has its own configuration of security functions. Some security vendors call these instances as VSGs (Virtual Security Gateways). Note that, the term Virtual is not same as the terms used by Vmware and Xen. Each virtual instance has its own security zones, firewall policy configuration, IPsec SPD, IPS configuration and so on. For all practical purposes, you can assume that multiple traditional security appliances are kept in one single box with common user interface.

As you understand by now, security zones are part of each virtual instance.

Link layer Interfaces (Ethernet, Wireless etc..):
Each virtual instance has its own interfaces. A given interface can't be part of two virtual instances. Again, imagine multiple traditional physical security appliances. Each traditional appliance has its own Ethernet ports and wireless ports. Similarly, each virtual instances has its own interfaces. If the security appliance is supporting, say 16 virtual instances with each instance supporting 4 security zones, you require 16*4 = 64 link interfaces. This number goes up very high if more virtual instances or more security zones are required. To reduce the hardware cost of adding these many physical ports, vendors support VLAN based interfaces for virtual instances. 4K VLAN interfaces can be created on each physical port. Virtual instances (VSGs) can make use of VLAN interfaces to provide virtualization with less number of physical ports. In some cases, it is not possible to have VLAN ID to map to virtual instance. One example of this type is 'packets coming from Internet' in data center environment. In these case, virtual instance needs to be identified by the destination IP address of the packet. In these cases, the IP address in 'destination IP address feild' of IP header is used to identify interfaces.

Link interfaces are assigned to each Virtual instance and Security Zone. Interface on which packets are being received is used to determine the virtual instance and security zone.

Virtual Security Gateway feature, in my view, is mainly useful for service providers and data centers who want provide security services for business customers. Each customer is treated as one VSG. With one security appliance, SPs can provide security services for many customers, thereby reducing cost dramatically.

With this background, let me describe the relation between VLAN, VSG and Security Zones by taking an example. Let us assume that a 'data center' is hosting services for 64 customers. Data center provider wants to deploy Firewall/IPS appliances to protect servers of these 64 customers. Let us also assume that the appliance has two physical ports. One physical port is connected 'Server' network (Call it as 'Server Port) and another port is connected to untrusted network (Internet Port). With this scenario Admin of this box need to do following:

• Install 64 port switch - Connect Uplink port to security appliance's Server port.
  
• Configure the switch to add VLAN ID 1 to VLAN 64 to packets coming from 64 ports and redirect them to uplink port (VLAN switching).
  
• Configure VLAN module of appliance to create 64 interfaces on VLANA1 to VLANA64.
• Install another 64 port swtich - Connect Uplink port to security appliance's Internet Port.
• Create VLANB1 to VLANB64
  
• For all 64 customers
• Associate VLANAx to 'Server Zone' and VSGx
• Associate VLANBx to 'Internet Zone' and VSGx
• Add security service configuration for each Virtual instance.
• That is all is required I guess.

What it the configuration support required for this?

• VLAN Interface configuration. I covered this in before as part of this.
• Virtual instance administration
• It is set of records and each record having
• Virtual InstanceID.
  
• Virtual Instance name.
• Enable/Disable
• Description
• Zone Description
• Zone ID
• Zone name
• Interface mapping
• Set of records. Each record having
• Interface name(Could be VLAN interface name, bridge interface name or WAN interface name etc.. )
• VSG ID
• Zone ID

Enterprise security appliances typically don't need to support virtualization and hence it can be considered to have one virtual instance.

With above back ground, I feel that TR-069 model may look like this:

• internetGatewayDevice.security P
• maxZonesPerVirtualInstance (Read Only)
• maxVirtualInstances (Read Only)
• currentNumberOfVirtualInstances
  
• internetGatewayDevice.securityDomains.VirtualInstance.{i} PC
• ID
• Enable
  
• Name
• Description
• internetGatewayDevice.security.VirtualInstance.{i}.Zone.{i} P
  
• Zone ID
  
• ZoneName
• internetGatewayDevice.mapping.{i} PC
• InterfaceReference (Fully Qualified instance from VLAN table, LANDevice and WAN Link).
• VirtualInstanceID
• ZoneID

IPS Signature development - Tips

IPS/IDS signature development is a complicated process. You don't have information at one place to gather and analyze before developing a signatures for detecting intrusion attempts. Challenges are:

• Non availability of right information at one place.
• Getting exploit scripts is not easy.
  
• Not having access to vulnerable application to verify the signature.

Some of the sites which I used to visit and guidelines I followed while developing signatures for IDS/IPS subsystems:

• Subscribe for Vulnerability notifications
• Many organizations provide this information such as CERT.
• Knowing more technical information about vulnerability information
• Check for exploit scripts
  
• visit sites such as milw0rm and secwatch
  
• Check vulnerability database in securityfocus
  
• Monitor mailing lists such as bugtraq and full-disclosure
• Monitor bleeding threats signatures for snort. ( I think now it is called as Emerging threats)
• Yet times, you may need to glee information from several resources.
  
• Yet times, assembly analysis of vulnerable code is needed to write signatures which produces less falses.
  
• Another very valuable source I find is SANS daily blog.
  
• If exploit is available, understand the vulnerability by going through the code
• Develop signature targeting the vulnerability.
• For XSS injection attacks, I used to monitor sites such as
• XSSDB at gnucitizen.org
• XSS Cheat sheet at http://www.gnucitizen.org/xssdb/
• XSS incident database.
• SQL Injection attacks: I used monitor following sites for changes
  
• SQL injection cheat sheet
• SQL injection incident database at web application security consortium.
• For P2P/IM application detection:
• Run the application, capture traffic and figure out the pattern.
• If application is available in source code from in public domain, try understanding the protocol at some details and figure out the patterns.
• Using above, develop signatures.
• Understand the protocol.
• You need to have complete understanding of IPS device capabilities with respect to rule language.
  
• Ofcourse, you need to understand vulnerability based on above information.
• You need to have good understanding of regular expressions.
  
• Testing signatures:
• Note that it is not possible to have all kinds of application packages for any IPS vendor to test the signatures.
  
• If exploit is available, develop simulation script and ensure that IPS/IDS between detects and stops the attack.
• If exploit is not available, develop simulation scripts based on information available and test the signatures.
  
• Use third party penetration tools to verify signature quality periodically.
  

As such there is no fixed methodology, but hopefully above guidelines are useful to jump start yourself.

IPS - Linux User space, Kernel space - Where should it run?

Just a thought...

Advantages of user space:

• Easier to debug : Advantageous to developers. Note that IPS is quite complex.
  
• Faster to develop new functionalities.
• No limitation on stack size. Stack size grows with its usage.
  
• Many debugging tools available for user space applications.
  
• Swap memory can be used - So, almost all memory in the system can be used. In case of kernel space, only limited amount of memory can be used.
• Many libraries available.
  
• Upon any crash due to software, faster to bring up.

Advantages of kernel space:

• Faster packet processing :
  
• No threading overhead.
• Save on buffer copy overhead.
• Straight access to TCP/IP stack functions such as routing functions and bridging functions.
  
• No or very minimal jitter when run in softirq/tasklet context. - Very useful for VOIP traffic.
• Real time characteristics of the traffic can be maintained.
  
• Physical memory access.
• Easily portable to Real time operating systems as not much of OS capabilities are used - Advantage for vendors who need to have flexibility of operating systems.

As a developer, I prefer doing this in user space entirely. But kernel space provides its advantages, mainly on performance and low jitter. IMO, combination of user space and kernel space is needed. If it onlyL4 inspection is required for a session, then it is better handled completely in kernel space. If protocol inspection beyond L4 is required such as SIP and HTTP protocol analysis, then it is better done in user space. I prefer this kind of hybrid approach.

TR-069 Dynamic DNS (DDNS)

Let me give some introduction on DDNS before giving some ideas on DDNS data model.

Many small offices would not want to pay for static public IP addresses and yet host servers for public access. For public access of servers, it is necessary that reachability information is constant. DDNS functionality enables this. DDNS providers facilitate this by allowing CPE devices to advertise IP address changes to DDNS providers. DDNS providers internally update their DNS servers with this new IP address.

DynDNS is a popular protocol used by many DDNS providers. Though it is not a defined by IETF or any other standard body, this seems to be quite popular. DDNS providers such as www.dyndns.org and 3322.org use this protocol to provide dynamic DNS service. TZO is another protocol used to update the service. DDNS provider www.tzo.com provides DDNS Service using this protocol.

As an end user, following steps are to be followed to get hooked into this service:

• Create account with DDNS provider using their web site. You may be asked to provide your email address, user name and password.
  
• Register your domain names with the DDNS provider by visiting their web site or by using one of their clients.
• Configure your router with DDNS provider information to update IP address automatically.
• Set up your internal servers for public access.
  
• Configure your CPE router to forward the traffic to your servers.

CPE router configuration: CPE routers normally support multiple instances of DDNS - One instance for each WAN link. For example, If there are two WAN links, then CPE devices will have two DDNS client instances. Each client instance can update the IP address for multiple domain names. Each record would need to have following information configured.

• Name of the record : To identify the record.
• WAN interface (link): DDNS client monitors the IP address of this link. Wheneverthe IP address of the link is changed, then it starts the process of updating.
  
• Update time period: This time period indicates the periodic interval to update the IP address, even if the link address is not changed. This configuration parameter is not used by some DDNS protocols such as 'dyndns'.
  
• Domain name 1 : Name of the domain name that was registered.
  
• Domain name2
• Domain name3
• Domain name4
• DDNS protocol to be used: Protocol that is to be used to update IP address. 'dyndns', 'tzo', 'dhrp' etc..
• Provider details
• In case of dyndns, the additional configuration required to contact DDNS provider are:
• DDNS provider IP address or FQDN: Reachability information to reach DDNS provider.
• Relative URL (Script name): DynDNS protocol is HTTP or HTTPS based. This parameter indicates the CGI script to be used to send update information. Unfortunately, this name is not standardized. Different providers are using different script names. Hence, this should be taken as configuration parameter.
• Protocol : HTTP or HTTPS
• Port : Typically, it is 80 or 443.
  
• User name and password: User name and password used to create account with DDNS provider.
• Trusted Certificate in PEM form: When HTTPS used. This certificate is used to authenticate the DDNS provider to avoid MITM attacks.
  
• Note: DDNS client always should send MX=NOCHG and wildcard = NOCHG to ensure that configuration done using DDNS provider website is not erased.
• TZO provider information: As I understand, TZO protocol defines server discovery. I guess this is mainly for load balancing. It involves following stages - Getting IP addresses of clusters and getting IP addresses of update servers and then updating IP address to one of the update servers.
  
• ProviderIP address/host name (Cluster lookup host): Using this IPaddress/host name, the TZO client in the CPE gets the all IP addresses of cluster. TZO client makes a TCP connection for this purpose.
  
• Cluster lookup port: Port used by TCP connection to do cluster lookup. Default : 21340
  
• Update Server lookup Port: Once TZO client gets the cluster IP addresses, it is expected to choose one of them and make another TCP connection to get the list of servers that can be used to update the new dynamic public IP address. This port is used by client to make the connection. Default : 21344
  
• Email address: Mail address used when the TZO account was created.
  
• Key: Key generated when the TZO account was created.

Based on above explanation, the TR-069 data model for DDNS could be:

• internetGatewayDevice.DynamicDNS.{i} : PC - New instances can be created.
  
• Name
  
• Enable
• LinkReference : Fully Qualified name of WanDevice->WanConnectionDevice->WanIPConnection or WANPPPConnection.
• UpdateTimePeriod : In seconds.
• DomainName1
• DomainName2
• DomainName3
• DomainName4
• DdnsProtocol : Takes one of the values of 'dyndns', 'tzo'
• internetGatewayDevice.DynamicDNS.{i}.dyndns
• ProviderFQDN
• ProviderURLScript
• ProviderProtocolSupports: Read Only, String, It takes comma separated protocol strings such as http, https etc..
  
• ProviderProtocol : take 'http' or 'https'
• ProviderPort
• Username
• Password
• ProviderCACertificate : Valid only if 'https'
  
• internetGatewayDevice.DynamicDNS.{i}.tzo
• ProviderClusterLookupFQDN
• ProviderClusterLookupPort
• ProviderUpdateServerLookupPort
• RegisteredEmailAddress
• KeyProvided

Why is IKEv2 better than IKEv1?

IMO, any new deployment should go with IKEv2. It provides several advantages over IKEv1.

• IKEv2 is light on bandwidth and faster
• Less number of messages to establish tunnel.
• IKEv2 provides inbuilt NAT Traversal.
• IKEv1 does not provide this facility. But an internet draft was created to enhance IKEv1 with this functionality. Since this draft is not standardized, there may be interoperability issues.
• IKEv2 has inbuilt tunnel liveness checks.
• If tunnel is broken down on peer, it has facility to detect and re-establish the tunnel.
• IKEv1 does not have this functionality. There is an internet draft available though.
• IKEv2 provides comprehensive authentication capabilities.
• It supports Pre-shared key authentication, certificate authentication. IKEv1 also has them.
  
• More importantly, it provides EAP authentication and hence it is suitable to integrate with existing authentication systems in Enterprises. IKEv1 does not have this capability.
• IKEv2 has companion document to work with changing IP addresses on devices .
• MOBIKE standard is only supported on IKEv2.
• IKEv2 has facility to negotiate multiple sets of selectors.
• Many networks/ranges can be negotiated in one exchange. Hence, number of policy records can be very less when sites have multiple networks.
• In IKEv1, each pair of networks need to be defined in one policy record in SPD.
• IKEv2 has clear method to choose subset of selectors when both sites are not configured with exact selector values.
  
• In case of mismatch, IKEv2 has better mechanisms to converge.

If you are newly deploying IPsec gateways or thinking of upgrading Ipsec based security gateways, consider using IKEv2.

IPsec and DDNS - Where do they meet?

If some one tells you that you require permanent public IP address for connecting offices via IPsec tunnel, then he/she is not up-to-date with the latest offerings.

It is true that original IPsec security gateways do require static public IP address for establishing tunnel. With Dynamic DNS capability built into these gateways, this is no longer the case. As long as there is public IP address (static or dynamic), IPsec tunnel establishment is possible. Both sites participating in the tunnel can have dynamic public IP addresses.

What is Dynamic public IP address?

IP address assigned by ISP is not constant. ISPs can change the IP address that is being assigned from time to time. More often, every time CPE device reconnects (in case of PPP) or require new lease (in case of DHCP), ISPs provide new IP address.

What dynamic DNS?

Dynamic DNS provides fixed domain name, but changing IP address. There are large number of providers (One example: www.dyndns.org) provide services to register for domain name and change the IP address for this domain name through proprietary but well published protocols. CPE device, whenever public IP address changes informs dynamic DNS providers through the protocol. Any peer who needs to communicate with the CPE device will get the latest IP address via DNS protocol. Only limitation is that all peers must communicate with the CPE using domain name.

IPsec and DDNS:

IPsec as an application can make use of domain name to communicate with the peer. IPsec component must have domain name resolution facility to get IP address of peer before connecting to it via IKE. IPsec tunnel establishment can happen even if both the sites have dynamic public IP addresses as long as they have DynDNS capability.

With gateway adoption feature as described in one of previous blog entries, tunnel need not get torn down even if the IP address changes, thus providing seem less connectivity.

Fragmentation before encapusulation (Red side fragmentation) - IPsec

I don't know why it is called 'red side fragmentation'.

What is red side fragmentation?

Fragmentation of IP packets is done before ESP/AH/IPCOMP encapsulation in IPsec such that there is no fragmentation required to transmit the encapsulated packets.

Why is this required?

I believe some of service providers tend to give less priority to packets that are fragmented. When they detect congestion, fragmented packets are given lesser priority and they may get dropped.

IPsec carries many important protocol traffic across sites. Any IPsec packet drops cause degradation of application performance. IPsec gateways need to ensure that this traffic is not dropped. As you are aware, IPsec adds additional encapsulation headers such as ESP/AH/IPCOMP and another IP header in case of tunnel mode. It increases the packet size by few more bytes. If big size packet is received from local hosts and any additions to this packet requires fragmentation. If the fragmentation is done after encapsulation, the resulting packets have fragmentation variables set (MF bit and Offset bit) in IP headers. To avoid this, fragmentation is done on clear packets and IPsec encapsulation, outer IP header insertion is done on each of clear fragments. In this case, the outer IP header does not have fragmentation variables set. By this, service providers don't give lesser priority to these packets upon congestion.

Many security devices supporting IPsec have this feature. As an evaluator of security devices, you should look for this feature.

OMA-DM Vs TR-069 - One difference

I found one difference between OMA-DM effort and TR-069.

DSLForum (TR-069) defines multiple technical reports (TRs) describing data models. It defined TRs for Internet Gateway Devices, VOIP TAs, NAS etc.. Within each TR, there are multiple profiles for each of the functions of devices. As I understand there are multiple working groups working on defining data models for different kinds of devices. Also, there are some working groups working on test specifications. It appears that DSL forum, at this time, concentrating on defining as many data models as possible and very active in that.

In case of OMA-DM, I did not see any effort going on in open mobile alliance group (based on public access I have to their site and I may be wrong) to define configuration parameters for different kinds of applications. I am not sure whether it is intentional to keep this effort out of this group. But this is one difference I find between OMA-DM efforts and TR-069 efforts by respective standard bodies.

TR-69 Vs OMA-DM

TR-069 is standard from DSLForum to remotely manage networking devices. OMA-DM is standard from "Open Mobile Alliance' to manage smart phones remotely. Though the target market segment is different, I don't see any major difference between both of them as far as their functionality is concerned. I believe that both the standards are suitable for both kinds of market segments.

Similarities:

• Remote configuration and monitoring of devices
• TR-069 is meant for managing networking devices.
• OMA-DM is meant for managing Mobile devices.
  
• Firmware upgrade.
• HTTP based access - OMA-DM support other transports too.
• SSL for security.
• Both use XML for exchanging messages.
• Both support configuration parameters in hierarchical fashion (Tree form).
• Both expect devices to make connection to the central configuration server.
• Both support mutual authentication.
• Both have mechanism for central configuration server to notify devices of different events.
  
• OMA-DM uses WAP-Push and SMS.
• TR-069 connects to device embedded HTTP Server.
  

I don't see many differences though. I will post new entry once I find more information.

Both of these standards have become very popular in their market segments and IMO both will be surviving for a long time.

Mutliple security Zones in Enterprise networks - Security Devices

Traditionally, security devices such as firewalls and UTMs are deployed at the Enterprise Edge. At Enterprise edge, one is satisfied with trusted zone (corporate network), untrusted zone (Internet), De-Militarized zone (DMZ) separation provided by security devices.

Increasingly, security devices are being deployed in Enterprise core where multiple department networks are present - Engineering, Marketing, Sales, Finance etc.. These security devices not only provide access control, intrusion prevention, network anti-virus and spam protection from Internet (untrusted network), but also provide isolation among the departments. Enterprises are increasingly providing internet connectivity for visitors too. Isolating corporate department networks from visitor network is very important from security perspective. To provide granular access controls and other security services among departments, three network zones provided by traditional security devices are not good enough. Security devices must support definition of zones and policy configuration among these multiple zones. For example, a firewall ACL rule can be defined to allow a specific traffic such as HTTP between Finance zone and Engineering zone. Security devices supporting multiple zones provide rule definition with 'From' and 'To' zones in additional traditional 5 tuples (source IP, destination IP, Protocol, Source port and destination port).

Some security devices in the market provide rule definition for each network interface. That is, each network interface is considered as a separate zone. Yet times, many Ethernet interfaces are connected to different segments of one department. In those cases, the rule definition needs to be duplicated for each interface. So, I prefer security devices providing concept of zones and associating Ethernet and other network interfaces to zones. Security policies are defined with respect to zones. By separating rule definition from interfaces, addition of new interfaces and deletion of interfaces to/from zones does not effect the rule definitions for that department.

SNAT and DNAT on same connection?

One small office network administrator asked me this question on why both source NAT and destination NAT required on the same TCP/IP connection. My answer: It is not required in many deployments, but some deployment do require this. Read on... Let me give a small brief on source NAT and destination NAT first.

In many deployments 'source NAT' is used when number of public IP addresses given by ISP are less than the number of machines that require internet connectivity. In many small office deployments in Asia and Latin America typically get only one dynamic public IP address. In few cases, upon request, few static public IP address are given for a fee. Source NAT in NAT routers is used with NAPT (Port translation) to provide internet connectivity for many PCs with one or more public IP addresses. Source NAT is used for outbound connections - Connection originated by internal PCs to resource in public network (Internet). In source NAT mode, source IP and port values are changed on client to server packets. Destination IP address and port values are changed on server to client packets so that the packets go to the PCs that initiated the connection.

Note that 'source NAT' does not mean that only Source IP and port values are changed in all packets. The term 'source' or 'destination' in SNAT and DNAT is used to represent the translation points on first packet of connection i.e client to server packets. Note that reverse translation happens on server to client side packets, that is if source IP is changed in client to server packets, then destination IP is changed in server to client packets of the connection. But the term 'source NAT' is used to represent this kind of NAT.

Destination NAT is typically done for inbound connections. When Enterprise have many servers inside (for providing services for users in public network) and if the Enterprise has less number of public IP addresses than the number of servers , then destination NAT is required on routers (NAT routers). Let us say that a small Enterprise have two servers - One Web server and one email sever. If that enterprise have one public IP address, then all HTTP and SMTP/POP3 connections land on this IP address. The NAT router having this public IP address must have DNAT facility to redirect the traffic to internal HTTP Server and Email Servers, by changing the destination IP address and possibly port on the incoming packets (client to server packets).

SNAT and DNAT on the same connection: So far we have discussed that outbound connections from Enterprise use SNAT and inbound connections to Enterprise use DNAT. SNAT and DNAT combination means that both source and destination IP addresses of the packets are changed at the same time. SNAT+DNAT is typically required for inbound connections when the Enterprise has more than one WAN connection being serviced by different NAT routers. These NAT routers are assigned with appropriate public IP addresses or they get IP addresses dynamically from ISP either via DHCP or PPP. If inbound connections undergo network translation by a router, it is imperative that reverse traffic on the connection also go through the same router for reverse address translation. Inbound connections can land on any of the routers (based on the IP address used by clients in public network) and get redirected to internal server. Internal servers are not intelligent enough to route the packets to appropriate router. Typically they use same default route for all outbound packets. In theses cases, all packets go to one of two routers. This router may not be able to service connections which were redirected by other routers. To overcome this, routers need to apply SNAT on the connections. This will ensure that reverse traffic from servers go to right routers.

Asymmetric Routing - Security Devices

It appears that asymmetric routing is very common in Enterprise networks. It was a surprise for me when I first heard about this few years back.

When packets of a connection take different routes in networks, then the network has asymmetric routing problem. Any stateful security device will have problem with this. Stateful security devices expect all packets of connections pass through them - Client to Server and Server to Client packets. If packets belonging to one leg of connections don't pass through, these devices fail to parse the protocol states and report them as 'attacks'. For example, if second packet of TCP connection (TCP Packet with SYN+ACK flag - First packet from Server to Client) is received by security device and not the first packet (TCP packet with SYN flag only - Client to Server packet), then the security device (firewall) treats this as an 'attack' or reconnaissance attempt by attacker. Typically these packets are dropped. Hence, the connections are not established, even though they are genuine.

Many times, this asymmetric problem is observed in Enterprise core networks, but not on Enterprise Edge. It is observed that this problem is not so much in large Enterprises as security administrators work with network administrators to fix the network. In SME markets, security device vendors can't expect them to fix their network problems immediately. Security device vendors are expected to have work around in their devices while administrators fix the problems over time.

Some security device vendors support feature 'Bypass Security processing'. This feature allows users to configure multiple bypass records with each record taking pair of networks. Any traffic having source IP address and destination IP address falling in any of bypass records is forwarded or switched immediately, without any security analysis. Forwarding or switching decision is taken whether the security device is operating in route mode or bridge mode.

As a security administrator, it is better to look for security devices that have bypass flexibility.

Are firewalls susceptible to DDOS?

Answer is : Many of them.

Packet filters are thing of past. Firewalls today are stateful inspection firewalls or proxy firewalls. In both cases, they maintain state information in sessions. Sessions are created for TCP/IP connections. These firewalls maintain ACL - Access control Lists - configured to implement company policies. First packet of any TCP/IP connection passing through firewall device is validated against ACL. If ACL rule allows the connection, then rest of packets of connection are allowed to pass through the firewall device.

Some of the firewall devices in the market have additional functionality such as protection from DoS (Denial Of Service) attacks, complex application traversal functionality via Application Layer Gateways (ALGs) and source/destination Network Address Translation (SNAT and DNAT). Don't think that firewalls stop DDOS attacks effectively. Don't be fooled by literature. Some firewall devices don't even protect themselves from the DDOS attacks.

Since firewalls maintain states, they are susceptible to DDOS attack unless firewall vendor has taken enough precautions to provide functionality beyond 'Access Control'. Don't bet that all firewalls have functionality to defend against DDOS attacks. Before I proceed further recommending the features to look for in firewall to minimize DDOS attack impact, let me give a brief on DDOS attacks.

DDOS Attacks: DDOS attacks by definition are distributed Denial Of Service attacks. The attack is mounted from multiple external locations at the same time on to internal servers/services. It results in flood of packets and thus disturb service availability. Some of DDOS attacks when mounted smartly do not even require enough bandwidth for attackers.

Firewalls can't prevent DDOS attacks completely - Some attacks can be detected and prevented, some can be detected and further action needs to be taken manually. But it is very important that firewall devices themselves should not become victim of these attacks and firewalls are expected to minimize the impact of DDOS attack.

Some DDOS attacks are nontrackable. Source IP is spoofed in these cases. If source IP of the attack packets are spoofed, the response packets from the victim can't be routed to the attacker. Hence these attacks are mostly send multiple packets with one packet per connection to overwhelm the resources at the victim end. In case of TCP based connections, attacker can send large number of SYN packets with spoofed source IP addresses. This is called SYN flood attack. If firewall device does not have defense against this, attacker can overwhelm the TCP listen queue in server machines and also he/she can overwhelm the session table in firewall, thus the complete network behind firewall is unreachable for genuine users as long as SYN flood attack continues. Fortunately, many firewalls have mechanisms to defend from this attack. Firewalls implement SYN-Cookie defense mechanism to allow connection only if originator responds with next packet in TCP (ACK) to the packet firewall sends (TCP packet with SYN and ACK flags). Firewall do this without creating any session from its session table. If your firewall does not have this feature, then you better go for firewall device from some other vendor supporting this defense mechanism.

Next question to ask in selecting the firewall is DDOS defense mechanism when the attacker overwhelms the network with full TCP connections - that is, attacker responding with ACK and keeping the connection for a long time. This attack also can overwhelm the TCP protocol servers such as HTTP, SMTP, IMAP etc.. and also can overwhelm firewall session table. Since TCP connection is established with full 3-way handshake, source IP address is known and attacker tracking is possible via service providers. But, firewall device should log these events and provide log analysis for administrator to take up the case with their service providers. Reporting and analysis is one important aspect of selection of firewall device.

UDP is one tough nut to crack if DDOS attack is mounted on UDP Service. By sending packets with spoofed IP address, attackers can easily overwhelm services and firewalls. Best precaution to take is to stop all inbound UDP Services. But yet times, it is not possible for companies to block these services. Some popular servers use UDP - DNS, IKEv1/v2, SIP, RTP etc. RTP is not an issue for firewalls as it requires some other signaling protocol such as SIP, H.323 and MGCP to go first.

Unlike TCP, there is no connection establishment phase. Hence it is difficult to figure out whether it is genuine UDP packet or coming from attacker. But the UDP flood on one service and an internal machine should not bring down other UDP and TCP services. Firewall devices should protect its session table from using up all resources and also ensure to make other services available. That is, if UDP SIP Service is being flooded, UDP DNS service and other services should not get affected. IMO, firewall should provide mechanism to control maximum number of simultaneous sessions for each internal service/server combination and the rate at which connections are getting established. Firewalls also should provide mechanisms to reserve sessions for some critical services/server combination for critical users.

Unless firewall provide these defense mechanisms, they can't prevent DDOS attacks and many times can't protect itself.

In summary, good firewall device must:

• Prevent from flood attacks such as SYN flood.
• Detect source of attacks in cases where source IP can't be spoofed. Logging and Reporting is very important feature to look for.
• Ensure availability of services even in cases where one of the services is being attacked. Session Table limit and reservation on per service/server combination and flexibility of configuration of same is very important to ensure availability of other critical services.

P2P Control in Enterprises - What to look for in security devices?

P2P application popularity has been soaring. They are mainly meant for sharing files - Audio, Video and date files. Due to sharing of copyrighted files, there has been multiple law suites on P2P software providers and p2P users. Still the popularity continue to raise. Now, audio/video distributors are using p2p mechanism for marketing purposes. Some of Linux OS distributors are using P2P applications for downloading Linux images. So, it appears that P2P application is being adopted for businesses as well.

One of the main advantages of P2P is its low cost of distribution: P2P uses distributed delivery mechanism. Big file is divided into smaller pieces and they get distributed. User P2P application downloading these files will provide upload service there by making distribution network bigger day by day. P2P client downloads multiple pieces from different locations and combine them together to create original file. Original distributor site is less loaded due to inherent distribution capability of P2P applications.

Due to its low cost of delivering content to multiple users, this became play ground for distributing illegal copies and became kind of social networking tool where group of people share and distribute big size files containing audio/video and data content.

Some of popular P2P applications are:

• AppleJuice
• Ares: Ares, KCEasy the applications that use Ares protocol.
  
• Bit Torrent Protocol: This protocol is implemented by many applications such as Azureus, BitTorrent, BitComet, Mu-Torrent , Shareaza and many more.
• DirectConnect
  
• eDonkey Protocol: eMule-Plus, xMule, aMule. Shareaza use this protocol.
• FastTrack protocol: kazaa-lite, Shareaza, Kceasy use this protocol.
• Freenet
• Gnutella: Bearhare, Limeware, Gnuclues, iMesh and may applications use this protocol.
• iMesh
• Monolito
• WinMX
• Winny

Many one-click hosting providers provide content directly on their site for download. They include rapidshare, megaupload, youtube etc..


P2P applications are increasingly being used in Enterprise networks for personnel use. Some times, these P2P machines become super nodes where they become distribution point for some content, some times with the knowledge of user and some times inadvertently. These put strain on Enterprise WAN links. Also, it reduces employee productivity as they get hooked into these tools.

Enterprises are looking for ways to detect P2P applications and control them either by blocking the traffic or by limiting the traffic. Enterprise requirements are summarized as below.

• Enterprises might need to allow some P2P applications as part of their business.
• Enterprise might host P2P distribution servers to distribute their content.
• Enterprises might allow some of their Employees to use some P2P applications for business use.
• Enterprises might need to control bandwidth used by different P2P applications.
• Enterprises might have different requirements of access control and rate control based on time-of-the day or day of the week.

In essence, Enterprises look for solutions which detect different kinds of P2P applications and throttle or block traffic. Enterprises are increasing looking at Network based security devices to provide this capability so that they can control at one central place.

What is the feature set Enterprise need to look for in security devices providing this function? Before this question is answered, it is better to know the inner workings of p2p applications.

Some of attributes of p2p applications from detection perspective:

• Port hopping: P2P applications came long way. Initial version of P2P applications used to use fixed set of ports. It was easy to detect them and block them. P2P application developers improved their applications/protocols to use any port that is opened by local firewall. They even use Port 80 and Port 25 which are standard ports for HTTP and SMTP. Almost all P2p protocols use port hopping. So, detection of these application by port number is no longer sufficient.
• Obfuscations - Using Encoding techniques: Due to port hopping nature of P2P applications, security device vendors started detecting the P2P applications by observing content patterns in all connections. Each P2P application can be recognized by certain pattern. Security device vendors provide these patterns as signature rules. This was very effective and continue to be effective. With popularity of these methods, P2P application developers started using encoding mechanisms to bypass detection by pattern match. Some of the examples are : Winny, Ares Galaxy and Skype.
  
• Obfuscations - Using standard protocols with encryption : Security vendors started devising techniques to detect encoded connections by applying decoding logic before matching the patterns. Engineers did reverse engineering on some of the protocols or by understanding open source code of these applications. To thwart this, some P2P applications such as BitTorrent adopted encryption and making use of standard ports. BitTorrent started using HTTP for file transfer and SSL for encryption.
  
• One Click File hosting providers: Almost all of them provide file share using HTTP protocol. So, detection should happen on HTTP protocol.
  

From above description, you can observe that it is a catching game for both security vendors and p2p application developers.

My observation as of now is that many P2P applications can be detected by content pattern matching. Some applications require decoding before matching.

What is it one need to look for in security devices claiming to be detecting P2P applications:

• Just don't look for literature on application supported by the device. Ask questions. How is it being detected? How well it can add signatures to detect future versions of applications.
• Look for flexibility of application detection signatures.
• Look for application support which require decoding
• Look for HTTP protocol intelligent keyword support in signatures to detect one click hosting providers. Without HTTP application intelligence, there could be many false positives.
  
• Look for system that has ability to do SSL decryption (Proxy based or inline passive scanning).
• Look for detection capability which spans across connections: P2P applications which are encrypted can successfully be detected by doing scanning across connections. Note that many P2P applications have handshake connections and data connections. By combing for a pattern across these connections, many P2P applications can be detected.
• Look for traffic anomaly: Some P2P applications can't be detected by pattern match or can't be decrypted. In those cases, look for anomaly in the traffic with respect to number of connections made, detecting the connection rate, byte rate etc.. Though one can't pin point actual p2P application, it provides enough hints on who might be running p2P applications.
• Always look for capability to block the P2P application traffic and/or rate limit the traffic on per IP address basis and schedule basis thus providing control for Enterprises on who can use which P2P application(s) and at what time of the day.
  

IGMP Proxy

IGMP Proxy is defined in rfc4605.txt.

IGMP proxy is typically used in Edge routers - Either office edge such as gateway routers and provider edge such as DSLAM. One main point to keep in mind is that IGMP proxy is useful only in simple tree topology where tree can be configured manually. For complex trees where manual configuration is not possible, multi cast routing protocols such as PIM should be used. IGMP proxy is very simple in the sense that it acts as IGMP router for downstream interfaces and as a host on upstream interfaces. It listens for IGMP reports coming on downstream interfaces from hosts interested in listing to multicast traffic. It consolidates report information coming from different hosts across different downstream interfaces and send consolidatedreport to upstream routers.

In Office edge routers, the interfaces connected to inside machines are typically downstream interfaces and WAN interfaces connected to ISP routers are upstream interfaces. In case of DSLAM, WAN interfaces towards its customers are downstream interfaces and upstream interface is connected towards Internet.

Multicast stream providers get unique Multicast IP address for each type of stream from IANA. This IP address is advertised. Any machines willing to receive this stream send IGMP membership reports to routers indicating their willingness to receive packets coming on a particular multicast IP address. IGMP protocol is used to send membership reports.

IGMP proxy devices receives multicast stream from upstream interfaces, it knows the downstream interfaces interested in this traffic based on membership reports it received before. It duplicates the traffic and send it to these downstream interfaces. It provides great advantage on cost savings as only one copy of stream is sent on WAN link. Gateway implementing IGMP proxy duplicates on multiple downstream interfaces.

IGMPv3 support source specific multicasting. IGMPv3 hosts can request the stream only when it is generated from a particular source. It provides some kind of security for hosts (note that IP address can be spoofed by rogue multicast stream generator) and more importantly it eases the network routing problems. IGMP proxy should honor v3 memberships with specific source IP address. If proxy receives v2 or v1 membership from local hosts on the same interface, then source specific multicast membership from v3 hosts will not have desired effect.

As discussed before, IGMP proxy acts as IGMP router on downstream interfaces. As an IGMP router, it sends IGMP queries periodically to ensure that hosts are still interested in multicast streams. IGMP routers typically send generic query periodically. Group specific query is sent upon a state change such as host leaving the group.

IGMP Proxy takes responsibility of creating multicast routes based after consolidating membership reports.

With the above background, let me present the IGMP proxy configuration required in gateways.

• Enable/Disable: Whether IGMP proxy operation is to be enabled or disabled. If it is disabled, this router does not even entertain reception of IGMP messages.
• Maximum number of groups : This parameter represents the number of groups allowed by IGMP Proxy. Note that 'groups' have one to one association with multiple IP addresses.
• Robustness: Please rfc3376 for explanation. It is mainly intended for lossy network. This parameter value is default value for 'Startup Query Count' and 'Last Member Query Count'. The queries are sent as many times as 'robustness' count separated by 'Startup Query Interval' and 'Last Member Query internval respectively. Default value is 2 as per rfc3376.
  
• Query interval: The number of seconds between two IGMP general query messages. Default : 125 seconds as per rfc3376.
• Query response interval : Amount of time the querier waits for response to general query. Default is 100 (10 seconds). Each unit corresponds to 100milliseconds.
  
• Startup Query Interval : Number of seconds between general queries during startup.
• Startup Query Count: Number of queries sent out upon startup.
• Last Member Query Interval: Amount of time router can wait before sending group specific query or group-source specific query in response to leave group messages. Default is 10 (1 second). Unit is equivalent to 100 milliseconds.
  
• Last Member Query Count: Number of queries sent upon receiving leave group message.
• Host Unsolicited Report Interval: This variable used by host portion of IGMP proxy. This value is represented in seconds and is used by host to send reports. Host is expected to send 'robustness variable' number of reports within this time to the routers to cover the possibility of report misses by routers.
• Downstream interfaces: List of downstream interfaces. Interfaces are typically of type: wireless LAN, USB Ethernet, Ethernet, VLAN and PPP etc..
• UpStream interfaces: List of upstream interfaces.
• Log : YES or NO. IT departments always would like to know the network view. IGMP membership information provides valuable information about usage patterns etc.. To provide historical information to know the join and leave intervals of machines for different groups, it is required that log entries are generated. This variable controls whether log is to be generated or keep quiet. Log entry should contain
• Machine IP address
• Group IP address (Multicast IP address)
• Time at which it joined or time at which it left.
• Interface used for join or leave.

Run time information: Multicast member ship view is very important for administrators to know what is happening at any given time. One would like to know things like - Machines and their membership information and interface information on which the machines are present etc.. To be precise:

• For each group in IGMP
• Group Address (Multicast address)
• For each downstream interface
• Interface name
• Time at which this group was added.
• Source IP addresses included.
• Source IP addresses excluded.
• Machines that are interested in this group.
  

• Statistics :

• Number Of General Queries Received,
• Number Group Specific Queries Received,
• Number Of Group & Source Specific Queries Received,
• Number Of Queries Transmitted
• Number Of V1 Reports Received,
• Number Of V2 Reports Received,
• Number Of V3 Reports Received,
• Number Of Leave messages Received,
• Number Of Reports Transmitted,

With above in mind, TR-069 model can be represented as:

• internetGatewayDevice.IGMPProxy P
• Enable : RW, Integer.
• MaxNumberOfGroupsAllowed: R, Integer
• LogEnable: RW, Integer.
  
• RobustnessValue: RW, Integer
• QueryInterval: RW, Integer
• QueryReponseInterval: RW, Integer
• StartupQueryInterval: RW, Integer
• StartupQueryCount: RW, Integer
• LastMemberQueryInterval: RW, Integer
• LastMemberQueyrCount: RW, Integer
• LastMemberQueryCount: RW, Integer
• HostUnsolicitedReportInterval: RW, Integer
• internetGatewayDevice.IGMPProxy.Interface.{i} PC
  
• InterfaceReference: RW, String : Once assigned, can't be changed. This full qualified name of interface instance of LANDevice, WANIPConnection, WANPPPConnection, VLANInterface.
• InterfaceType: RW, String. "UPSTREAM", "DOWNSTREAM". Once set, it can't be changed.
• internetGatewayDevice.IGMPProxy.GroupInfo.{i} P
  
• GroupAddress: R, String.
  
• internetGatewayDevice.IGMProxy.GroupInfo.{i}.interfaceInfo.{i} P
  
• InterfaceReference: Read Only, String.
• Time : At which group is added. Date & time. Read Only, String type.
• IncludedSourceIPs: R, String. Comma separated IP addresses.
• ExcludedSourceIPs: R, String. Common separated IP addresses.
• internetGatewayDevice.IGMPProxy.statistics P
• internetGatewayDevice.IGMPProxy.statistics.hostside P
  
• NumberOfGeneralQueriesReceived : R, Integer
  
• NumberOfGroupSpecificQueriesReceived : R Integer
  
• NumberOfGroupAndSourceSpecificQueriesReceived : R, Integer
  
• NumberOfReportsTransmitted: R, Integer
  
• internetGatewayDevice.IGMPProxy.statistics.routerside P
• NumberOfQueriesTransmistted : R Integer
  
• NumberOfV1ReportsReceived: R, Integer
  
• NumberOfV2ReportsReceived: R, Integer
  
• NumberOfV3ReportsReceived: R, Integer
  
• NumberOfLeaveMessagesReceived: R Integer

WAN Links - Sharing Load and Failover - TR069 based configuration

CPE devices provide a feature called 'WAN link load balancing and fail over' facility where they have multiple WAN links to reach Internet. These WAN links are either taken from one ISP or from different ISPs by offices/homes. They provide redundancy capability and also can be configured to provide load sharing.

As far as I know, DSL forum did not define profile for this. I tried to give profile and required configuration parameters needed to configure this feature in CPE from ACS using TR-069 standard.

What constitutes this feature:

• Multiple WAN links in CPE.
  
• WAN links grouped together into multiple bundles.
• Each bundle having following properties
• Share the TCP/IP connections load across links in the bundle or just use only one link, user others when current link fails.
  
• In case of TCP/IP connections sharing, whether to bring up the all links always or bring up/down based on number of TCP/IP connections at that time - Define high threshold and low threshold in percentages. When number of TCP/IP connections reach high threshold, bring up new link in the bundle. When the connections reach low threshold, bring down one of the expensive links.
• Each link having following properties
• Method to use to check liveness : Ping, DNS resolution, None.
  
• Domain name, if method chosen is Ping or DNS.
• Liveness check interval in seconds: Liveness check is done using this interval period.
  
• Number of times the livness check should fail before marking the link 'Down'.
• Link status : UP, Down
• Cost of the link : 1 to 10 - 1 being highest cost and 10 being lowest cost.
  
• WAN interface: Interface identifier or logical identifier identifying the link.
  
• Protocol Bindings: Yet time, it is required that some protocol traffic go through some particular link in the bundle. For example, if email server is hosted by one ISP, all email connections should be sent via that link. ISP may not accept emails coming from the site via other ISP links.
• Each protocol exception record takes following parameters
• Destination IP address - Range of IP addresses.
• Protocol - UDP, TCP or UDP & TCP and any other protocol value.
• Port range: Valid in case of TCP and UDP.

Inner workings:

• Route dictates the outbound interface for packets.
• If outbound interface is WAN link, bundle is determined.
• If this is first packet of connection, then least loaded link is chosen in the bundle. If needed new link is brought up. IP address of chosen link is used for SNAT. Consider protocol bindings in choosing the link.
  
• When there are multiple WAN bundles, it is expected that ACS configures routes with source IP address - Source based routing. It enables usage of multiple WAN bundles. Note that there would be multiple default routes - one for each bundle. It is expected that ACS configures source based routes to make use of multiple bundles.

Profile as per above description:

• internetGatewayDevice.WanSharingBundle.{i}
• Enable
• Share or Failover only?: Indicates whether the load is expected to shared or use multiple links for failover only.
• HighThreshold
• LowThreshold
• NumberOfLinksInThisBundle
• internetGatewayDevice.WanSharingBundle.{i}.link.{i}
• Enable
• LinkStatus
  
• LivenessMethod : None, Ping, DNS
• DomainName : If ping or DNS is chosen as liveness method.
• Liveness interval
• FailureCount
• WanInterface : This is Full Qualified WANIPConnection or WANPPPConnection instance under WANDevice.
• CostOfLink
  
• NumberOfProtocolBindings
  
• internetGatewayDevice.WanSharingBundle.{i}.link.{i}.ProtocolBinding.{i}
• Enable
• MinSourceIPAddress
• MaxSourceIPAddress
  
• MinDestIPAddress
• MaxDestIPAddress
• Protocol
• MinPort
• MaxPort

These are my high level thoughts. More thinking should go in to make it more generic. I hope that it provided decent introduction for further work.

SIP vulnerabilities - Different types

There are many vulnerabilities being discovered in SIP implementations - SIP UA, SIP Proxy and even in SIP Border session controllers. Based on review of some of the vulnerability reports they can be categorized to different types.

• Buffer overflow attacks in
  
• SIP request first line fields - method, URI and version.
• SIP response first line fields - version, status code and reason phrase
  
• Duplicate header fields
• SIP header fields
• SDP fields
  
• Missing header fields
• SIP header fields
• SDP fields
  
• Invalid data in the
• SIP header field names
• SIP header field values
• SDP field names
• SDP field values.
  
• Short messages
• SIP request message size
• SIP response message size
  

IPS/IDS devices can do job of detecting attacks exploiting above types of vulnerabilities without false positives only if they have rich set of SIP intelligent syntax in their rule language.

Many IDS/IPS devices do support detection of traffic anomaly with respect to traffic rate, connection rate etc.. They also support detection of protocol anomalies. SIP protocol is text based protocol and SIP standard does not specify the length limits for start line fields, header field names and values. Due to its flexibility, protocol (RFC) violation detection is limited. To beef up the anomaly detection, I believe that IPS/IDS devices should support detection of protocol content anomalies for zero day detection (if not protection). At the minimum, content anomaly detection should include first line fields. SIP header fields and SDP fields. Since there is no standard on limits, IDS/IPS devices should provide flexibility for administrator to change the anomaly rules. IT departments can tune these rules based on the deployment requirements.

This anomaly detection is kind of first alert system. Upon alert, security professionals can find out the intentions of the attackers and take further actions such as beefing up the security by creating newer, stringent rules, patching the victim systems and updating their honeypot configuration for tracking the attackers etc..

Any anomaly detection should have some baseline. I am not an expert on SIP devices and SIP deployments. Based on the type of vulnerabilities, I feel baseline and associated rules to detect anomalies can be created (by security professionals or signature developers) by some guidelines such as:

• For each field in start line, SIP header and SDP fields
• Determine all possible values for fields.
  
• Determine typical lengths of field names.
  
• Determine typical lengths of field values.
• Determine typical set of characters observed normally
  
• Determine typical header fields that normally exist in SIP requests and responses.
• Determine header fields which are normally unique.

Create rules to detect any anomalies based on above baseline. Also, it is necessary to detect SQL injections and XSS injections by validating the field names and values.

SIP Security - IPS/IDS role

VOIP uses TCP/IP based communication like data services. Due to this, data threats are applicable for VOIP based systems - VOIP phones, VOIP infrastructure appliances etc.. Most of these threats and possible solutions are well discussed in many forums. Here I tried to give gist of different types of threats that are possible on VOIP systems.

• Service disruption (Denial Of Service): Service disruption can happen in many ways.
• Attacker exploiting SIP vulnerabilities and bringing down the service.
• Attacker exploiting vulnerabilities in software other than SIP and bringing down the service and device (phone or appliance or server).
  
• Attacker sending large number of SIP requests towards phone or SIP appliance and making it unusable.
• Attacker sending large number of packets to bring down the voice quality.

• Taking control of SIP device:
  
• Attacker exploiting vulnerabilities in OS and application software and injecting his/her own code to get the shell access.
• Attacker exploiting poorly configured devices.
• Taking advantage of default user name and passwords on devices: Many administrators and users forget to change the factory default credentials. Attackers take advantage of this to get control of the device.
• Taking advantage of week passwords using brute force attacks.
• Taking advantage of devices that are on public network with remote access enabled.
• Service theft: Attackers once they break into SIP phone use this to make many outbound phone calls.
  
• Eaves dropping: Hackers can eaves drop on media streams and steal sensitive information. This is possible by
• Attackers breaking into the system as explained above in 'Taking Control of SIP device' and snooping on the media streams.
  
• Attackers spoofing MAC addresses (ARP poisoning) and becoming Man-in-the-middle to snoop on media streams and any other content.
• SPIT (SPam over Internet Telephony): Similar to Email spam. Unsolicited marketing calls, commercial calls fall in this category. Unlike traditional telephony, cost of making calls using VOIP is way less. Spammer only require high bandwidth data connection which is very cheap.
  
• VOIP Phishing (Vishing): Email type of phishing fraud can be extended to VOIP by frauds. They can leave a message indicating it is very important to callback and callback number as if legitimate entity (such as bank) is called by spoofing 'From' address (Caller ID). Innocent user might call this number and provide identity information to illegitimate parties (frauds).
  

Protecting from these attacks:

No single solution can solve all of above problems. I believe Intrusion Prevention System (IPS) has a role in preventing some of above attacks on VOIP phone and other infrastructure. IPS device can be kept in the line of SIP/Voice traffic to recognize attacks and prevent them in going across. I try to describe IPS role for each of above threat types.

• Service Disruption -> SIP vulnerabilities and Vulnerabilities of other software running on the devices: IPS devices are actually designed to detect exploits targeting known vulnerabilities. IPS devices have facilities to keep up to date with exploits and vulnerabilities using auto signature download mechanisms. IPS devices also provide facilities for administrators to create their own signatures, if need be. One should look for following capabilities while selecting IPS device to protect SIP infrastructure.
• SIP protocol intelligence: Look for IPS device having SIP protocol intelligence. Without this, there would be too many false positives. Without this, it may not be possible in some cases to develop signature to cover vulnerabilities.
• HTTP Protocol intelligence: Almost all SIP infrastructure devices provide configuration via HTTP. Any vulnerabilities in HTTP Server brings down the device as well as may allow 'root' access to the device. Hence, one should look for IPS devices that have HTTP protocol intelligence.
• Other protocol intelligence: Make a list of services running in your VOIP infrastructure and look for IPS devices having these protocol intelligence.
• Service Disruption -> flood attacks: IPS devices are ideal for detecting and preventing from flood attacks. IPS devices typically have functionality to detect traffic anomaly. IPS devices doing anomaly checks with application intelligence detect flood attacks without any false positives. Look for IPS devices that support detection of traffic anomaly on specific SIP method or on content of SIP header lines in request or response. IPS devices can detect DDOS attacks that occupy bandwidth, but may not be able to prevent them on real time basis. Due to its detection, it provides information for administrators to take out-of band action.
• Taking Control of SIP Device -> Vulnerabilities: As stated above, IPS devices are ideal for detecting traffic that exploit vulnerabilities.
• Taking Control of SIP Device -> Configuration Errors: IPS devices may not be able to detect this effectively. Using IPS rule editor functionality, administrators can create signatures to look for specific strings in the traffic.
  
• Taking Control of SIP device -> Brute force attacks : IPS devices can detect this if protocol based error is returned by SIP devices upon bad authentication credentials. IPS traffic anomaly based signatures can detect this.
• Service Theft: IPS devices may not be able to prevent service theft completely. But IPS device logs can be used to detect any service theft. IPS devices typically log each transaction. Look for IPS device that support SIP based log analysis (inspection) facility. In my view, at the minimum, IPS device should have following functionality:
• Log inspection and report generation based on multiple filtering criteria such as SIP URI, date & time range, IP addresses etc.. Report output should at the minimum contain information about each local SIP phone (SIP URI). It should give number of calls received, calls originated, average duration of call, time of call etc in addition to detailed entries with each entry indicating time of call, Called party ID, duration of call and other SIP related information to identify the pattern.
• Eaves-Dropping: IPS devices can detect whether voice traffic is sent in clear using signatures.
  
• SPIT: IPS devices having SIP application intelligence can detect SPIT using SIP header line content. For example, IPS devices can detect spammers based on 'From' addresses. This kind of detection is not sufficient when spammer keep changing the 'From' addresses i.e spoofing 'From' addresses. IPS devices can't do turing where callers are provided with random challenge to play back. This turing facility helps in rejecting automated spamming systems (no human). SIP proxy firewalls are better suited for this functionality.
  
• Vishing: Vishing problem can be solved only by educating end users. IPS devices can't detect and prevent from users making calls to illegitimate party. IPS devices may help some extent if thare are known 'black' lists. I am not sure even SIP proxy firewalls can solve this problem completely.

IPS device play very important role in detecting some of critical VOIP threats and preventing them. I strongly advice Enterprises deploying VOIP infrastructure to look for IPS devices with SIP intelligence.

Encapsulation-less Route based IPsec VPN

Some one who read my blog asked me a question on mechanism to identify right Security Policy entry, now that there could be multiple policy entries with the same selectors.

In a deployment where there are multiple branch offices, head office security gateway has as many Encapsulation-Less Routerbased VPN (ELR VPN) interfaces as number of branch offices. It is quite possible that each branch office has different security requirements, hence there would be multiple SPD policy entries in head office router. When a branch office router initiates the IKE exchange, head office securty router should identify the right SPD (Security Policy Database) policy entry to negotiate security parameters for data Security associations. In typical policy based VPN and route based VPN, the SPD policy record selection is done based on selector values being negoatiated. But in Encapsulation-less Route based VPN (ELR VPN), selectors are always 0.0.0.0. Hence, this can't be used to select the SPD policy record.

ELR VPN implementation must use some other paramters to select the SPD policy entry. At the same time, implementations should not be creating proprietary extensions to IKE. So, we decided to go with 'remote ID'(ID of the initiated party) as index parameter to select right SPD policy record from SPD policy database.

In above example, each ELR VPN is associated with branch office identities. When IKE exchange is initiated by branch office router, it sends its identitity. Using this identity, head office router selects policy record that corresponds to branch office.

Hope this helps.

TR-098 LANDevice - Clarifications and suggestions

TR-098 LANDevice term is misnomer. It is actually a 802.1D bridge combining multiple Ethernet ports, USB LAN ports and WLAN Ports. Moreover it is very confusing to the reader of TR-098 ammendment1 specifications which defines sepearate 'Bridging' profile.

After getting some clarifications from some of TR-069 experts, now I have some understanding of differences between LANDevices of Basic profile and 'Bridging' profile.

LANDevices: LANDevices is a bridge device having not only multiple ports, but also can be assigned with IP address(es), configured to have DHCP Server instance. The bridge interface created for each of LANDevices instance becomes Layer 3 interface (IP Interface). Hosts connected to ports of this bridge device get IP addresses from same IP address range configured as part of DHCP Server settings.

Bridge Profile bridges (Layer2 bridging): Conceptually this is similar to LANDevices in that it has multiple ports, but it can't be configured with DHCP Server settings and can't be assigned with IP addresses. In essence it is only meant for pure bridging of the packets. It is mainly useful to connect internal devices directly to Service provider networks such as IPTV STB etc..

I believe that bridging profile was made complicated. It should have had a Ports.{} tables within the briding profile rather than having 'Filter.{}' table and 'Marking.{}' table. LANDevice definition did not even have provision to add new interfaces to it. In my opinion, interfaces should be defined in different profile and add these interfaces to LANDevice bridges or Layer2bridging.

Suggested new profiles and changes to existing profiles:

Interface table: This table represents the physical or logical interfaces of the device. These are fixed interfaces in the device such as physical interfaces - Ethernet, USB etc.

• InternetGatewayDevice.EthernetInterfaces.{i} - This is renamed from 'EthernetLANProfile'. I Removed LANDevices tree node and I kept EthernetInterfaces directly under InternetGatewayDevice.
  

• New paramters: These parameters are in addition to the parameters that were already present.
  


• Name (Read only): Indicates the name of the interface on the device. Indicates 'eth0', 'eth1', etc..

USB LAN Profile also need to have similar changes as above.

VLAN interface: I suggest to have this new profile.

• InternetGatewayDevice.VLANInterface.{i}: ACS can add new entries to it. This configuration is used to create new VLAN interfaces from Ethernet interfaces based on VLAN ID.
• Parameters under this tree:
• Name of VLAN interface (RW variable, string type, maximum size of string 32 bytes): Device creates interface with this name.
  
• Enable (RW Variable, integer type): TRUE (value 1) is to enable the record and FALSE(0) to disable the record. Device should discard packets on disabled VLAN interface.
  
• ParentInterface Reference (RW Variable, String type): This is fully qualified name in EthernetInterfaces or USBInterfacess table. Example: InternetGatewayDevice.EthernetInterfaces{i}. Device creates VLAN interface on this Ethernet interface.
• VLAN ID (RW Variable, Integer type): VLAN ID. Device creates VLAN interface with this identifier.
  
• UntagUponReceive (RW Variable, Integer type): If true, Packet is delivered to stack from VLAN interface by removing VLAN header from the packet. If false, VLAN header is delivered to the sdtack. Default value is TRUE.
• TagWhileTransmit (RW Variable, Integer type): If true, packet delivered to VLAN interface for transmission will include VLAN header. If false, VLAN header is not included in Ethernet packet. Default value is TRUE.
• PriorityMarkWhileTransmit(RW Variable, Integer type): Indicates the priority value to be kept on transmitted packets. This field value is considered by device only if 'TagWhileTransmit' is set to TRUE.
  
• InternetGatewayDevice.VLANInterface.{i}.stats. : This block represents the statistics that are to be gathered by device. 'Sent' variables indicate the number of bytes or packets sent to the VLAN interface for transmission and 'Received' variables indicate the number of bytes or packets received by VLAN interface from parent Ethernet interface
  
• BytesSent (Read Only variable, integer type)
• BytesReceived(Read Only variable, integer type)
• PacketsSent (Read Only variable, integer type)
• PacketsReceived (Read Only variable, integer type).
  

• LANDevices.{i} changes:
• I believe that LANDevices also can be created dynamically. First change is to make it as 'PC' as requirement. Second change is to have 'Ports.{i}' to add bridge ports to the LANDevice.
  
• InternetGatewayDevice.LANDevice.{i}.Ports.{i} : This table and entries can be added, hence requirement is 'PC'.
  
• This block contains following parameters
  
• Interface (RW Variable, String type}. This is fully qualified name from EthernetInterfaces, USBinterfaces, WLANInterfaces and VLANInterfaces.
• L2Bridging changes: It needs to have only following parameters
• Number of Bridges.
• InternetGatewayDevice.Layer2Bridging.Bridge.{i} : ACS can create new instances, hence the requirement is 'PC'. Each table entry represents one bridge. Typically, this is used to bridge LAN and WAN interfaces.
• Name (RW, String type, 32 bytes long): Name of the bridge entry.
• Number of Ports (Read Only, Integer type): Number of ports in the bridge.
• Enable (RW, integer type). Takes 1 or 0 values. 1 indicates the bridge is enabled and 0 indicates that bridge to be disabled.
  
• Status (Read Only, string type) : Status of the bridge.
• InternetGatewaydevice.Layer2Bridging.Bridge.{i}.Port.{i}: This table represented bridge ports.
• Interface : Interface name of the port.
• EtherTypeFilterList, EtherTypeFilterExclude, SourceMACAddressFilterList, SourceMACAddressFilterExclude, DestiMACAddressFilterList, DestMACAddressFilterExclude: Same as TR-098 definition.
  

Route based VPNs - Explained

IPsec standard predominantly talks about policy based VPN. Policy based VPN contains multiple policy records, with each policy record having source, destination networks/hosts, service port and security paramters such as encryption, authentication algorithms etc.. These records are arranged in ordered list. Packets are matched against the policy table. If there is a policy record match, appropriate secuirty is enforced on the packets such as encryption and adding additional headers to reach the remote gateway. Remote gateway decrypts the packets and sends clear packets to intended host.

Thought policy based VPN gives granular control for administrators, it has its own disadvantages. If pair of security gateways have multiple networks and services for which data security need to be applied, then policy record(s) must be configured with these networks on the gateways. Every time new network is added in a site, policy records should be updated. Configuration update should be done not only on local gateway, but also in remote gateways. Source IP selector of policy record gets modified with new network on local gateway and Destination IP selector needs to get modified with new network on remote gateway. Admins not only need to configure routes to reach new networks in remote gateways, but also add or modify IPsec policy records. Since networks are to be added to the policy records explicitly, dynamic routing protocols also can't be used across sites.

Route based VPN solves above problems. In route based VPN, a point-to-point L3 interface is created and all traffic sent to this interface are tunneled to the remote gateway. As many interfaces are created as number of remote gateways. For a given pair of gateways, only one tunnel is created. Once this is done, administrator only needs to add routes to remote networks via tunnel interfaces. If dynamic routing protocols are used, admin need not even create routes explicitly.

Multiple types of route based VPNs are implemneted by appliance vendors. They are:

IP-in-IP route based VPN: Interfaces are created with IP-in-IP. IP packets are encapsulated with new IP header. Outer IP header IP addresses are gateway IP addresses. One IPsec policy record is created with these two gateway IP addresses as selectors. Any packets sent to this interface is encapsulated with outer IP header and then IPsec processing with ESP/AH encapsulation is done. When the encrypted packets are received, packets are decrypted, outer IP header is removed and then internal packet is routed to intended host.

GRE Route based VPN: It is similar to IP-in-IP route based VPN, except that it gets encapsulated with GRE+IP header. IPsec policy is created with GRE protocol and gateway IP addresses.

Encapsulation-less route based VPN: In this mode, there is no additional encapsulation such as IP-in-IP or GRE. Only IPsec ESP tunnel is used to encapsulate the packets with gateway IP addresses. In this case, Ipsec policy record is created with Source IP and destination IP selectors as 0.0.0.0 (ALL). So, selector negotiation happens with 0.0.0.0 IP addresses.

Since L3 interface is created, any dynamic routing protocols such as RIP, OSPF and BGP work on these interfaces and don't require any changes to the routing protocol implementations.

Session based UTM clustering - Challenges & Solutions

UTM Clustering where multiple UTM devices work together to increase the performance of certain UTM functions. Anti Virus functionality of UTM is computationally intensive and this functionality benefits greatly with clustering. Since many security functions are stateful in nature, UTM clustering technology distribute the load with sessions. That is, packet belonging to one session is processed by one of the devices in the cluster. That is why, I call this as 'Session based UTM clustering' or simply 'UTM Clustering'.

In recent past, UTM clustering lost its luster due to multi core processors such as Intel Quad processors. But, still I see that clustering is continue to be used, but I doubt 'Session based UTM clustering' is going to survive in long run. Having said that, I feel load sharing based on virtual instances (Virtual Instance based UTM Clustering) is going to survive in Service provider market. With this background, let me get back to the topic i.e challenges and solutions in Session based UTM clustering.

Technology description

At very high level, one of the devices becomes master and other devices are participants. Master device receives traffic from all ports. On first packet of any session, It decides whether the session to be handled by itself or can be load balanced. In case of load balancing, it decides the device based on load balancing algorithm such as hash on source IP/destination IP or least loaded etc.. Further packets of the session are given to appropriate device. I am calling this functionality as 'Cluster plane'. Latency of the packet increases if the packets are handled by participants - Packet comes to master device first and handed over to participant device.

I believe clustering solutions should not call for multiple set of IP addresses - with each set assigned to a device. All devices in the cluster should share same IP addresses.


Challenges & Solutions

Cluster plane should not require major modifications to existing UTM functions. It is understood that some changes are required to take advantage of clustering. As long as changes are limited, then there is more success of adaptation of clustering by vendors.

Some security functions don't work when sessions are load balanced blindly based on load balancing algorithms. Special care should be taken to handle these scenarios. I tried to describe these scenario to some extent to give an idea of the challenges. I try to give possible solution to these problems. In some cases I try to describe the solutions which are balance between complexity of implementation versus limitations.

Inter dependency among sessions: Whenever there is dependency among sessions, all the sessions must be handed over to the same device. Some of the scenarios are:

• Firewall ALG Control and Data connections : Each application context has multiple 5 tuple sessions. These are called complex protocols/applications. These typically require special module in firewall called ALGs which create pin holes and do NAT translation of IP addresses in the payload. Firewalls maintain some state information across these sessions and they are dependent. In these cases, it is required that clustering ensures that these dependent sessions are assigned to same device.
• NAPT functionality: Outbound sessions are typically NATted with one or few IP addresses. New