Are Multicore processors Linux friendly?

Multicore processors have become the trend for past three years.  Intel, Cavium and RMI have Multicore processors.

But are they Linux friendly?

Since Linux has SMP functionality, they are in theory Linux friendly.  And they are to large extent as long as Kernel controls the hardware accelerators and Ethernet ports. 

There are many reasons why Network device vendors don't like to deal with the hardware from Kernel space. Let us take the two accelerators in question - Regex accelerator and Crypto accelerator.  Any device vendor providing security functionality mostly implement using Linux user space daemons for several reasons such as

• User space programming is easier to debug.
• Some security functions require proxies which work on top of socket interface - hence user space applications.
• More memory can be accessed from user space.
• Taking advantage of SWAP.

Any application will not be limited to one user space daemon. There could be many user space daemons for typical network infrastructure device. Also, there could be processes which are created dynamically to take more load.  Let us take UTM as an example:

• IDS/IPS - One user space process, multiple threads.
• Network AV - HTTP/S Proxy -  One user space process, multiple threads.
• Network AV - SMTP/S proxy - One user space process, multiple threads.
• Network AV - POP3/S Proxy - One user space process, multiple threads.
• Network AV - IMAP/S proxy - One user space process, multiple threads.
• ClamAV or any equivalent Anti Virus package -  Multiple user processes created dynamically at run time.
• Spam Assasin or equivalent Anti Spam package - Multiple user processes.
• IPsec :  Kernel level function.

Crypto acceleration is required by 

•  Network AV - HTTPS proxy,  SMTP/S Proxy, POP3S Proxy and IMAP/S proxy.
•  IPsec in kernel space.

Regex acceleration is required by:

• ClamAV daemons.
• Spam Assasin daemon
• IDS/IPS daemon
• Content Security daemon (HTTP Proxy).

To improve performance and also to have isolation, many vendors would like to deal with the hardware directly from user space without Kernel doing de-multiplexing of requests/responses. That is, the accelerator device need to be shared by multiple daemons. Each daemon should be looking at its own copy of accelerator.   If one process dies, it should not affect other devices.

Unfortunately, the Multicore processors today don't have that capability.  I hope new Multicore processors would have this capability. 

Let us see the expectations by the software on hardware accelerator devices.

• Accelerator device should be instantiable.
• Each instance can be memory mappable by the appropriate user space daemon.
• Only owned process/thread should be able to submit and get hold of responses.
• Each instance of hardware device should have its own interrupt and this interrupt line should wake up the appropriate thread.
• When a user process dies, it should not affect other processes using the same device with different instance. 
• When the user process dies, software should be able to stop the instance of device.

Intel and VIA implemented crypto as instructions and hence they may not have above issues. But many Multicore processors implemented them asynchronously and would have issues if they don't have support for 'instances'.

Comments?

Techniques to defend aganist DNS Cache Poisoning attacks

This subject is covered very well in many forums, IETF drafts and RFCs. My purpose of this article is to provide some idea on how network security devices can play a role in defending the DNS cache poisoning attacks. Before that, let me give some background.

Many DNS attacks, discussed recently are on DNS Caching Servers, DNS resolvers and there are less number of attacks on Zone Authority DNS Servers. Attacker sends DNS response packets to DNS Caching Servers before authoritative DNS Servers respond to queries sent by DNS Caching Servers, there by poisoning the cache with IP addresses for domain names of their choosing. Typically DNS Caching servers are located in company premesis and ISPs. These DNS Servers cache the responses until TTL expires. When query is raised to these caching serves, they reply immediately with response if it corresponding entry is present in its cache. If not, it sends the query to some other pre-defined uplink DNS Servers or to the authorative DNS Servers corresponding to the domain in question.

Attackers send the DNS response as if it was from uplink DNS Server or authoratative DNS Servers. Since DNS works on UDP, it is quite easy for attacker to spoof the DNS response. The difficulty in making the attack successful by attacker lies in making the response acceptable by the Caching Server. DNS Caching Servers, typically accept the response only:

• they have sent the query.
• response contains the same transaction ID as the query it had sent.
• response contains the destination port same as the source port it used to send the query.
• response contains the source port same as the destination port it used to send the query.
• response contains the source IP same as the destination IP it used to send the query.

But, the attackers seem to be able to penetrate above defenses. According to recent vulnerabiltiy reports, many DNS Caching servers randomize the transaction ID already, but it seems that it is not good enough. One of the syggestions to defend against this attack is to make source port of the query also random. It helps in making attacker life difficult as it requires more responses and hence time. It appears that many DNS Caching servers and DNS resolvers patched their software and randomizing the source port. That is good news. But, it appears that even this defense is broken. See this article here: http://tservice.net.ru/~s0mbre/blog/2008/08/08. As seen in the blog, it takes more time, but attacker can make the exploit successful.

Many DNS Caching servers implemented some additional defeneses such as:

• Additional corelation checks between Query and responses, such as checking for "Domain name" in Query section matches with Query and Answer section in the response : Many attackers are getting around this defense by sending their own queries with arbitrary domain names as part of queries and sending same in the DNS responses . By sending queries themseleves gives them even more control. That is, they need not wait for subscribers of DNS Caching Server to intiate the query, thus greatly increasing the chance of cache poison. Attackers themselves are sending the query with unknown subdomains which certainly triggers the query from the Caching server. Attacker can time their responses right after sending the query with un-resolvable and randomly created domain names. One might ask, how is Cache is getting poisoned if attackers are sending random subdomain name. At best it is kind of DoS attack. From the recent attack exploit scripts (script1 and script2), one can understand that cache poisoning is not happening with the information present in the answer section of DNS response, but using NS Resource Records (Athority section) and Additional Resource records of DNS response. Attackers are ensuring that the Domain name in Question section in both query and response is same. To understand more about this kind of attack, read 2.3 section (Name Chaining) of RFC3833.txt
  
• Preventing from Birthday attacks - Again this defense is being overcome by sending random domain names in each query by attackers.
• Ensuring that the NS record entry has some portion of the domain name in the Question section to make sure that arbitrary NS record is not honored: This kind of defense also is being broken by attackers. Attacker are actually creating full domain name in Question section with random string followed by the victim domain name and creating response with victim domain name in Authority section. Thereby, the defense is being bypassed by attacker. For example, if attacker wanted to serve his/her own IP addresses for www.veryimportantsite.com, then the type of queries and responses he/she sends to victim caching server look like this:

-----------------------------------------------------------------------
Query:
QUESTION SECTION:
.www.veryimportantsite.com. IN A

Response:

Question section is same as Query.

AUTHORITY SECTION:
www.veryimportantsite.com. 6000 IN NS attacker.veryimportantsite.com.

;; ADDITIONAL SECTION:
attacker.veryimportantsite.com. 6000 IN A 2.3.4.5

----------------------------------------------------------------------------

As you see above, defenses used by DNS Caching servers are not going to work. In above case, DNS Caching Server is going to use 2.3.4.5 as the Authoritative DNS Server for www.veryimporantsite.com domain name. Any queries to this caching server for domain names www.veryimportant.com and any other subdomains within it, 2.3.4.5 server is contacted. Since this server is hosted by attacker, he/she can choose to provide IP addresses for victim domain names. Basically, in this case, NS List is corrupted with some poisoned entry. Replace veryimportantsite.com with google.com, then it will have devastating effect if the DNS Caching server is one of popular ISPs' servers. Many users behind this ISP will be directed to site of attackers' choosing when they visit www.google.com or any subdomain under www.google.com.

Many people agree that the right solution is getting rid of UDP and use DNS over TCP or use DNSSEC. It is going to take time for everybody to adopt this. Until then DNS Caching Servers need to improve their randomization and security administrators install security devices to prevent the attack or get early signs of attempts.

How can security devices help?

Due to the stateful nature of network security devices, DNS response packet is not accepted by the firewall functionality of devices if 5 tuples of response don't match up with 5 tuples of DNS query. That is, DNS response packet must have its SIP equal to DIP of the query, DIP equal to SIP, SP equal to DP and DP euqal to SP. In addition, many firewall devices also check for matching transaction ID. One might think that Firewall is not adding value here since many DNS Caching servers are already doing these checks. But firewall goes one step further. It only allows DNS response from secuirty zone to which DNS query is sent. Also, it is good practice to set up the rules to allow DNS traffic as needed. Typically in Enterprises, only outbound DNS is allowed. If there are Authoritative DNS Servers in side, DNS requests are allowed only to these particular servers. This will reduce the probability of successful poisoning attacks. Let us analyze.

Assume that, a company has DNS Caching Server in its "Intranet-DMZ" zone and ISP DNS Server, ofcourse, is in untrusted (External) zone. Enterprise administartor creates a rule 'Intranet DMZ Zone' to 'External Zone' on Destination Port 53 Destination IP as ISP DNS Server with "Allow" action. Due to this rule, any DNS queries from the attacker will be dropped. So, the attacker only needs to depend on queries generated by DNS Caching Server due to genuine queries from its local users. That dramatically reduce the success of the attack. If there are internal attackers, they need to send both queries and responses.

Some firewall devices come with local DNS Caching Server (DNS Resolvers) functionality. In these cases, it should take same precautions such as randomizing the source port and transaction ID while sending DNS query to uplink DNS Servers.

I feel that security devices implementing firewall and IPS can add more defensive measures such as:

1. Many DNS resolves and Caching servers are already enhanced and donot send DNS queries on persistent sessions. That is, Each DNS query is sent with different randomized source port. Security devices can remove the session once the corresponding DNS response is received. Then count the number of DNS responses received for which there is no session entry.

As you observe from the attack scripts, for every query, large number of responses are sent by the attacker. By having some logic on counting the number of orphan responses in a particular quantum of time indicates that some attacker is trying to exploit DNS Caching Servers. Security devices can warn the administartors for further analysis.

2. Once the attack attempt is detected, security device can send one more query with the same information as in the original query and match the responses. If the content looks similar, then it can be cached and send response to the original querier. Ofcoruse, this requires some DNS resolution functionlity in the security device. Note that this functionality can also be implemented within the DNS Caching Servers too.

3. Security devices implementing NAT should not make the guessing of ID and source port easier. When NAT port is selected, it must ensure that it is random port.

Data Center Firewall features

What makes firewall a good data center firewall?

Before going further on what features are expected by data center IT/Security professionals, it is good to revisit the data centers. Data center providers are mainly hosting providers. They host their customer applications and machines. Some customers of data centers share a machine resource, some like to host their application in a virtual system and some like to host their applications in a dedicated machine(s)/blades. To provide availability and share the load, application servers are installed in multiple machines with "load balancers" distributing the load across the server farm. As we all know, HTTP/HTTPS servers by far the single most server application in data centers. Most of the times, services provided by hosted servers are meant for general public.

Increasingly, there is a trend by Enterprises offloading hosting of Intranet servers to external data center providers. Intranet servers are typically provide access to Employees and limited access to their partners. For example, many email services, sharepoint and wikis are being offloaded to data center providers by many small and medium Enterprises. Many of these services require user authentication. Enterprises don't like to duplicate the user databases in multiple machines/applications. So, you also see the trend of 'Central Authentication Database' across internal servers and servers hosted outside. Many web applications are providing SAML based authentication for federate identity. Since web services need to talk to outside identity providers, there can be outbound connections. Note that, traditionally, servers in data centers only see inbound connections.

Enterprise administrator also requires facilities to upload the content and do other administrative activities on hosted servers. Typically FTP, SSH are some of the services required by administrators. Some applications might have web interface running on Port 80/443 for administration. To provide added security beyond user authentication, data center providers likes to control admin access from particular network(s), typically Enterprise Networks.

With more and more services (both Intranet and Extranet) being hosted in external data centers, the need for securing them is high. Collaborative services/servers such as wikis, share point, CRMs and other work flow servers are typically used to be part of Enterprise networks and only accessible for local users. They are being hosted in external data centers for reasons such as providing access from anywhere for employees, partners, contractors etc.. and also reduce the administration headache. Since they are exposed to access from anywhere, they are open for attacks from attackers. So, need for detection and prevention of exploits becomes much more than what data centers are used to. Quick look at the vulnerabilities published by NIST indicates (nvd.nist.gov) that SQL/XSS/LFI/RFI injections are on rise. You can also see number of wikis, blogs and other collaborative applications are targets of attackers.

Intranet servers when placed in external hosting providers' network, Enterprises would like the communication channels to be secure to protect data from eaves dropping. HTTP over SSL/TLS is one common method used to achieve data confidentiality on the wire. For security devices, placed outside of these servers, to do better job of access control, intrusion detection and malicious injections, it is necessary for these devices to see the traffic in clear. To achieve this, security devices should have capability to decrypt the SSL and do traffic/data analysis and if required redo the SSL. By the way, Since security devices are expected to be kept right in front of the servers, there may not be any need for redoing SSL. But important take way is that the security device should have capability to terminate the SSL connections.

From last few years, many web applications are using SOA (Service Oriented Architecture) which is built upon XML standards. Traditional ways of plain POST requests, JSON and PHP Objects are fast becoming thing of past. Any security device doing intrusion and data analysis need to move beyond POST, JSON and PHP objects and start interpreting SOAP and XML.

Data center providers provide services to many customers. Each customer requirement from security perspective is different. One generic security policy does not fit in these environments. You could have as many firewalls as number of customers, but that is not scalable from cost, space and cooling perspective. Virtualization in firewall/security devices comes in handy. Virtualization with VMWare/Xen also does not scale well. Old traditional virtualization scales well and suites well for data center providers.

Since security device comes in the way of traffic, things like performance of security devices should be high to support traffic rate that can be processed by servers/services it is securing. Latency, stability, availability and failover capabilities are some more important factors data center providers consider while selecting the security devices.

With above background, it is very easy to map to the features expected by data center providers on security device protecting their application and server infrastructure.

• Access Control : As you see above, access control some times need to go beyond IP addresses and TCP/UDP ports. Some web applications provide administrator and normal user access via same TCP/UDP port. Hence it is not possible to distinguish administrator and normal users from IP addresses and ports. Since many data center providers don't like admin access to be given from any IP address (for providing better security), but from some specific networks, it is required that the access control go beyond to application level information such as URL, Query parameters etc..
  
• Intrusion Detection and Prevention at L3-L7: As explained above, typical traditional intrusion detection systems without web application intelligence will not be able to detect intrusions all the time. There are many evasions being employed by attackers. Some evasions are at the IP and TCP level and more evasions are at the HTTP protocol level. Hence protocol intelligence is required. In addition, with SOA based web services, intrusion detection systems need to have intelligence to extract data from SOAP/XML messages. In addition to web application intelligence, they also need to have intelligence of other common services provided by hosting providers such as DNS, FTP, SIP etc..
  
• SSL Proxy: Network device should be able to terminate the SSL for further analysis on the protocol data.
• Virtualization: One physical hardware box is expected to support multiple virtual instances to reduce number of security devices in the deployment. Each virtual instance would need to have its own security policy configuration. It should be as good as different physical firewall devices. I, personally don't prefer VMware/Xen/KVM based virtualization for these environments. I prefer Traditional virtualization where only configuration data and run time states are instantiated for every context.
• DDOS attack detection and prevention.
• Traffic Anomaly detection and traffic Control.
  
• Performance: To achieve multi gigabit speeds, look for hardware architecture which is scalable.
  
• Stateful failover and high availability
• Logging & Auditing capabilities
  
• Intuitive central Management system

Optional features: Though they are not required, some data centers might find them useful

• Server side NAC: Provide facility for user based access control. NAC does user authentication and provides control access to the different features of an application based on the URL and other fields in the protocol. It also helps in correlating user actions and might be useful in auditing.
  

My intent here is not to go into many details, but provide some ideas on the features security vendors would need to think while providing security device solutions to data center market.

Guidelines for defining data models

TR-106 defines data model guidelines for creating new data models. For interoperability and to provide better clarity, I have defined some more guidelines.

Before you proceed further, please read this.

Guidelines:

Guidelines that are being followed today in defining data models.

• Data type: such as string, base64, integer, unsigned integer, Boolean, char etc..
• Range: In case of integer, unsigned integer values.
• Enumerations: Set of values that this parameter can take. Valid for both integers and strings.
• Min/Max length of string: In case of string and base64 types.
• Read/RW attributes for parameters
• Create/Delete for table nodes.

Additional guidelines:

General:

1. Values of some parameters don't reflect changes done by ACS. These are called action parameters. Each action parameter should have associated result parameter. Action parameters are always strings and it takes value "apply". ACS sets this value to perform the action. These parameters should be defined with attribute called "ActionParameter". This attribute should be set to 1 for action parameters. Associated result parameter fully qualified name should also be set with attribute "AssociatedResultParameterName".
2. For each associated action parameter, there should be result parameter. Result parameters are always strings. "success", "not available" are two values defined by this guidelines. Any other string value is considered as error in performing the action by application. Possible error strings are defined by applications defining the data model. Whenever action parameter value is set, associated result string value should get automatically set to "not available". When the application processing is done successfully, then this parameter value would be set to "success" by device. If application processing returns error, appropriate string value is set to this parameter.
3. Always define default values for all parameters except for mandatory parameters of table objects.

Table objects:

1. Identify the parameter that uniquely identifies the instance in the table object. Indicate that in an attribute "KeyParameter". ACS can check for the uniqueness of this value whenever the instance is created.
2. For each parameter that can't be changed after instance is created, indicate in DM that "DontChangeAfterCreate". Key parameter of the table object typically has this attribute set. There could be other parameters too based on application.
3. Identify all parameters that are mandatory during instance creation. Use "Mandatory" attribute for these parameters. ACS can ensure to send values for all mandatory parameters in "SetParameterValues" method that follows "AddObject" RPC method.
4. Indicate the default value in the data model if the parameter is not mandatory. This information is needed when ACS user asks it to reset the some specific optional parameter to default value.
5. For every table object, one parameter "Number Of Entries" must be present as per TR-106. I suggest to have one parameter "Number off Entries Supported" to represent the number of rows the device can hold. Both these parameters take integer values and both of them are READONLY. Note that names of the parameters could be different for different data models (table nodes). Hence there is need to associate these parameters to appropriate table nodes. To associate these parameters for the table object (table node), these parameter should have attributes associated with it indicating the full qualified table object. I call this attribute as "AssociatedTableNode". In addition to this attribute, two more attributes are needed - "CurrentEntriesIndicator", "MaxNumEntriesIndicator". CurrentEntriesIndicator attribute should be set to parameters indicating the current number of entries and MaxNumEntriesIndicator attribute should be set to parameters indicating the maximum number of entries that device can take for corresponding table.
6. If table is representing the ordered list, then it should have special attribute for the table object called "OrderList". Also it should have associated parameter name that indicate the priority of the ordered list. Call this attribute as "PriorityParameterName". This information is used by ACS as described in previous blog here. Priority parameter name takes integer values. Lower values indicate higher priority instances.
7. If table is an ordered list, it should also have one pair of action and result parameters for revalidation purposes. In addition to attributes it defined earlier for these two parameters, it should have additional parameter representing the table object name with attribute "AssociatedTableNode". ACS uses these attributes to know the parameters names to revalidate the states in the device.
8. As described under section "Nested table objects and special cases" of earlier blog here, there is a need for creating a table object instance with at least one instance of child table. ACS needs to know this special relationship so that it generates the screens such a way that it takes at least one child instance parameters as part of parent instance creation screen. This relationship information also useful for ACS to validate this information before sending configuration to devices. Each child table of this type should have one attribute "OneInstanceNeededByParent" set to 1.

Jericho forum - Is network security device market dead?

In one of the meetings I participated few weeks earlier, one person asked me a very interesting question - Will there be any security devices market in future? When I asked him why that question, he referred me to Jericho forum. Though I have some idea about jericho forum before, it got me interested to know more details about this.

When I first browsed through the forum publications, I thought that the question that was asked was fair. At first glance, it appeared that Jericho forum is proposing to add security along with application and data. But after spending few hours on position papers, Brochure and FAQ, it appears that Forum is not advising people to throw away their firewalls and security devices, but enhance security down to applications, data. Having said that, position papers still confuse readers with some inconsistent statements. I think that Jericho forum did not position their security concerns and resulting architecture very well and hence the confusion and mis-characterization in security industry.

Jericho forum described two main challenges - Business transactions that tunnel over HTTP/HTTPS and exploits/malware escaping traditional firewalls/security devices. I add one more challenge beyond HTTPS, that is, data itself may not be in clear - either it is encoded, encrypted or compressed.

It is true that traditional network address/service level firewalls are not good enough to protect resources from data level attacks and data misuse. Many applications are being developed on top of Port 80/443 (HTTP/HTTPS). Web Services (SOA) architecture is being used to develop multiple applications on a single machine with HTTP/HTTPS as transport. Any application service level filtering is possible only by devices having HTTP/HTTPS and web services intelligence.

It is also true that many newer malwares evade traditional signature detection - either by sending malware executables via HTTPS or constantly morphing themselves to avoid detection. One of the techniques behavioral analysis requires gathering the run time information such as registry entry modifications, listening port, any outbound connections, files being modified etc.. by running the executable on appropriate operating system.

With the challenges described and positioning it is doing, first impression I got was that Jericho forum is advocating adding entire security along with each application in the same machine. It took me a while to get rid of this impression. I guess the term "De-perimeterization' is confusing. I would like to think that Jericho Forum is proposing that security at Enterprise boundary is not good enough and the security is, additionally, needed closer to the applications/resources. So, there are multiple perimeters, with some perimeter having few machines or even one machine or one application. By the way, traditional firewalls and Ipsec VPN devices do very good job of providing access control to desktop systems based on the type of user and provide security connectivity to other branches of organization.

Though adding all security functions along with the application on the same machine provides better security, there are complexities:

There could be multiple machines running same application in cluster mode. In some deployments, it is observed that 100s of machines are used to share the load. In those, it is wise to move security functions such as "L4-L7 access control", "Intrusion Detection functions" to specialized security devices. It saves CPU cycles on application servers. It provides single control for administrators to manage security functions of the applications or set of servers and hence the management becomes easier. Some security functions such as terminating wireless connectivity and mobile device management don't really belong to one specific LOB (Line Of Business) application. They need to be outside of the application servers.

Having said that, some security functions can't be done well outside the LOB machines such as behavioral detection of malwares or when the data is encrypted or compressed with proprietary algorithms. They are better done as in end systems.

There is cost to apply some security functions outside the LOB servers. For example, Many LOB Servers implement security protocols such as SSL, XML Security etc.. Any access control device providing control at the XML field level must terminate the SSL connection, authenticate the user and decrypt and validate the XML documents before doing access control. There is inherent benefit too - It saves CPU cycles on the LOB Servers as it sees clear traffic. But, there may be some concerns in CSOs that some network elements has access to the clear data. If it is micro perimeter, then there may not be any concern. I guess Jericho forum is driving this point where the security perimeter is as close to the applications and data.

Security device vendors would like to make their solutions as generic as possible. They don't like to tie up the device functionality to one or few applications. That is where, standardization helps. I am happy to see that Jericho forum in their COA (Collaboration Oriented Architecture) position paper, chose the SOA and XACML. Both of these architecture heavily dependent on XML messaging. It provides common understanding for network elements outside of LOB servers and there by creating eco-system of vendors comprising security vendors, application vendors.

Having said that, I feel that the LOB applications must have their own security based on the application - Such as authentication, multiple roles, role based access, Auditing etc..

In summary, CSOs need to understand that Enterprise boundary security with traditional network level firewalls is not good enough to protect the data and resources. Application specific security is must. Some security functions can be done outside of LOB servers, but the security device must be as close to the LOB servers as possible. So, I don't see network security device vendor market drying up.

OpenDNS - Domain filtering Cloud computing Service

I recently came across *opendns* service. Visit www.opendns.com to find out more details on this service. "opendns" name is a confusing name given the type of service they are providing. Initially, I thought it is some thing similar to dynamic DNS.

Operation:
OpenDNS mainly provides domain blocking capability. Domains are arranged in multiple categories. It provides facility for users to configure the categories which are to be blocked. It also provides facility for users to create white list and block lists of domain names.

This service is using DNS protocol. It expects the user machines or routers to use their DNS Servers for domain name resolution. As part of DNS resolution, it appears to be extracting the domain name from DNS request packet, search in their local database, get the category and look in user preferences. If category is configured to be blocked or if domain is in the block list, then openDNS server seems to sending DNS response with its own IP address. Due to this, user browser session ends on this IP address. OpenDNS seems to be doing search on the domain name (Host field of HTTP request header) again to determine the category and it shows nice page indicating why it was blocked.

Comments:
This service is good for residential users and even for business users. Residential users get benefited by blocking adult sites for kids and also stop while visiting phishing sites. Businesses also benefit as it stop users going to phishing sites. Having said that, this works fine only when CPE devices work in conjunction with openDNS service. Before going into the capabilities required in CPE devices, let me list down some limitations/issues in using opendns service.

• Privacy issues: Some businesses find it difficult to trust opendns provider due to privacy issues where *openDNS* provider comes to know the sites business users visiting. Business may like to have facility for some users to bypass this service and for some mandate this service. Also, businesses like to have facility to bypass openDNS based Domain name resolutions for some specific domain names.
  
• User or group based white list/block lists/category selection: There are different types of employees in businesses. Also, there are different types of home users - kids, parents, visitors, teens etc.. OpenDNS provides only one profile for all users. This may not be sufficient for many businesses and residential users.
• Evasion: Kids can evade these filters if they use IP address in their browsers.
• Updating Dynamic Public IP address with the opendns account
  

How can CPEs help?

User/Group based lists: User/Group based lists support is only possible if openDNS updates its functionality. One possible way is to have special DNS request with added information such as GroupID. OpenDNS Service can rovide facility in openDNS portal to create category selection/blocklist/whitelist onper group basis. Since one can't expect all PCs to support this special enhancement in the DNS protocol, this kind of support is possible with CPEs implementing DNS proxy to convert DNS requests to add Group ID.

Privacy: CPE devices can help in mitigating privacy issues by providing support to create 'skip' lists - Source skip list and Domain skip list. If the source IP address of the DNS request packet from internal PCs matches the entry in 'Source Skip' list, then it bypasses openDNS based resolution. It can do this by sending the DNS request to one of ISP Domain Name Servers. 'Domain Skip list' is checked for domain names inside DNS request sent by local machines. If there is a match, then it bypasses the openDNS resolution.

Evasion: CPE devices can monitor HTTP requests and check the 'Host' header line. If the 'Host' header line does not have domain name, but IP address, then we can certainly say that domain name is not used while browsing the site. CPE devices can provide configuration on type of action to take. It can provide options like 'Inform' and 'Deny'. 'Inform' action informs parent in case of RG environment or admin in case of business environment. 'Deny' action drops the connection and might even present local HTML error page to the user. Here too, we should 'skip' lists to help scenarios where some sites are only reachable via IP addresses - for example Intranet sites or partner sites etc..

There is another kind of evasion possible too. Local users using their own DNS Server or some public DNS Servers. CPE can check all DNS requests and ensure that only specified DNS Servers are used. It could even do Destination NAT to the required DNS Server IP address.

Dynamic IP address update: Today it is expected that special program is run in the PCs behind CPE routers. It does not work well if we have many machines or machines which do not run the software provided by OpenDNS folks. CPE device can help in those matters where it updates the dynamic public IP in openDNS Servers. CPE devices are already equipped with updating dynamic public IP addresses in DYNDNS servers. They could do additional job of upating in openDNS Server too.

SME IPS and Cloud Computing

Cloud computing providers are betting on small and medium businesses flocking to them. Large number of SME businesses are already using email service provided by cloud computing providers. It appears that this trend is being spread to other services such as File Service, backup service and web application services.

Businesses offloading their intranet and extranet services to the providers would be left with desktops and some minimal servers in their network. I have my own doubts on merits of moving Intranet services to providers, but that discussion belongs to some other topic.

Desktops normally don't provide any services i.e they don't run any servers. May be printers and other networking equipment have some services, but they are limited to internal machines. Hence firewall protection allowing only internal machines is good enough.

Basically, the requirement of server side security function beyond firewall is going to be less in these environments. In addition, many hackers are now moving towards soft targets i.e desktops and applications running on desktops such as browsers, viewers etc..

Many IPS/IDS devices in the market today protect servers better than clients. Due to movement of services to providers and with increase of client side attacks, IDS/IPS vendors must support better client side detection to survive.

IDS/IPS vendors realized this and moving towards this, but not as fast as one would like to see. By Mid-2009, I believe that many IDS/IPS boxes in the market will have sophisticated engines to support client side attack detection and prevention.

Proxy based Networking applications - Multicore and developer considerations

Many networking applications such as Anti Virus, Anti Spam,  IPS  to name few are being implemented as proxies in network infrastructure devices.  Proxy implementations terminate the connections from external clients and make connection to the destination servers. They get hold of data from client and server and send it to other end after doing applications specific processing.  To ease the development, proxies are implemented on top of sockets in user space.  For each client to server connection,  two socket descriptors would be created - One socket is created as part of accepting the connection from client and another socket is created as part of making connection to the server. 

In unicore processors, it is typical practice to have one process per proxy.  Process handles multiple connections using non-blocking sockets. It is done typically using either poll() and epoll_wait()  mechanisms.  This allows the process to work on multiple connections at the same time.

Many networking applications listed above use the proxies in transparent mode.  Transparent proxies avoid any changes in the end client and server applications and also avoid any changes to the DNS Server configuration.  Systems with transparent proxies intercept the packets from both client and servers.  It is expected that forwarding layer (of Linux ) of the system  to intercept the packets and redirect the packets to the proxies.  Redirection typically happens by overwriting the IP addresses and TCP/UDP ports of the packet such a way that the packets go to the proxies running in user space without any making any changes to the TCP or UDP or any other stack component by the developers.


Process skeleton looks some thing like this:

main()
{
     Initialization, Daemonize & configuration load.
     Create a listening socket.
     while(forever until termination)
     {
            epoll_wait();
            Do any timeout processing.
            for ( all ready socket descriptors )
            {
                 if ( listening socket)
                 {
                        accept();
                        Create application specific context.
                        Might initiate the server connection.
                        May add socket fds to epoll list.
                 }
                 If socket is ready with new data
                 {
                      Application specific processing();
                     As part of this, the oscket fd may get added to the epoll list again.
                 }
                 if ( socket is has space to send more data )
                {
                      Application specific processing() which sends the data.
                      If more data to be sent, socket might be kept again in epoll list.
                }
                if ( socket has exception )
                {
                        Application specific processing ();
                        Connection may be closed as part of application processing.
                }
  
            }          
     }
     Do any graceful shutdown activities.
     exit(0);
}

Increasingly Multicore processors are being used in network infrastructure devices to increase the performance of solution.  Linux SMP is one of the popular operating system choice by developers.  What are the things to be considered while moving to Multicore processors?

Usage of POSIX compliant threads in the proxy processes:
Create as many threads as number of cores in the processor.  Core affinity can be done for each thread.  Yet times one might like to create more number of threads beyond number of cores in processor to take advantage of asynchronous accelerators such as  Symmetric & Public key crypto acceleration,  regular expression search acceleration etc..  In those cases, thread might wait for the response.  To allow the core to do other connections processing ,  multiple threads per core are required. 

Mapping of thread to the application processing:  

Developers use different techniques.  Some use pipelining model. In this model, one thread gets hold of the packets for all connections and pass the packets to next thread in the pipeline for further processing.  Last thread in the pipeline sends the packet out on other socket.  Though this might use multiple cores in the processors, this may not be ideal choice for all applications.  I feel run-to-completion model is good choice for many applications.  Run-to-completion model is simple. Each thread waits for the connections. Once the connection is accepted, it does everything related to the applications in the proxy including the sending out the processed data.   The structure is similar to the process model, but the loop is executed by each thread.  That is, connections get shared across the threads with each thread processing set of connections.  Advantages which this approach are:

• Better utilization of dedicated caches in the cores.
• No or less number of Mutex operations as one thread does all processing.
• Less number of context switches.
• Less latency as it avoids multiple enque/deque operations to pass packets from one pipeline stage to another.

Load balancing of the incoming connections to the threads: 

There could be multiple ways of doing this.  One way is to let master thread accept the connections and give to one of the working threads to do rest of processing on the connection.  Master thread can use different load balancing techniques to assign the new connection to least loaded thread.  Though this approach is simple and confined to the process,  it has some drawbacks.  When there are shortlived, but large number of connections being established and terminated in quick succession,  master thread can become the bottleneck.  Also, cache utilization may not be that good as master and worker threads might be running on different cores.  Since master thread is not expected to do much other than accepting the connections,  a core may not be dedicated, that is, core may not be affined with the master thread. 

Another technique that can be used to let each thread listen on its own socket, accept the connections and process them.  As we know, we can't have more than one listen socket with respect to IP address and port combination.  So, this techniques uses multiple ports as many as number of threads. Each threads listens on a socket created with unique port.  It should be noted that external clients should not be knowing about this.  The client connections to the server will always be with one standard port.  Hence this technique requires additional feature in the intercept layer.  Intercept layer (typically in the Kernel in case of Linux) is already expected to do IP address translation to ensure the packets are redirected to the proxies.  In addition it can do port translation too. Port to translate with can be found based on the load on each port. For example, the port selection can be 'round-robin' or it could be based on 'least' number of connections on the ports.

Do all relevant application processing in one proxy:

Network infrastructure devices are complex. Yet times, on the same connection multiple application processing is required. For example,  on HTTP connection, the device may be expected do 'HTTP Acceleration such as compression', TCP Acceleration and 'Attack checks'.   If these are implemented in different processes as proxies, latency would increase dramatically as the each proxy terminates and makes new connection to next proxy. Also performance of the system goes down. Certainly it has one advantage, that is, maintainability of the code. But performance wise, it is good to do all applications processing in one single process/thread context. 

Considerations for choosing Multicore processor

Certainly cost is the factor. Besides the cost, other things to look for are -

• Frequency of core is very important:  As discussed above, a connection is handled by one thread.  Since thread can be executed in one core context at any time, performance of the connection is proportional to processor frequency (speed).  For proxy based applications,  higher frequency cores are better choice compared to multiple low powered cores.
• Cache :  Proxy based applications typically do lot more processing than typical per-packet based applications such as Firewall, IPSec VPN etc..   If the cache size higher, more instruction memory can be cached.   So, higher the cache size, better the performance. 
• Division of cache across cores:  Since threads can be affined with the cores, it is good to ensure that the data cache is not same across the cores. Any facility to divide the shared cache into core specific cache would be preferrable.
• Memory mapping of accelerator devices into the process virtual memory:  By having access to the hardware accelerator from the user space, one can avoid memory copies between user space and kernel space.
• Hardware based connection distribution across cores :  This is to ensure that the traffic is distributed across cores. Intercepting software in Kernel forwarding layer need not make any load balancing decisions to distribute the traffic across threads.  Intercept layer only need to translate the port so that the packets go to the right thread.

Other important considerations that are needed for any applications are:

• Facility in hardware to prioritize the management traffic at ingress level : To ensure that Management application is always accessible even when devices is under flood attack.
• Congestion Management in hardware at ingress level:  To ensure that buffers are not exhausted by application that do lot of processing.
• Hardware acceleration for crypto,  regular expressions and comperssion/uncompression.

Programming considerations for performance

• Each poll() or epoll_wait() calls are expensive, so avoid calling epoll_wait() as much as possible : Once the epoll_wait comes out, read the data from the ready socket as much as possible. Similarly write data as much as possible on the ready sockets.
• Avoid locking as much as possible.
• Avoid pipelining - Adopt run to completion model.

I hope it helps developers who would be developing proxy based applications.

Hardware timer block in Multicore processors for network infrastructure devices

Some use case scenarios of timers for different functions of network infrastructure devices is given here.  

One of the main challenges with software timers is to ensure that jitter and latency of the packets don't go up during the period when some timer block related operations occur.  Latency of the packets or even packet drop happens when CPU takes too long a time to process some timer block related functions.  Any timer block functions that go through the timers in a tight loop would have affect on packet processing if the number of timer elements checked or acted on in the tight loop are more.  The threshold of number of elements that are checked in the tight loop that causes packet latency disruption depends on the frequency of CPU. Based on the software timer block implementation,  traversal of some timers happen for different operations.  Let us see some of the challenges/problems with software timer modules.

• Software timers depend on hardware timer interrupt. In Linux, timer interrupt occurs fore very jiffy( typically 1msec or 2msec).  Due to this any software timer can  have error up to jiffy.  If applications requires  smaller error, say in terms of, micro seconds,  then only method I can think of is to have timer interrupt to occur in terms of microseconds.  This may not work in all processors.  There is too much of interrupt processing overhead in cores and reduces the performance of the system. Fortunately many Applications tolerate millisecond error in firing the timers, but some applications such as QoS scheduling on multi-gig links running general purpose operating systems such as Linux require finer granular and accurate timers.
• Many networking applications require large number of software timers as described in earlier post.  This will lead to traversing many timers on per jiffy basis.  For example, if an application creates 500K timers/sec, then there would be 500 timers on per jiffy basis. For every 1 millisecond,  it needs to traverse 500 timers and may have to fire all 500 of them. This can take significant amount of time based on the amount of time the application timer callback takes. If takes good amount of time to process, you have packet drop or increased packet latency or both the issues. Some software implementations maintain the timers on per core basis. If there are 8 cores,  each core may be processing 62 or 63 timers every millisecond.  This is ideal case, but what if the traffic workload is causing only few cores starting the timers. Only few cores would be loaded to process the expired timers.  Basically the load may not get balanced across the cores.
• To reduce the number of timers to traverse for every hardware timer interrupt,  cascaded timers wheels are normally used by software implementations. This implementation does have different timer wheels for different timer granularity and when the timers are started, they go to appropriate wheel and bucket. Due to this any bucket of timer wheel contains the timers that will get expired. Though it reduces the number of timers to traverse for every timer interrupt, but it may involve movement of large number of timers from one timer wheel to another as described in the earlier post. This movement of timers may take significant amount of time and again could be the cause for packet drop and increased latency.
•  If there are periodic timers or need to be restarted based on activity software timer implementation spend good amount of time in restaring them.

Do hardware timer blocks in Multi-core processors help?  

In my view hardware timer block can help when your applications demand large number of timers, periodic timers or very accurate timers.  If your application requires 'Zero Loss Throughput', then hardware block is going to help certainly as it takes away the CPU cycles used to traverse the timer list or movement of timers in software implementations.

What are the features expected by network infrastructure applications from hardware timer block in Multi-core processors?

• Large number of timers are expected to be supported, ranging in Millions. 
• Decent number of (say 1K) timer groups are expected to be supported.  There are multiple applications running in cores that require timers.  Applications that are being shutdown or that are being terminated due to some error conditions should be able to clear all the timers that it had started.
• Accessibility of timer groups by applications running in different execution contexts. There should be good isolation among timer groups. There should be some provision to program the number of timers that can be added to a timer group.  There should be provision to read the number of timers that are in the timer group.
• Applications running in Linux user space 
• Applications running in Kernel space.
• Applications running in virtual machines. 
• Application should be able to do following operations. All operations are expected to be completed synchronously.
• Start a new timer:  Application should be able to provide
• Timer identification : Timer Group & Unique timer identification within in the group.
• Timeout value (Microsecond granularity)
• One shot or periodic timer or inactivity timer
• Priority of timeout event (upon expiry) : This would help in prioritizing the timer events with respect to other events such as packets.
• If there are multiple ways or queues to provide the timer event upon expiry to the cores,  then application should be able to give its choice of way/queue as part of starting the timer. This would help in steering the timer event to specific core or distribute the timer events across cores.
• Stop existing timer :  Stopping the timer should free the timer as soon as possible.  One existing hardware implementation of timer block in Multi-core processor today has this problem. If the application is starting timers and stopping them in continuous fashion, eventually it runs out of memory and memory will get freed only upon actual timeout value of the timers. If the timeout of these timers are in tens of minutes,  then the memory is not released for minutes together.  Good hardware implementation of timer block should not have this exponent usage of memory in any situation.   Timer stop attributes typically involve
• Timer identification
• Restart the existing timer.
• Timer identification
• New timeout value
• Get hold of remaining time out value at any time, synchronously by giving 'Timer identification'
• Set the actvity on the timer - Should be very fast as applications might use this on per packet basis.

Firewall/NAT/ADC appliances targeting Large and Data center markets would greatly benefit from the Hardware based timer blocks.  All hardware timer blocks are not equally created. Hence check the functionality and efficacy of hardware implementation. 

• Measure the latency, packet drop and jitter of the packets over long time. One scenario that can be tested is given below.
• Without timers,  measure the throughput of 1M sessions by pumping traffic across all sessions using equipment such as IXIA or smartbits. Let us this throughput is B1. 
• Create 1M sessions, hence 1M timers with 10 minutes timeout value.
• Pump the traffic from IXIA or smartbits for 30 minutes.
• Check whether the throughput is almost same as B1 across all 30 minutes.  Also ensure that there is no packet drop or increase in latency of packets, specifically at 10, 20, 30 minute interval.
• Measure the memory usage:
• Do connection rate test with each connection inactivity timeout value 10 minutes.
• Ensure that upon TCP Reset or TCP FIN sequence the session is removed and hence timer is stopped.
• Continue this for 10 or more minutes.
• Ensure that the memory usage did not go up beyond reason. 
• Ensure that timers could be started successfully during the test.

Assurance of firewall availability for critical resources : TR-069 support

I guess I have been harping that network security devices are stateful in nature. Let me say that again here :-) They create session entries for 5 tuple connections. DDOS attacks can consume these resources. There are several techniques used to maximize firewall availability. Some of them I discussed before are - Session inactivity timeout functionality, TCP syn flood detection and Syn Cookie mechanism to prevent SYN floods and connection rate limiting.

Above techniques do not guarantee that legitimate connections are not dropped. Rate throttling feature does not distinguish from genuine connections to DDOS connections. But, some resources are very important and access to/from these resources must be made available all the time. That is, some assurance of firewall availability for these critical resources is required.

During DDOS attack and worms outbreak, systems in corporate network should have access to central virus database server to get newer virus updates. Even if some systems in corporate network are compromised and participating in DDOS attacks, other systems should continue to access critical resources while problem is being fixed. Similarly, access to corporate servers should be maximized during DDOS outbreak.

Though all issues can't be solved, enough facilities should be there for assurance of firewall availability for these critical accesses.

Many firewall today support feature called 'Session Reservation and Session Limits'. Using this feature, certain number of sessions can be reserved to individual machines/systems. This feature also limits the number of simultaneous sessions for some non-critical systems/machines.
One use case example: Let us say that a Medium Enterprise has 500 systems. Say that this company bought a UTM firewall with 50000 session entries. Administrator can reserve 20 sessions and limit 100 sessions for each PC. That is, 10000 entries are reserved. Rest of 40000 sessions are free for all. When all 40000 sessions are used up, then reserved sessions are available for PCs. Each PC can use its reserved 20 session entries. Thereby, when there is a DDOS attack, even after 40000 session entries are used, these PC continue to have access 20 more session entries. No other system can occupy these reserved sessions.

Session reservation database is set of rules. Each rule contains following information:

• Rule ID: Identification of the rule.
• Description: description of this record.
  
• IP Address information: IP addresses for which this rule applies. All the action information in this rule is specific to each IP address.
• Connection Direction: Outgoing or incoming. Indicates whether the sessions to be reserved for connections made by machines represented by 'IP addresses' or for connections terminated by these IP addresses. 'outgoing' indicates this rule is applied for connections originated and 'incoming' indicates whether this rule is applied for incoming connections.
• Zone : Indicates the zone ID. If 'Connection Direction' is outbound, then zone indicates the destination zone. If 'Connection Direction' is inbound, then zone indicates the source zone.
  
• ReserveCount: Number of sessions reserved for this rule.
  

Session Limits database also contains set of rules. Each rule contains following information:

• Rule ID: Identification of th rule.
• Description
• IP address Information: IP addresses for which this rule applies.
• Connection Direction: Outgoing or Incoming.
• Zone: Zone ID
• Limit Count: Number of maximum sessions for each of IP addresses.

TR-069 data profile:

• internetGatewayDevice.security.VirtualInstance.{i}.firewall.maxSessionReservationRules: R, unsigned Int
  
• internetGatewayDevice.security.VirtualInstance.{i}.firewall.maxSessionLimitRules R
  
• internetGatewayDevice.security.VirtualInstance.{i}.firewall.sessionReservations.{i} PC
• ruleID: RW, Unsinged Int, Value between 1 and maxSessionReservationRules.
• description: RW, String(128)
• ipAddressType: RW, String(32). It takes values such as 'immediate', 'ipobject'. Immediate indicates that IP addresses are given as values and 'ipobject' indicates the IP address information points to one of the IPObjects.
• ipAddresses: RW, String(64) - f the type is immediate, then it can be single IP address in dotted decimal form, subnet by providing network IP address and prefix in terms of number or range of IP addresses with '-' in between low and high values. If the type is 'ipobject', then it has one of ipobject names from internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.IPValueObject.{i} table or internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.IPFQDNObject.{i} table. 'any' is special value indicating all source IP values. Examples: 10.1.5.10 or 10.1.5.0/24 or 10.1.5.1-10.1.5.254
• connectionDirection: RW, String(16). It takes values 'outgoing', 'incoming'.
• zoneID: String(32), RW - One of the Zone IDs. It takes value of ZoneName from internetGatewayDevice.securityDomains.VirtualInstance.{i}.Zone.{i} table.
• reserveCount: RW, Unsigned Int.
• internetGatewayDevice.security.VirtualInstance.{i}.firewall.sessionLimits.{i} PC
• ruleID: RW, Unsinged Int, Value between 1 and maxSessionLimitRules.
• description: RW, String(128)
• ipAddressType: RW, String(32). It takes values such as 'immediate', 'ipobject'. Immediate indicates that IP addresses are given as values and 'ipobject' indicates the IP address information points to one of the IPObjects.
• ipAddresses: RW, String(64) - f the type is immediate, then it can be single IP address in dotted decimal form, subnet by providing network IP address and prefix in terms of number or range of IP addresses with '-' in between low and high values. If the type is 'ipobject', then it has one of ipobject names from internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.IPValueObject.{i} table or internetGatewayDevice.security.VirtualInstance.{i}.NetworkObjects.IPFQDNObject.{i} table. 'any' is special value indicating all source IP values. Examples: 10.1.5.10 or 10.1.5.0/24 or 10.1.5.1-10.1.5.254
• connectionDirection: RW, String(16). It takes values 'outgoing', 'incoming'.
• zoneID: String(32), RW - One of the Zone IDs. It takes value of ZoneName from internetGatewayDevice.securityDomains.VirtualInstance.{i}.Zone.{i} table.
• limitCount: RW, Unsigned Int.

Common mistakes by TR-069 CPE developers - Tips for testers

There are some mistakes which can go undetected for a long time. Testers and certification agencies need to watch out for those. I tried to list down some of those mistakes in TR-069 based CPE devices.

Problems associated with Instance numbers: It is expected that instance numbers are persistent across reboots. But, this does not seem to happen always. Typical problems observed are:

• Instance numbers are not stored in persistent memory: If instance numbers are not stored and retrieved from persistent memory, then CPE device may provide duplicate instance number next time when it comes up for AddObject RPC methods. ACS may reject and new records will not get created. On a side note : Surprisingly some ACS systems don't even care to check for duplicate instance numbers.
  
• Instance numbers are stored and retrieved, but the relationship with actual table records is not maintained: That is, ACS might have one view of instance number to the row and CPE devices have different mapping. This will have problems in modification of values. ACS thinks that it is modifying parameters of specific row, but CPE modifies some other record.
• Instance numbers are stored and retrieved for records which were created and populated with values, but not saved for unpopulated records: This scenario occurs when AddObject is successful, but before the record is populated with values, the CPE is restarted or got restarted for some reason. If CPE does not restore these instance numbers, the configuration of that record will not be successful (when CPE comes back) as ACS sends the configuration with instance number returned by CPE before.
  

Problems associated with ParameterKey: ParameterKey is also expected to be stored in persistent memory across reboots. Some ACS systems use this as a configuration checkpoint. ACS systems know the CPE configuration by reading paramterKey value from the CPE. ACSes use this to figure out the 'configuration diff' between what it thinks CPE has versus the latest configuration it has for CPE. Only this difference is typically sent to CPE. Due to this, it is expected that parameterKey is stored in persistent memory along with the rest of configuration. Typical Problems observed are:

• ParameterKey and configuration is not stored in persistent memory: There are two issues due to this. ACS reconfigures the device every time it restarts. This could be a problem as the device is unavailable until ACS configures the box.
  
• Configuration is saved and retrieved, but not ParameterKey: Once the device restarts, ACS thinks that CPE does not have any configuration and reconfigures the device from scratch. Since configuration is actually retrieved, there would be many duplication errors. ACS might get confused and might stop pushing the configuration until admin is intervened. In some cases, I observed some intelligent ACS sets the device to factory defaults and configures by sending entire configuration without any manual intervention from ACS administrator.
  
• Configuration and parameterKey are saved at different times: This is dangerous. Essentially the mapping between configuration and parameterKey is broken. When device comes back, ACS view and device view of configuration is different.
  

Problems associated with Access Control and Notifications: Access control is one feature CPE vendors forget to provide. Typical problems observed are:

• No support provided for Access Control: This is one of the important features for managed service providers. If end user changes the important configuration and makes mistake, debugging may take significant time for Service providers. Due to this , service providers would like to allow change of only specific configuration by subscribers. Without having this support in device makes that intention difficult.
  
• Support provided for Access control and notification only for some variables, but not all.
• CPE sends notifications for unasked variables

Many problems, I believe, are result of configuration parameter mismatch between local configuration methods and TR-069 method. For example, Web interface provided on CPE might have been developed long ago before TR-069 based remote configuration method is implemented. Some CPE developers forget to integrate their web interface backend functionality with TR-069. In some cases, they do poor job of integration.

Cloud computing security is going to catch up.

Please see this article in information week.

Google and IBM are teaming up together to provide cloud services. Google is already providing email and storage services and they want to go beyond that.

One interesting thing that was mentioned in the article is
"With the exception of security requirements, "there's not that much difference between the enterprise cloud and the consumer cloud," Google CEO Eric Schmidt said earlier this month during an appearance in Los Angeles with IBM chief Sam Palmisano."

One more quote from the article:
"The cloud has higher value in business. That's the secret to our collaboration."

Another thing I observed in the article is their planned usage of Xen.

Combining all of them put together:

• Cloud computing requires security. Otherwise, Enterprises may not be able to offload their servers to cloud.
• Cloud computing makes use of Virtualization.

I was giving choices in my earlier blog on *Cloud computing and Security*. Though information week article is not giving enough information on how the security services are going to be offered, but they will start thinking soon.

I am beginning to think that both kinds of models which I suggested earlier would be used.

• Flexibility for Enterprises to put their preferred vendor security products as virtual appliances.
• Providing security using one mega security appliance.

My prediction is that mega security appliance is required to provide typical infrastructure security. Virtual appliance flexibility will be provided for specialized security.

Packet processing applications - Updating to Multicore processors

I described the need for session parallelization and some tips on programming here. There are many network packet processing applications which don't take advantage of SMP systems. I tried to describe the steps required to convert these applications to run on SMP systems with as less modifications as possible.

Target packet processing applications:

• Packet processing application that maintain sessions (flows) such as firewall, IPsec VPN, Intrusion Prevention Software, Traffic Management etc..
• Packet processing applications where significant amount of cycles are spent on each packet. Session Parallelization technique uses some additional checks and queuing (enqueue and dequeue) operations of the packets in the session. Session Parallelization technique only works, if the number of CPU cycles required for packet processing is much more than the CPU cycles it takes for few checks and queue operations. Otherwise, you are better off taking locks during packet processing. For example, IP forwarding packet processing application can be parallelized using session parallelization technique.

Let us examine the typical packet processing application requirements.

• Upon first packet of the session, they create a context - Session context. These session contexts are kept in some kind of run time data structure such as Hash buckets, linked lists, arrays, some container etc.. - I call it as Home base structure.
• Subsequent packets of the session gets the session context from the home base Data structures.
• Session contexts are deleted either by timer or by user actions or due to some special packets.
• Session contexts are typically 'C' structures with multiple members (states)
  
• Some members are set during session context creation time and never changed. I call them "SessionConstants".
• Some members are manipulated during packet processing and those values are needed by subsequent packets. And these are not required by any other function other than packet processing. I call them "Session Packet Variables".
• Some members are manipulated by both packet processing and also in non-packet processing contexts. I call those "Session Mutex Variables".
• Some variables are used as containers by other modules. That is responsibility of storage and retrieval is responsibility of those modules. I call them "Session Container Variables"
• Any packet processing application might contain multiple modules and each module has its own sessions (control blocks). To improve performance and other reasons, yet times, modules store other modules' control block references in their sessions. Also, current module session might be referred by other modules. when the session is actively used by other modules by having pointer to the session, it is necessary that session is not freed until all active references are dereferenced. At the same time, delete operation can't be postponed until it is deferenced by every other module. To satisfy both the requirements, each control block typically contains two variables - "Reference Count" and "Delete flag". Reference Count is incremented when external modules store the reference of the session. When the session is being deleted (either due to inactivity timer, due to special packets or due to user actions etc.. ) and if the reference is used, then modules set the "Delete flag" of the session. Any search on the session ignores the sessions having "Delete flag" set. Also the module is expected to inform the external modules that the session is being deleted. Upon this notification, external modules are expected to remove the reference and as part of it they decrement the reference count of the session. Session is freed only after all references are removed and Delete flag is set.

Process to follow to make applications SMP aware (Example: Applications running in Kernel space)

• Identify the Session or Control block of the target module.
• Define two additional variables in the control block - Reference Count and Delete flag.
• Identify the home base structure. Since the sessions can be created in multiple CPU contexts, define the lock for this structure. It is better to define Read/Write lock. Read lock can be taken when the session is searched in home base data structure. Write lock needs to be taken while adding or deleting the control block to/from the home base data structure. Ensure to increment the reference count while adding to home base structure. Remove it from the home base structure only if reference count is 1 and Delete flag is set to TRUE.
• Some times home base could be some other module's control block. In this case, it is responsibility of other module (Container module) responsibility to store, retrive and reset this control block reference atomically.
• Always ensure to initialize the control block completely before adding it to the home base data structure.
• Ensure that control block reference count is incremented within the home base lock for both add and search operations.
• Identify session constants. There is nothing that need to be done for SMP.
• Identify Session Packet Variables. There is no need to lock these variables if "Session Parallelization" technique is employed.
• Identify Session Mutex variables. Further identify the logic groupings of these variables. For each of these logical groupings:
• Define a lock. This lock can be global lock or session lock. Session locks are good for performance reasons, but it requires more memory.
• Ensure to define MACROS/inline/functions to manipulate and access variables of the group.
• Identify container variables. If this control block is home base structure for other module control blocks, then define MACRO/inline/functions to add/retrieve/reset. These macros would be used by other modules.
• Define this set of MACROS for each container variable.
• Define a session lock for each container variable. Above MACROS use this lock to protect the integrity of the container value.
• Since ADD macro is keeping the reference of foreign module control block, ensure to increment the reference count of foreign control block. To allow this, ADD macro should expect the fucntion pointer being passed to it by the foreign module. It is expected that the function ponited by function pointer is used to increment the reference count.
• Expect the "Increment Reference Count" function pointer passed to the RETRIEVE macro. Since RETRIEVE macro returns the pointer of foreign module, it is necessary to increment the reference count to ensure that pointer is not freed prematurely.
• Ensure that complete functionality of ADD/RETRIEVE/RESET MACROS work under the lock defined for each container variable.
• Some guidelines on Reference Count and Delete Flag variables.
• Define "IncRefCount", "DefRefCount" and "SetDelete" and "GetDelete" macros.
• Define a lock to manipulate and access "Reference Count" and "Delete flag" variables.
• Use this lock only in above macros/inline functions.
• Use above macros for manipulating and accessing these variables.
• Do first level of cleanup of control block when "SetDelete" macro is called. This cleanup involves informing external module of its intention of going down and also removing any foreign module references. Foreign references are ones this module stored as part of its packet/session processing. Removing foreign references involves decrementing the reference count of foreign reference and invaliding the reference in the control block.
• Do final cleanup such as freeing up memory for control block only when "Reference count" is 1 and "Delete flag" is set to TRUE. Before freeing up the memory, remove it from the home base structure. This condition typically happens as part of "Decrement Reference Count" macro.
• In non-SMP scenarios, the reference counts are incremented when external modules store the reference of the session in their modules. In SMP scnearios, while one processor processing the packets of the session, another processor can delete the session. Due to this, it is expected that reference count is incremented even while packet processing is happening. That is, whenever "Search" for session is done or when the session is being retried from container module, the reference count must be incremented. As indicated above, retrival of the session, including incrementing the reference count must be done either in data structure/container lock. Since incrementing the reference count happens within its own lock, you would see lock within lock in this scenario. That is, reference count lock is taken with data structure or container lock taken. This is Ok as data structure or container lock is never taken under reference count lock even in cases where session is added or removed from the data structure/conatiner.

DDOS Mitigation functionality - Tips for Admin

There is some discussion going on in focus-ids mailing list on DDOS attack mitigation. That discussion prompted me to write this article. ISIC, UDPSIC, TCPSIC, ICMPSIC are some tools used to measure the network security device effectiveness of detection and mitigation of DDOS attacks. As we all know one of the main intentions of DDOS attacks is to make the service, network or target unavailable. By looking at the packets, you can't see the difference between normal genuine packets and packets generated by DDOS attacks. This makes it difficult to stop these attacks based on signature based methods.

One of the properties of many of DDOS attacks is that they try to make the discovery of source of attack difficult to find. There are two types of DDOS attacks that are common.

• Spoofing of source IP address in the packets: DDOS attacks are generated by spoofing the source IP address of the packet. ISIC, UDPSIC, TCPSIC and ICMPSIC tools simulate these kinds of attacks. Any packet that is sent back to the source does not reach the attacker. Due to this, TCP based sessions don't get established. Note that non-TCP sessions don't have connection establishment phase.
  
• Botnets : The attacker instructs the agents which were installed on compromised hosts across the globe to bombard the target. Attacker keeps changing the hosts that attack the target. Thereby, in effect making the source discovery ineffective.
  

Now, in addition to Botnets, there is a third kind of DDOS attack. Recent DDOS attack on cnn.com by Chinese hackers is one example. Here too, sources are known, but there are many. Note that here the attack on cnn.com is not generated by botnets, but supposedly by out of patriotism. Some in China felt that CNN is biased in its reporting on Olympic torch and its linkage to Tibet religious freedom. As I understand, it was simple attack, where it connects to cnn.com website and accesses some URL for every 2 seconds. This attack executable was distributed and many home users in China, out of patriotism, executed it.

DDOS attack incident detection may be easier, but mitigation is difficult. If the intention of the attack is to consume the bandwidth of target site, there is nothing much the target network administrator can do. Target company/organization needs to depend on its ISP to block the flood of packets. Gathering as much information as possible and providing that information to ISP is one of the things the administrators can do.

The current trend of DDOS attacks go beyond consuming the link bandwidth. With less number of hosts participating in the DDOS attack, these attacks consume the CPU, memory bandwidth of target networks/servers. I feel the network security appliances providing DDOS attack mitigation functionality can help in this scenario. It can not only provide detection, but can stop bombardment of servers.

There are multiple products *DDOS mitigators* in the market claiming to solve some of above problems. Many IPS boxes also support this feature.

If you are hosting some servers, you can be a victim. As an administrator, I look for following features from these appliances.

DDOS attack consumes 1Mbps link by making 512 connections/sec (approximately) . Any DDOS mitigator, ideally should be able to process 512 connections in every second for 1Mbps link. If the connection is maintained for 20 seconds (which is typical), then the connection capacity needs to be 10K. For 100Mbps link, DDOS attack mitigation appliance needs to support 51200 connections/sec and should have 1M session capacity. With this capacity and connection rate, it can do better job of protecting internal networks/servers/other stateful security devices without itself getting bogged down.

DDOS mitigators are expected to limit the amount of traffic that goes to the internal servers/machines/networks etc.. Each resource in the network would have some limitations on how much traffic, connections, connection/sec it can take. Adminis, once they make a list of resources and their limitations, should be able to configure DDOS mitigators. DDOS mitigators must ensure that the resources are not flooded and it should shape the traffic accordingly. DDOS mitigators need to provide features like:

• Ability to configure
• connections/sec
  
• Packets/sec
• Bytes/sec
• On per resource basis - Server/machine basis, Network basis
• From a given source with respect to IP address range, Subnet.
• Ability configure to filter traffic on combination of 5 tuples.

As you have observed, admins not only would like to shape the traffic to internal resources with respect to connections/sec, maximum number of connections and throughput, but also would like to have these limits from particular source(s). Yet times, I also observe that there is a requirement to limit the amount of traffic within each 5 tuple connection, between any IP address combination. Mitigators need to provide this flexibility without expecting admins to create many rules. Many times, it is not possible to create rules with all combinations of IP addresses. DDOS mitigators need to provide flexibility of creating rules with ranges, subnets along with provision to configure granularity to apply the specified traffic rates. For example, admin should be able to configure ANY to 'Internal HTTP Server IP addresses' with 10 connections/sec for every combination of source IP and destination IP. If there are 100 different sources are trying to access internal HTTP Servers, DDOS functionality should be able to rate limit the number of connections to 10/sec for each of sources independently.

As with any security device, it must also support multiple zones and provide flexibility with respect to zones. In case of hosting environments, provider may be servicing multiple customers. So, virtual instance, with each instance belonging to a customer is needed. In case of Enterprise environments, normally only one virtual instance would be used.

Flexibility is expected to be provided to disable limiting of traffic for some source networks. These networks could be networks belonging to remote offices. This feature is called white listing.

Ofcourse, it is expected that DDOS mitigators provides facilities to stop half open connections by providing TCP syn flood protection, UDP based session exhaust protection facilities, facilities to configure service inactivity timeouts for interactive protocols etc..