Key Management System - Required features for network elements' security

Every network element in Enterprise network requires access to security keys at some time or other.  Mobiles, laptops typically have security keys for securing emails,  for storing the confidential data and for backingup the data in remote locations.  Servers in data centers have private keys and certificates for providing authentication credentials to external users.  Network infrastructure devices also require symmetric keys for storing the data securely in file system, private/public key pairs and certificates to provide authentication credentials to remote VPN users etc..   Key Management Server provides a central mechanism to store the keys and retrieve the keys on demand basis.  

PCI (Payment Card Industry) standard specifically requires that keys are not stored in the network elements' persistent memory.   It is expected that the keys are downloaded every time that element is powered up.  It is also expected that the keys downloaded are kept in temporary memory which would get erased when power is off.  In case of battery powered devices, they are expected to be stored in Hardware Security Modules (HSM).  This mechanism would protect the data when the media or device is stolen.

Centralization of  all key management operations  gives control for Enterprises to manage the life cycle of keys. This kind of control is required now more than before due to cloud computing and smart phones. As we all know, Cloud computing allows Enterprises to rent VMs on demand basis.  Virtual appliances are typically installed on the VMs.  Since these rented VMs are hosted in Cloud computing providers' data centers,  Enterprises would like to ensure that the not only data, but also keys are secure.  Having Key Management Server within Enterprise trusted boundary would ensure that administrator to do Key management operations such as Revoke (expire), key rotation.   When VM is no longer required, all keys associated with the VM can be either revoked or archived.  This kind of control also helps to revoke the keys in case of VM break-in. 

What are the devices and applications that require keys and Key Life cycle management:


End network elements such as laptops, desktops and smartphones :  

• Secure Email :  PGP  is normally used to secure the emails on the wire.  PGP requires private key/public key or certificates to secure the emails.
• Crypto file systems :  These storage systems (within laptop or on dedicated remote file systems) are used to secure the confidential information.
• Data backup :  Backup data should be secured and only can be decrypted by user.
• IPsec VPN Client

Servers and Network infrastructure devices:

• HTTP Servers/SIP Servers/other SSL based Servers:  
• SSL based proxy applications such as Network Anti Virus,  SSL VPN,  Management Servers such as HTTPS,  SSH etc..
• Crypto file systems to store the confidential data.
• IPsec VPN Servers

Centralized Key Management consists of two components - Key Management Server and Clients.  Typically there would be one key management server and as many clients as number of network elements requiring centralized keys. 

Generic requirements of Key Management Server:

• It is expected to provide storage of different types of keys.  Some applications generate keys on their own and store the keys in Key Management Server.  Some applications expect Key Management Server to create and store the keys.
• Private key,  public key or certificate and certificate chain.  
• Symmetric key.
• Get Operation : Key Management Server is expected to serve the keys to the clients requesting.  
• Get Attributes:  This operation is expected to be provided by Key Management Server to provide different attributes associated with the key.  Keys are not served as part of this operation.
• Back up of keys in case of any trouble in Key Management Server :  There are two operations are expected - Export and Import.  Export is to backup the keys from the Key management server and import is to get the keys from backup. 
• Key Expiry : Expiring the keys -  Typically keys are rotated every 1 or 2 years.  Some times, expiry of keys need to be done in case of key compromise.
• Key rotation :  As part of expiring the keys,  it is required to create new key to replace the old key.  To ensure the continuity of applications, both keys may need to be active for some time (overlap time).  During overlap time, old key is typically used for 'process' operation (decryption). New keys is used for both protect and process operations. 
• Archive of keys:  Once the key is expired,  it can be archived.  Archiving is different from backup.  Backup of keys is done on active keys and archiving is done on the inactive keys.  Resume operations is expected to be provided in case keys need to re-activated. 
• Deletion of keys:  Once the keys are determined that they are no longer required,  deletion operation can be done. Typically deletion happens after few years after it is archived. This is to ensure that there is no useful data that was encrypted using the keys that are targeted for delete operation.
• Key Management Client and Server communication must be secure (SSL is one option).  KMS is expected to have scalability to support large enterprises. 
• Network Element (Device) registration :  A device might have multiple applications requiring Key Management facilities.  If the device is stolen, all the keys of all applications in the device need to be revoked.  Revocation might involve actively communicating with the device and also to communicate with other devices using same keys or some parts of the keys.  To facilitate this,  key management server is expected to provide revoke operation on all keys correspond to the device.  KMS is expected to provide facilities to register the devices, revoke all keys on a device and deregister the devices.   There could be many devices that exist in the Enterprise.  It may be expensive for IT to register the devices.  There should be facilities for Key Management client on the device to auto register the device if it does not exist in the KMS.  KMS is also expected to provide control for users to revoke the keys on the device.  That is, if laptop or smart phone is stolen, user itself can go to KMS and revoke the keys rather than depending on the IT to do which may take longer and it is possible for thief to get hold of data in between.
• Binding among the keys:  
•  In public key cryptography,  pair of keys are required together - Private and public key pair. Yet times, certificate chain is also required along with this pair of keys.  Any key management operation is typically done on these keys together.  That is, when the key is revoked, its counterpart is also expected to be revoked. This binding is 'peer binding'. 
• There is another kind of binding -  Key derivation binding.  Here the binding parent-child binding. Some operations on the parent are applied on the derived keys.  Actions on derived keys may not have any affect on the parent key. 
• Some times keys are used for wrapping and unwrapping the other keys.  This is normally used in export and import operations.  When keys are exported,  keys are encrypted using wrap key.  As part of import operations, keys are unwrapped using wrap key. For lack of better term, let me call this as 'wrap binding'.
• Key deployment in clients:
• Key Management Systems are expected to provide the keys requested by the clients as long as ACL rules allow it to do.  
• Yet times,  multiple objects may be retried by the clients together.  Key Management System should provide mechanism to bind the objects and get the all the objects corresponding to that binding. Private/Public key and certificates retrieval is one example.
• When new keys are created or registered by one client,  it is needed that these keys are deployed on other systems (clients).  There are two common cases:
• VPN Server creating or registering the keys:  Public key and certificate objects are expected to be installed on all known VPN clients.  How does KMS know the devices to install the keys to?  KMS can take 'application registration'  in this case. This is in addition to Device registration. VPN Server as part of installing the keys in KMS can indicate the 'application Identification' of the clients and also indicate the keys that need to be installed. KMS can use this information to find out devices from the device/application registrations and deploy the keys in those devices and application.
• Cluster of Servers :  In Enterprise, multiple servers are used to serve the content. These servers share the load among them.  It is necessary that keys used are same across all servers. Even one server registers the keys, all keys need to be deployed in all other servers in the cluster.  KMS is expected to allow KM clients to indicate the information so that it can deploy keys to all other servers.  Again device and application registration facility allows this.
• Access Control List: Access Control list play very important role in providing control for IT departments to do key management operations. It provides control for trusted users to do key management operations - Create/Register,  Backup, Revoke,  Key Rotation,  Archive, Delete, Get, Get Attributes - by associating user groups to permissions (key management operations).  Each ACL rule is really a combination of 'User group' and 'Permissions' on a given key object. It is normally accepted in the industry to create the ACL rules as part of creation/registration of keys.   If the creator group is not present in the KMS during the key creation time, creator group is also created with the creator user as part of the user group. There are two types of ACLs - Key object based,  non-Key based.  Key Object based ACL is part of each Key object and is normally created by the creator.  Non-Key based ACL are non-key specific and normally created by the administrator.  
• Non-Key based ACL:  These ACL rules control the access when the keys are not present.  These ACL rules are normally created by the administrators. Each ACL rule contains the 'user group' and 'different types of permission'.    Different types of permissions can be:  Create/Register keys,  Create user group as part of key object creation,  Allow auto device registration by the creator of key object,  'Allow application registration by the creator'.  'allow creation of ACL rules on object by the creator',  'Allow archive', 'Allow resume', 'Allow import', 'Allow export',  ' Allow derive' by creator etc..   If administrator does not give 'Allow user group',  'Allow auto device registration' or 'allow ACL rules on objects', then administrator is expected to do these operations.  In addition to creating non-key based ACL rules,  KMS should provide facilities for administrators to add 'admin' users automatically to the user groups created by the creators to ensure that Enterprise always have control of keys. This is useful to process the data  when the IT gets hold of the laptops and smart phones after employees leave the company.  Yet times, administrator themselves might delegate some operations to some other IT staff.  Non-key based ACL should allow multiple IT personnel to do the key operations on existing key objects. These ACL rules may be delegated based on the 'applications'.  These ACL rule, hence, should contain   'user group',  'application identifier' and 'permissions'.  'Permissions are same as 'key based ACL rules'.  'Application identifier' can be 'Any'.
• Key based ACL:   These rules are normally created by the creators. Each rule contains 'Key object',  'user group' and permissions.  These ACL rules are sent to the KMS as part of create/register operation.  As discussed in addition to the ACL rules,  KM client also can send deployment information such as 'send the keys'.   KMS is expected to send the keys to the clients based on the application identification.  ACL rules indicate what rule information can be retrieved by the clients.  In case of VPN client, ACL rule are created by the creator such a way that only public key and certificates can be retrieved.  In case of cluster of servers application, ACL rule is created such a way that private key, public key and certificates can be retrieved by all servers in the cluster. 'Permissions in the ACL rule' can be - Get, Get Attributes,  Revoke, Derive, Expire, Archive, Backup, Delete etc.. 

Performance considerations in Proxy based nework applications

Many details on performance considerations on proxy based networking applications are given here. 


There are some more performance considerations in developing proxy based applications. Here they are:

• Use Hugelbfs system to for running code and for application context memory (for connections):  Please see this for getting understanding of this technique.
• Use User space RCU wherever possible:  See the RCU related information here.
• Use Futexes as part of RCU implementation for add/delete operations.  See about Futexes here. 
• Use posix spinlock kind of Mutexes only for small portion of the code.
• Use UIO based Interrupt indication to the User space processes while dealing with memory mapped hardware accelerators.

Responses on the post related to WAN optimization and Infineta systems

I have got many responses on this post.  Some were asking more clarifications on what I meant by deduplication efficiency across restarts.  Some questioned that how I can make a statement that there is no persistence data across restarts in Infineta solution.  By the way, I did not make a statement. To clarify further, I don't know whether the solution has capability of keeping data intact across power cycles.  I just want to say for record that, it is only my reading based on the statement that the performance of the solution  will be in multiples of gigabits/sec, up to 10Gbps.  I guess we will come to know eventually when the product is out.

One response suggested that it is very well be that the solution might have multiple hard drives, with each drive hanging on different hardware bus,  to achieve multigiga bit  performance. Though it seems like a possibility and I am not sure whether it is practical.

One possibility is that multiple blades are combined into one solution with each blade working almost independently as far as Dedup is concerned.  This type of solution is suitable for deployments having multiple servers that need to be synchronized (for backup, replication and others) where content of  server or set of servers being optimized by different blades.  If there are few servers than the blades, then this type of solution capacity can't be utilized completely in those specific deployments.

One response was asking on whether I know of any performance benchmarking criteria to evaluate WAN optimization solutions. I am not aware of any standards or defacto standards. If I come across any, I will certainly post them in this blog.

One more WAN Optimization company - Infineta systems

It is good to know that WAN optimization technology is attracting VC money. I believe that this market will continue to grow for few more year before it saturates. Though some research reports place this market to be at $5 billion by end of 2014,  but I feel that this is much more than this. Any organization having multiple branch offices with consolidated central servers benefit from the WAN optimization.

Okay.  Coming to Infineta.  What is different about this technology compared to existing WAN optimization?  I tried going through the Forrester report.  Finally it all coming down to 'performance'.   According to this report existing WAN optimization products peak at 1Gbps and Infineta systems performance seems to be in terms of multiples of Gbps upto 10Gbps.  According to this report, this kind of performance is required to connect data centers of a given organization.  Reasons given for this kind of performance are:

• Replication, Mirroring,  Backup of data and VM images among Data centers for reasons such as Business continuity/Disaster-recovery. 
• Amount of data exceeding perabytes.
• Reducing latency of above operations.

According to job descriptions, it is clear that they are using multi-core processors and FPGAs. I did not find any mention of hard disk capacity.  I have a feeling that persistent storage is not used.  It would be difficult to achieve multiple of Gbps throughput with disk access.  If that it the case, it is interesting to know how the efficiency of de-dup is compared with other established WAN optimization vendors. 

• Amount of DDR memory.  This directly would be proportional to dedup efficiency.  Larger the amount of memory,  higher the dedup efficiency  would be. Without hard drive capability, storage would be limited and it may have difficult time to achieve the de-dup efficiency compared to others in the market, in my view.
• Lost of cached blocks across power restarts.  If there is no persistent memory, data that was stored in the DDR memory would be lost across power cycles or when the system it taken out for maintenance. This requires rebuilding of the data cache again.  This will reduce de-dup efficiency right after power recycle.