Dual Stack Support in LTE eNodeB - Technical bit

This technical bit summarizes the type of functionality expected out of Dual stack in user plane of eNodeB.

Introduction

eNodeB connects to UE on Air interface side  and connect to multiple types of devices in the core network over backhaul network.  It communicates with MME,  S1 Gateways via Security Gateways.

GTP-U layer is the relay module which transfers the packets to/from UE to wireless core network.  GTP-U tunnel is normally terminated on S1 Gateway for normal traffic.  GTP-U tunnels are also terminated with other eNodeB in handover cases.

GTP-U packets can be optionally secured using IPsec.  Though transport mode is good enough, but tunnel model is commonly used.  IPSec tunnels are terminated at the Security Gateway in the core network.  Security Gateway are typically placed at the edge of core network and its placement is between eNodeB and S1 and other gateways in core network.

eNodeB Dual Stack Requirements:


eNodeB must ensure that it works with its peers.

UEs:

• IPv4 only UEs :  These UEs generate and consume Ipv4 packets only.
• IPv6 only UEs:  These UEs generate and consume Ipv6 packets only.
• Dual Stack UEs:  These UEs can generate and consume both IPv4 and IPv6 packets at the same time.
• IPv4 Radio Bearers
• IPv6 Radio Bearers
• IPv4 and IPv6 Radio Bearers where both Ipv4 and IPv6 packets can be seen on the same RB.

S1 Gateways:

• IPv4 Only S1 Gateways  :  GTP-U tunnels are IPv4 tunnels 
• IPv6 only S1  Gateways -  GTP-U tunnels are IPv6 tunnels.
• Dual Stack S1 Gateways  -  GTP-U tunnels to these gateways could be either IPv4 or IPv6.
• In a given deployment, there is a possibility of having any of above types of S1 Gateways.

IPsec VPN Gateways:

eNB typically contains IPsec part of the eNB itself.  On the core network though,  Ipsec gateway is normally not combined with S1 Gateway.  It is a separate device/blade that sits in the core network.  Ipsec tunnels from eNB are terminated at this gateway whereas GTP tunnels are terminated at the S1 Gateway.

With this above background,  following functionality is expected typically in eNB:

GTP-U Layer:

• Since there are different types of S1 gateways (IPv6 only,   IPv4 only and Dual stack),  MME can decide to put different UEs on various types of S1 gateways.  Hence GTP-U layer must be able to support IPv4 tunnels and IPv6 tunnels. 
• GTP-U must be able to transport both Ipv4 and Ipv6 packets between UE and Core network on IPv4 based GTP tunnels and IPv6 based GTP tunnels.
• GTP-U layer must be able to set DSCP values on outer IP header (IPv4 or IPv6) and should be able to copy DSCP values from inner IP packet (Could be IPv4 or IPv6).  Hence GTP layer should be able to copy DSCP value from  (For Uplink packets) 
• IPv4 to IPv6
• IPv6 to IPv6
• IPv4 to IPv4
• IPv6 to IPv4

IPSec Layer:

As noted above,  one security gateway might be front ending multiple S1 Gateways with respect to base stations.  Since there could be multiple GTP-U tunnels on one IPsec tunnel,  IPsec layer must support following:

• Must be able to work with Security Gateway on core network whether Security Gateway in core network supports Ipv4 or IPv6 tunnel.
• Must be able to transport both IPv4 and IPv6 packets (GTP-U tunnel packets) on one tunnel.
• Must be able to do DSCP copy from GTP-U header to Outer IP header (could be IPv4 or IPv6). 

QoS Layer:

Since both Ipv4 and IPv6 packets traverse on the same Ethernet Port or VLAN port, it is necessary that the shaping and scheduling does not require two different types of configuration. It should be possible to create

• QoS ACL  to have both Ipv4 and IPv6 rules.
• A rule with both Ipv4 and Ipv6 address tuples.
• Multiple rules pointing to the same Queues.

LAG, Load Rebalancing & QoS Shaping

LAG feature exposes only one L2 interface to the IP stack for each LAG instance.  It hides all the links in the LAG instance underneath it.  It sounds good in the sense that IP stack & other applications are completely transparent with respect to number of links that are being added and removed.

Though many applications and IP stack don't care about the LAG, links and its properities,  one application QoS would need to worry about the link properities - specifically its bandwidth (shaping bandwidth).  In ideal world,  even QoS does not need to worry about links and its properties. As we all know, to ensure that there is mis-ordering of the packets in a given conversation,  distributor function of the LAG module distributes the conversations across the links, not the packets. If there are large number of conversations compared to the links, there is always possibility of equal distribution of the traffic across the links. But when there are small number of conversations, which by the way not so uncommon, then there is a possibility of unequal distribution with respect to the traffic.  That is, there could be more traffic in some conversations compared to others. If high traffic conversations go to few links, then there is unequal distribution.  Let me cover QoS and changes required in QoS to work with LAG.

Load Rebalancing:

LAG distributor normally implements the concept of 'Load Rebalancing'.  Load rebalncing happens in three cases.

•  When LAG observes that there is unequal distribution.
•  When new link is added to the LAG instance.
•  When existing link is removed, disabled or broken.

Though new link and removal of existing link to/from the LAG instance is not the focus of this article, let me just give a gist of  issues that need to be taken care.  Packet mis-order issue must be taken care well.  When the new link is added,  if hash distribution is changed immediately, some of the existing conversations might be balanced to other links. If it is done arbitrarily, then there is a possibility of packets being received by collector in out-of-order for brief amount of time.  To make sure that new link is used effectively,  there are two methods can be used. Both can be used togehter though.

•  New conversations would use new hash distribution.
•  Current conversations can be put onto other links only if the conversation is idle for X milliseconds - Time at which we know that packets would have been collected by the collector.

When link is no longer active,  then packet mis-ordering is no longer a big issue.  The traffic has to flow and new distribution can take effective immediately and distirbute the conversations that belong to the old port to existing ports immediately.

Now on to redistribution due to unequal utilitization of links:

Redistribution can be done in two ways - Changing the hash algorithm or fields to be used in hash algorithm.  Second is to some how increase the number of conversations.  Second method of increasing the conversations would work only in cases where tunnels (such as Ipsec) are conversations.  By increasing the number of tunnels,  there is a good possibility of increasing the distribution. Actual 5-tuple flows are sent on multiple tunnels. See this link here on how LAG & IPsec work together.

Changing the hash algorithm or adding/removing fields to the hash algorithm would have mis-order issues.  In some deployments mis-order once in a while is okay.  In those cases, this methoed can be used. To use this method, rebalancing should not happen very frequently.  Typically following mehtod is used - If a link utilization is more than X% (Typically 5 to 10% - configurable parameter) away from the average usage of the trunk, then it is candidate for redistribution.  Stop doing redistribution for configurable amount of seconds to ensure that there are no frequent redistributions.

QoS:

Typically QoS shaping & scheduling function runs on top of L2 interfaces.  Trunk link would be given the shaping bandwidth. Shaping is typically implemented using token bucket algorithm.  Whenever there are tokens available,  scheduling function is invoked.  Scheduling function selects the next packet and sends the packet out.

LAG instance which is actiing as L2 interface has the shaping bandwidth which is sum of all the links. If the scheudling decision is taken purely based on the LAG trunk bandwidth, there is a possibility that scheduled packet would get dropped if the packet goes on link which is already completely utilized. This happens when there is uneven traffic in the convesations.  Rebalancing helps, but it takes some time rebalance the traffic. Hence QoS shaping and scheduling function should not only consider the LAG instance bandwidth, but also the individual link bandwidth while making scheduling decision. By considering both,  at least the paket from the high traffic conversation is not scheduled and resides still in the queue, there by avoiding packet drop.
At the same time, it is not good to under utilize other links. Scheduling, in this case, can move to other traffic that fall in other under-utilized links.

LAG is important feature, but it has its own challenges.  IPsec and Qos implementations need to work with LAG properly to utilize LAG effectively.

Comments?

eNodeB and IPsec

eNodeB secures the traffic over the IPsec tunnels to the Serving Gateway (SGW) over backhaul network.  Also, eNB creates many tunnels to peer eNBs for X2 and handover traffic.  Though all features related to Ipsec are valid in eNB scenarios too,  some features are worth mentioning in eNB context.

LAG and IPsec:

Please this link here to understand the issues and solutions related to LAG and Ipsec in general. This scenario is very much valid for eNB to SGW communication. Note that traffic from all GTP tunnels in non-handoff scenario go between eNB and SGW on one or few (when DSCP based tunnels) Ipsec tunnels.  When LAG is used between eNB and LAG,  similar issue of not utilizing more than one link would arise.  Both the solutions suggested in earlier article are valid in this scenario too.  In cases where it is difficult to get multiple public IP addresses to the LAG link, then scenario 2 - forceful NAT is only option I can think of.


Capabilities expected in eNB and SGW:

Using LAG effectively requires many tunnels.  It is good to have 1 + (number of links - 1 ) * 32 Ipsec tunnels for good distribution across links.  User traffic, in this case GTP traffic should be balanced across these IPsec tunnels.

Typically there are two GTP traffic tunnels for each cell phone user - One is typically created for Data traffic and another for voice traffic.  Without LAG, normally two Ipsec tunnels are created - One for data traffic coming/going  from/to  all the cell users and another for voice traffic for all voice traffic coming/going to cell users.  GTP traffic is distributed across these two Ipsec tunnels based on DSCP value.

Now, we have lot more Ipsec tunnels.  There should be additional logic in eNB and SGW which distributes the GTP traffic across these multiple Ipsec tunnels.  This logic should distribute the traffic from a given conversation to one Ipsec tunnel.  Each GTP tunnel traffic can be viewed as one conversation.   That is, GTP tunnels are distributed across the Ipsec tunnels.   One way is to look at the TEID (Tunnel Endpoint ID) and use hash on TEID to distribute the traffic across Ipsec tunnels.

Ipsec implementation on eNB and SGW should have capability to create multiple tunnels - number of tunnels to be created should be configurable.  eNB and SGW implementations also should have capability to bring up the tunnels on demand basis too. That is,  they should ensure that these number of tunnels are UP and running as long as there is traffic.  Note that all ipsec tunnel negotiation would have same selectors and ipsec implementations should not be having intelligence to remove old tunnels with same selectors.

If persistent feature is selected on SPD rules, then the implementations should ensure that all ipsec tunnels are UP and running all the time.

As described in the earlier article, it is necessary that Ipsec implementation have capability of doing 'Red side fragmentation' so that the LAG always sees UDP header in every packet which is required for its distribution.

DSCP based Ipsec tunnels:

LTE uses packet based network for  voice, streaming,  interactive and non-interactive data.  Hence it is necessary that Ipsec tunnel honor this priority to ensure that voice and other real-time traffic is given priority.  If both data and voice is sent on the same tunnel, there is a possibility of traffic getting dropped due to sequence number checks as part of anti-replay checks in the receiver.  Even though packets are marked with increasing sequence number in both data and voice traffic and encapsulated in the Ipsec tunnel,  due to local QoS and QoS in intermediate devices, voice traffic may be sent before the data traffic - that is traffic is reordered.  As you know, receiver window right edge moves with newer sequence number.  Due to this, some data packets which have lower sequence number get dropped if they are less than the lower edge.  To avoid unnecessary drops, there are two methods used - Increase the Anti-replay window size  or use different SAs (tunnels) for different kinds of traffic. Second method is normally used.

Due to this feature and above LAG feature, number of tunnels that need to be created in eNB and SGW can go up significantly.  Hence both eNB and SGW should have enough memory and computation power to handle multiple tunnels.

Persistent tunnels

To reduce the latency of initial traffic, it is necessary to have this feature. Tunnels are UP and running all the time even when there is no traffic.  This feature is good if the links are always-on and if there is no cost based on the traffic amount.


DSCP and ECN Copy settings:


Ipsec implementations expected to copy DSCP and ECN bits from inner header to outer header.  Inner header DSCP value is set by applications and this should be continued even when the traffic is tunneled.  This will ensure that the nodes between the eNB and SGW will also give QoS treatment.  Hence it is necessary to copy the DSCP bits from inner header to outer header.

ECN bits indicate the congestion to the peer so that peer entity can inform the source entity to apply the congestion.  TCP protocol has a way to inform the source entity when the receiver gets the IP packets with CE (congestion experienced) bit on in ECN bits of IP header.  Intermediate nodes, including eNB and SGW should honor this by copying from inner header to outer header while encapsulating and copy from outer header to inner header while decapsulation.

Peer IP address adoption 

eNodeB gets the IP address from backhaul provider dynamically. It is possible that IP address might be changed by the provider while traffic is going on.  Ipsec tunnels are expected to be UP and running even if the IP address changes on the gateways.   This internet draft discusses the mechanism to adopt the peer gateway address change.  This feature is expected to be present to ensure that voice traffic does not observe too much of jitter and latency.  Note that tunnel establishment takes hundreds of milliseconds as it involves IKE negotiation of the keys.  This can introduce jitter and latency when the voice traffic is going on at that time. Implementation must implement this draft to eliminate jitter and latency issues when the IP address changes on the remote gateway.

IP Fragmentation and Reassembly

Normally many vendors of eNB give performance with respect to UDP traffic without involving IP fragmentation and reassembly. Even though this gives one data point, this may be misleading to customers. Most of the traffic in Internet today is TCP and same is true in LTE world too.  TCP MSS is chosen such as way that TCP data packet with IP and TCP header would be MTU size.  When the traffic undergoes IPsec encapsulation, it is almost certain that packets would need to be fragmented as it exceeds the MTU of the link.  Though DF bit facility is available for end points to know the Path MTU,  this feature is not used in Ipv4 end points today.  Since packets are fragmented,  reassembly is required on other side.

This feature is implemented in Ipsec implementations, but I am afraid that many implementations, though have very good IPsec performance on non-fragmented packets, they are not optimized when fragmentation and reassembly is required.  Customers need to watch out for this as significant amount of traffic would be fragmented and reassembled.

IPv6 Support

Ipv6 is fast becoming choice of service providers and LTE core network.  Hence Ipv6 support is expected in ipsec implementations.  eNB and SGW must support both Ipv4 and IPv6 tunnels.  Also, they should be able to send IPv4 and Ipv6 traffic on IPv4/Ipv6 tunnels.

TFC

Traffic Flow Confidentiality feature is normally given less importance.  I was told that LTE networks require this feature be implemented in Ipsec tunnels so that the anybody who gets hold of backhaul network traffic will not be able to guess the type of traffic that is going on in the tunnels based on traffic characteristics such as - frequency of traffic,  size of packets,  distribution of packets  etc.. 


AES-GCM

AES-GCM combines both encryption and integrity in one algorithm. Hence it is called combined algorithm.  This algorithms is supposed to be 2x faster than AES-CBC algorithm. Also it is supposed to have half of latency of AES-CBC.  Hence it is good for both performance and also for latency which is required for voice traffic.  Hence it is becoming popular algorithm in eNB and SGW.

Validation Engineers and customers I believe should look for above features in eNB and SGW.

Comments?

IPv6 and eNodeB in LTE world - Technical bit

Does IPv6 migration of UEs and PDN Gateway require any support from eNodeBs?  eNodeB, MME, HLR, AAA and SGW need not be upgraded immediately along with PDN gateway to support Ipv6 external connectivity to the UEs.

eNodeBs communicate with the other 3GPP network elements for signaling and transporting the UE traffic over GTP tunnel.  They relay traffic from PDCP layer to the GTP tunnels and vice versa.  As such they don't look deep into the data as such. It could be IPv4 or Ipv6 packet.  GTP layer in eNodeB is expected to pick up the data, snap the GTP-U/UDP/IP header and send it over to the SGW by securing with IPsec.  GTP-U can use IPv4 connectivity to talk to SGW and eNBs etc.. Similarly signaling communication can happen on Ipv4 network.

PDCP RoHC (Robust Header Compression) is the only mandatory component in the eNodeB which tried to interpret the packets to/from UEs.  I believe the first generation of eNBs would expect PDCP RoHC to support both IPv4 and IPv6.  Some eNBs have some sophisticated functionality such as 'Application Detection'. Any component that interprets the user traffic would need to have IPv6 intelligence.  I believe eventually all the components such as GTP-U, IPsec,  QoS  require IPv6 support as operators move to IPv6 only networks.

By the way, I saw one IETF draft on IPv6 in 3GPP EPS.  Please find it here.  

Avoiding double IP reassembly in eNodeB and IPsec Gateway in LTE - Red Side Fragmentation

I have given one use case long time back for Red side fragmentation feature. Please find it here.  There is one more use case where one can avoid double IP reassembly.

In LTE world,  you have eNodeB and SGW (Serving Gateways) communicate using IPsec  over backhaul network.  Ipsec functionality is normally part of eNodeB.  But on the SGW side, IPsec is normally deployed in a separate network element (IPsec Gateway).  Due to this, there could be double IP reassembly in eNodeB. Avoiding reassembly can improve eNodeB performance. This can be achieved by configuring 'Red side fragmentation' on Ipsec GW in core network near SGW. 

Let us step back and see the packet processing.  Any packet between eNodeB and SGW are first tunneled using GTP tunnel.  Then they get tunneled using IPsec to secure the traffic.  Let us all assume that MTU of all the links is 1500. It is not a bad assumption at all as interfaces of these network elements in many deployments are Ethernet interfaces.  Let us also assume that 1500 byte packet is being sent out from SGW to eNodeb (downlink packet).  GTP in the SGW adds GTP/UDP/IP header to the packet.  Since MTU of transmitting interface is 1500, this packet would be broken down into two - one packet with 1500 bytes and another packet with rest of data.  Now this packet goes to the IPsec gateway .  IPsec adds ESP/IP header to the first 1500 byte fragment and second fragment.  First fragment after Ipsec is done exceeds the MTU and it gets fragmented.  So, there are now total three packets for the original 1500 byte packet.  eNodeB upon receiving the packets, first needs to reassemble the fragments for Ipsec inbound processing consumption.  For GTP-U consumption, it needs to reassemble the fragments which went through the Ipsec inbound processing.

How do we avoid the double reassembly?  I can think of three options.

• Combine GTP and Ipsec in the same network elements. Packets will get fragmented only after GTP and Ipsec outbound processing is done.  Hence only two fragments would be generated and peer only needs to do only one reassembly.  Note that this option may not be possible on SGW side due to scalability reasons.  GTP and IPsec are always  together in case of eNodeB though.
• Configure IPsec gateway to do reassembly before it does IPsec outbound processing:  Here GTP-U might have broken the packet into two. Ipsec gateway reassembles them and passes the reassembled packet to the IPsec outbound processing.  Ipsec gateway after outbound processing would fragment the packet if necessary.  Peer (eNodeB) will only see two fragments and only one reassembly is required. But note that this would have overheads in Ipsec gateway. This may be okay as ipsec gateway is normally deployed as separate element and the CPU might be entirely dedicated for this.
• Better option is in my view is to enable 'Red side fragmentation' in the IPsec gateway.  1500 byte GTP-U packet it receives get fragmented before IPsec processing is done. Three packets in total would go through the IPsec processing and would be sent out. There is no further fragmentation.  On eNodeB,  three fragments would get reassembled after Ipsec inbound processing is done and before GTP-U detunnel processing.  Only one reassembly is required.  It has its own advantage, but I believe this is better option. Disadvantages being - It requires 3 fragment reassembly on eNodeB.  On IPsec gateway, more packets would go through Ipsec engine. Since many Ipsec gateway use hardware accelerators, I am thinking that this additional processing will not affect the overall throughput.

Comments?

LTE Network Sharing (MOCN) - eNodeB

Typically eNodeB and Core network components are owned and operated by same operator. It is quite common to share the physical infrastructure among multiple operators, specifically the cell sites - physical location, building etc... But every operator used to have their own base stations and transport card etc..  As I understand that is called passive sharing.  This sharing is now extended to active component sharing such as eNodeB. That is called network sharing or active sharing.

Networking sharing allows one frequency spectrum ,  eNodeB shared by multiple operators. Dedicated cells for each operator but still sharing rest of E-UTRAN infrastructure is also possible.

There are multiple business reasons for network sharing.Main reason being cost savings.

• Cost savings by sharing infrastructure - eNodeB and Frequency spectrum.
• LTE deployments will require major investments as new eNBs, new antennas need to be installed by operators.  To reduce the number of eNB one needs to own,  in some cases such as rural areas there by providing connectivity while reducing the costs.

3GPP body as part of LTE effort visualized these scenarios and created specifications to handle network sharing scenario. 3GPP specifications  22.951 and 23.251 describe the network sharing requirements and architecture & functional description of network sharing. Main feature that allows network sharing in LTE is that eNodeB broadcasts multiple PLMN IDs (Public Land Mobile Network ID which is combination of Mobile Country Code and Mobile Network Code - Each operator has unique PLMN ID) to the UE using SIB (system Information Block).  UE is expected to select the PLMN ID based on its selection process.  Using the selected PLMN ID,  UE is expected to make RRC connection with eNodeB.  eNodeB uses this PLMN ID to select the core network and in turn MME.

This feature of eNodeB serving multiple operator is also called 'Multi Operator Core Network' (MOCN).

Whenever a hardware is used for Multiple operators, there would be fairness would come in picture.  Due to this feature, identification of contexts (whether it is PDCP, GTP, RLC, MAC, IPsec or Qos) will need one  additional parameter - Operator ID.  Note that TEID which is used to terminate the GTP tunnels may not be unique across multiple operators.  In addition to this,  I believe following features would be expected from eNodeB to support MOCN.

• Additional identification parameter, Operator ID in user plane modules.
• Fairness in allocating resources in eNodeB (Buffers,  Contexts etc.. ) and Radio resource management if same cell is used by multiple operators.
• VLAN Support - One or few dedicated VLANs for each operator.  
• DHCP Client (Ipv4, IPv6) - Multiple instances :  eNodeBs typically get the IP address from the DHCP Server.  Since there are multiple VLANs due to multiple operators, DHCP client also needs to be capable to get multiple IP addresses - one for each operator (VLAN).  
• If other L3 connectivity protocols used instead of DHCP such as PPP, then one needs to ensure that these L3 connectivity protocols too get multiple IP addresses - one for each operator.
• eNodeB should ensure that right source IP addresses are used for GTP tunneling and Ipsec tunneling.
• Fairness to ensure that one operator traffic does not overwhelm the CPU
• Radio bandwidth is normally taken care as part of radio resource management on per operator basis.
• Incoming traffic from backhaul network is expected to be policed at the ingress port. Each operator VLAN may be configured to police the traffic or schedule the traffic coming from different VLANs to CPU fairly (weighted if configured).  Due to scheduling, packets may be pending in the queues for future scheduling.  This can eat up buffers and new packets may not get received.  So, there should be limits on number of buffers each VLAN can occupy at any time.  
• Outgoing traffic to Bachaul network also need to be controlled on per operator (VLAN) basis.  Note that all VLANs share the same physical link. Hence the outgoing traffic needs to be controlled on per VLAN basis to ensure that physical link is not overwhelmed.  Traffic shaping and scheduling on per VLAN basis is expected.  Within each VLAN,  priority based queuing might also may require traffic shaping & scheduling. Hence, eNodeB is expected to provide hierarchical shaping and scheduling.  

LTE PDCP from eNodeB perspective

Packet Data Convergence Protocol (PDCP) is one of the User plane protocols in LTE. It is present in UE and eNodeB.  This protocol sends and receives packets to and from UE and eNodeB over air interface. This protocol works along with other L2 protocols RLC  (Radio Link Control) and MAC (Medium Access Control).

PDCP layer works on top of RLC and it transfers the UPLINK packets to the GTP layer which in turn tunnel the packets to core network (Evolved Packet Core - EPC).   It receives the downlink packets from GTP layer and send them onto RLC which in turn sends them to UE. That is PDCP layer sits in between RLC and GTP layers.

This particular post talks about PDCP layer details in eNodeB.  PDCP is user plane protocol. Control plane protocol RRC configures the PDCP entities in the user plane.

PDCP layer is described in 3GPP 36.323 standard.

PDCP functions : 

PDCP layer is expected to do following:

• Security function over the air interface :  
• Ciphering and Deciphering of user plane and control plane data.
• Integrity protection and verification for control plane data:  Note that there is no integrity protection offered to the user plane data.
• Sequence number is used to detect anti replays.
• Header compression and decompression for user plane data:  Note that there is no header compression for control plane data.  ROHC (RFC 4995) is used to reduce the headers to save bandwidth of air interface. ROHC is mandatory for voice traffic.  Note that in LTE, both voice and data use packet switching.  Typically for every 32 bytes of voice data around 40 bytes of headers are added (RTP, UDP, IP) in case of IPv4 and around 60 bytes get added in case of IPv6. That is quite a bit of overhead.  ROHC is expected to reduce the over head to few bytes.  For data traffic, ROHC is not mandatory, but it is good to have.
• Handover :  As discussed in earlier post  there are two types of handovers - seemless and lossless.  Seemless handover is typically used for radio bearers carrying control plane data and user plane data that is mapped to RLC UM (Unacknowledged mode).  In seemless handover,  header compression contexts are reset and Sequence numbers (COUNT) values are set to 0 in the target eNB.  PDCP SDUs that have not been transmitted will be sent to the X2 interface (S1 interface in case there is no X2 connectivity)  GTP tunnel to the target eNB.   Lossless handover is typically applied to the radio bearers that are mapped to RLC AM.  In this handover mode too,  header compression context is reset, that is,  ROHC context is not transferred to the target eNB. In lossless handover pending downlink packets -  PDCP SDUs for which no Acks are received from UE,  PDCP SDU which were not transmitted and new GTP packets that are coming from S1 interface in source eNB - are sent to the target eNB.  Similarly, uplink packets which are received out-of-order also sent to the target eNB. Control plane in source eNB sends the 'Next Transmit Seq num' and 'Next expected receive sequence number' to the target eNB for each RAB.  Optionally it also sends the bit map of PDCP sequence numbers of the packet which it would expect UE to retransmit.  This information is passed to the the target eNB via SN-STATUS-TRANSFER.  I guess this information would be used by target eNB PDCP to send the PDCP status transfer control message.
• Discard function :  This allows packets to be discarded if PDCP layer did not successfully send the packets for 'discard timeout' time.  
• Duplicate discarding : If PDCP layer receives duplicate packets (packets with same sequence number), the it discards them and does not send them to upper layers.

Some points to note :

PDCP specification goes in great lengths on PDCP data transfer procedures and details out internal implementation such as state variables to be maintained for received and transmit operations. These state variables are used to assign sequence numbers during transmit time and verify/discard/send up the packets which are received from RLC layer.  I will not go into those details here as the specification is very clear on them.  One thing to note is that these procedures are described from UE context. Same are valid for eNB PDCP implementation too. But note that UE PDCP transmits the UL packets to RLC  and receives DL packets from RLC.  In case of eNodeB,  PDCP layer transmits DL packets to RLC  and receives UL packet from RLC.  So, keep in mind while going through spec document.

There are two kinds of PDCP bearers:  SRB (Signalling Radio Bearer) and DRB (Dedicated Radio Bearer). There are only two SRBs - SRB1 and SRB2.  These are used by control plane protocol to send the packets to the UE.  DRBs are used for sending voice and data.  There would be as many DRBs as number of QoS streams.

There are two kinds of packets in PDCP -  Data packets and Control packets. Packets that are sent on  SRBs and DRBs use data packet format. Control packets are used by ROHC to provide feedback to the compressors from decompressors.  Control packets are also used by PDCP layer to send the PDCP sequence number status to the peer (Packet sequence numbers that are received out-of-order).  

As discussed before sequence numbers are sent along with the data packets to the  peer to do the in-order delivery of packets to its user entity.  To preserve the the bandwidth on the air, only least significant bits of sequence number is sent.  Most significant bits are called HFN.  Based on the window size, there are two sizes are chosen for sequence number that are sent along with the packet - 7 bits (User plane Short SN)  and 12 bits (User plane long SN). Typically short SN is used for UM mode and long SN is used for AM mode.

There is one PDCP context for radio bearer. PDCP context is identified by four tuples - Virtual Instance ID,  Sector ID,  C-RNTI and LCI (logical channel identifier). Please see this post for more details on virtual instance ID and sector ID.  Both sides of PDCP - RLC and GTP - would use same identifiers to identify the PDCP context. Hence only one search table is good enough (Implementation information).

There is one to one correspondence between PDCP SDU and PDCP PDU. That is there is no segmentation and concatenation functions in PDCP layer.  Addition of PDCP header,  applying compression and security on the PDCP SDU makes the PDCP PDU.  Similary deciphering, decompression and removal of PDCP header makes PDCP SDU from PDCP PDU.

PDCP Status report is expected to be generated as part of PDCP reestablishment if the RB is setup to send the status report. This report is sent for PDCP PDUs that are received from RLC (UPlink packets).  Fields that are to be sent along with the status report is described in section 5.3.1 of 36.323 spec.

PDCP Layer in eNB also may receive the PDCP Status report from the UE indicating out-of-order packets it had received. PDCP layer is expected to work on removing the PDCP SDUs that are pending in the transmit queue and acknowledged by the peer via status transfer message.  Also, it is expected to send the status report to control plane as CP may require to send to the target eNB if it is in handover stage for that UE.

I always wondered how and when PDCP generates the status reports. From the UL & DL Data transfer procedures in Section 5.1 of 36.323,  one can observe that the PDCP PDU on receive direction (from RLC) don't get stored in PDCP layer.  They are given to the upper layers immediately after Security and RoHC processing.  PDCP layer assumes that the packets are given in order by the RLC and hence don't need to store the packets to do inorder delivery to the upper layers.  Description of "PDCP Status Report" in section 5.3 of 36.323 says that status report is sent for the PDUs which were received out-of-order. It gives an impression that packets are stored in the PDCP layer for inorder delivery to the upper layers.  If not, how does PDCP layer send the bit map indications of the out-of-order packets.  So, I thought section 5.1 and section 5.3 are contradictory.  Then there is some enlightenment.  This only happens during PDCP reestablishment time.  Control plane protocols when they indicate reestablishment to the local RLC,  RLC layer sends the PDCP PDUs (RLC SDUs) it has to the PDCP layer as is basis. That is, there could be some missing packets at that time.  THis is only case, PDCP layer gets  the packets with some missing PDCP PDUs.  PDCP layer is also informed of reestablishment by the control plane.  PDCP upon receiving the packets from RLC layer is expected to send the status report to the peer with bitmap of packet sequence numbers so that the peer PDCP can remove the SDUs at its transmit side that were acknowledged in the status message.

I also had some confusion on PDCP discard for some time.  PDCP specification says that (Section 5.4 of 36.323) timer is started for every packet that is submitted to the PDCP layer by upper layers. If there is no successful transmission acknowledgment from the local RLC for the packet within 'discardTimer' timeout value, then PDCP can drop the packet.  I was thinking for some time that how does remote PDCP layer know about this drop. I thought remote peer waits on the packet (sequence number) endlessly . But from UL/DL data transfer procedures, it is clear that PDCP receiver does not wait on any missing sequence number packet as its window keeps moving right.

PDCP Interfaces:

PDCP layer interfaces with three neighboring modules -  RRC control plane,  RLC and GTP.  Ofcourse, there is initialization, configuration, monitoring etc.. Following sections describe the interface with RRC, RLC and GTP.

RRC to PDCP interface :  Following interface would need to be exposed by PDCP user plane layer to the RRC in control plane.

•  Interface to create PDCP Contexts for SRB and DRB in PDCP layer :  Parameters for this function at high level are:
• Virtual Instance ID,  Sector ID, C-RNTI (Cell - Radio network Temporary Identifier).
• LCI (logical channel identifier).
• Reference to the control plane:  Some opaque information to pass along with indications to the control plane.
• SRBOrDRB boolean flag
• If SRB:
• Unacknowledged or Acknowledged Mode
• If it is Unacknowledged, then the direction of the packets (Transmit only, Receive only or both).  In acknowledged mode, it is assumed that the directios is 'Both' always.
• Integrity Information (Y/N)
• Algorithm 
• Key
• Cipher information (Y/N):
• Algorithm
• Key.
• There is no RoHC for SRB.
• Sequence number size is not configured for SRB by RRC. Use configuration 'Default SN Size for SRBs' to find the sequence number size.
• If DRB:
•  Active/Inactive:  Normally it is active. But in handover cases,  target eNB might create the PDCP context as part of Handover preparation and program the PDCP starting TX sequence number and Receive expected sequence number and Bit map when the X2 protocols sends SN_STATUS_TRANSFER message (Refer to 36.300 Figure 10.1.2.1.1-1:  Intra MME/Service Gateway HO).  Since these two events happen at different times, it is required that PDCP does not start processing the packets until PDCP sequence numbers are known to it.  To facilitate this, I believe control plane first creates the PDCP context in 'Inactive' mode and activates it at later time using some other API function.  If PDCP receives the SDUs from the upper layer while it is inactive, it is expected to hold them from processing until it is activated. 
• Unacknowledged or Acknowledged Mode
• If it is Unacknowledged, then the direction of the packets (Transmit only, Receive only or both).  In acknowledged mode, it is assumed that the directios is 'Both' always.
  
• Integrity Information is not valid for DRB packets.
• Cipher information (Y/N):
• Algorithm
• Key.
• RoHC (Y/N)
• Profile IDs:  Bit mask of compression profiles (RTP/UDP/IP,  UDP/IP,  ESP/IP, IP, TCP/IP , v2 RTP/UDP/IP, v2 UDP/IP, v2 ESP/IP and v2 IP)
• maxCID:  Maximum flows.  
• Large CID is derived from the max CID.  If maxCID > 15, large_cid is true else large_cid is false.
• Sequence number size : 5 bits, 7 bits and 12 bits.
• Handover case:  if this PDCP context is established in target eNB, it also sends the PDCP sequence numbers and Bit map of packets that were not received by source eNB.
•  Interface to terminate PDCP contexts :  I am not sure whether there is any need to provide deletion of each individual bearer.  I have a feeling control plane does not delete each one of them.  So, it is required to have terminate bearer function for all bearers belonging to UE.  Parameters:
• Virtual Instance ID,  Sector ID, C-RNTI
• Interface to prepare PDCP context for re-establishment:  As part of this, PDCP is expected to wait for the packets sent by RLC which came in out-of-order from UE.  These packets would be processed and given to GTP-U.
• Interface to set PDCP reestablishment on per SRB and DRB basis in PDCP layer :  This interface function is expected to be called by control plane when it requires reestablishment of PDCP context.  This function is expected to send PDCP status transfer message to the UE and also it is expectedt set the Cipher and integrity information to the context. If there are any pending DL packets, they get retransmitted with new cipher context. Parameters:
• Virtual Instance ID, Sector ID, C-RNTI, LCI to identify the bearer.
•  Cipher Information (Y/N): As part of reestablishment new keys may be established.
• Algorithm
• Key
• In case of SRB, integrity information (Y/N):
• Algorithm
• Key 
• Please refer to the Section 5.2 of 36.323 to understand how to setup transmit and receive sequence number for different modes.
• ROHC context is reset if it is applicable.

•  Interface to indicate the handover of  DRBs:  This function is expected to be called by control plane in the source eNB as part of handover execution phase.  When the PDCP layer gets this indication from the control plane, it should start forwarding the un-acknowledged downlink packets and uplink packets that are received out-of-order. It is expected that this function is called by control plane after it instructs (re-establishment) RLC. 
•  Interface to send control messages via SRBs:  SRBs are used by control plane. This function can be called by control plane to send the packets on SRB.

PDCP to RRC interface: 

•  Interface to inform RRC of received Status Transfer message from peer : Using this interface point, PDCP informs the content of status transfer message to the control plane.  It sends the 'Reference information' that was set in the PDCP context by control plane during PDCP context creation. It helps control plane to corelate its context easily. Information from this indication would be used by RRC during handover execution phase.
•  Interface to inform SRB data indications : This interface function gives the messages received on SRB from peer PDCP to the control plane.

PDCP to RLC interface:

• Interface to send PDCP PDUs including PDCP control and data PDUs :  This interface point is needed to send the PDCP PDU to RLC.  RLC also uses same identification parameter to match its context as PDCP does. Parameters include Bearer identificatio (Virtual instance ID,  Sector ID, C-RNTI, LCI), PDCP PDU packet buffer and message ID.  It is expected that message ID is returned when the RLC calls ack function to provide success & failure of delivery.  It helps PDCP implementation to find the matching SDU, stop the discard timer and remove the entry.

RLC to PDCP interface :

• Interface to indicate the new PDCP PDUs (new packets) - Multiple of them:  This function can be used by RLC to give PDCP PDUs to the PDCP layer.   Multiple packets can be given at one time.  RLC might be buffering the packets if they come in out-of-order. When the missing packet comes along,  All those packets can be given at once. 
• Interface to indicate the pending PDCP PDUs - During reestablishment time (Multiple packets can be sent using one call) :  This function can be used by RLC to give pending PDCP PDUs in the RLC.  Along with the last packet, it can indicate that it is last packet. 
• Interface to indicate acknowledgment of PDU sent using PDCP to RLC interface functions :  This function is expected to be used by RLC to give success/failure ack to the PDCP PDUs that were sent to the RLC before.

PDCP to GTP interface:

• Interface to send PDCP SDUs to the GTP : This function is used by PDCP to give PDCP SDU (IP packet) to the GTP layer.
• Interface to send Downlink forwarding packets (Upon handover) to the GTP layer & Interface to send Uplink forwarding packets (Upon handover) to the GTP layer:  This function is called when the PDCP layer is informed of handover for a given context.  Both UL and DL packets are to be sent along with sequence number to the GTP. Last packet is expected to be indicated explicitly.  Since GTP waits for the last packet indication,  it is necessary that GTP is informed of 'no more packets indication' even if there are no packets to forward to target eNB.

GTP to PDCP interface:

• Interface for GTP layer to send new packets to the PDCP layer in downlink direction :  This function is called by PDCP upper layers to send the packets to PDCP.
• Interface for GTP layer to send DL-forwarded packets (During handover) & Interface for GTP layer to send UL-forwarded packets (During handover) : THis function is normally called in target eNB during handover execution phase.  These packets are sent to the PDCP layer with the sequence number. 

I have written this description based on my understanding of PDCP specifications. If somebody finds any inconsistency or if i had made any mistakes, please drop a comment or send an email.

LTE GTP User plane eNodeB - Tutorial

This particular tutorial is limited to explaining some concepts related to GTP-U between eNodeB and S1 Peer Gateway.  Also it is limited to giving information from the context of eNodeB.

GTP-U specifications are defined in 3GPP TS 29.281 at www.3gpp.org.

GTP-U in eNodeB tunnels IP packets received from PDCP layer to S1 gateway by tunneling them in GTP tunnel in uplink direction.  In downlink direction, eNodeB detunnels the packets from GTP tunnel and pass the packets onto the PDCP layer to send them to mobiles (UEs).

GTU-U is like any other tunnel protocol adds protocol specific headers and then finally adds the outer IP header to route the packets to peer -  GTP-U header, UDP header and outer IP header to the packet as part of tunneling.  As part of de-tunneling, it removes the GTP-U, UDP and outer IP headers.

As part of S1-AP and X2-AP protocols,  GTP tunnel contexts are created in GTP-U module. 

Identification of GTP-U tunnel in eNodeB:

• PDCP layer in eNodeB gets hold of IP packets that are sent by UE to the core network and passes them to the GTP layer.  In addition to providing the packets, it also provides C-RNTI (Cell Radio Network Temporary Identifier), Logical Channel ID and a downlink handover flag (handover stage) or uplink handover flag. One of the handover flags is set only when the corresponding DRB in PDCP was informed by the RRC in control plane to forward the packets to the target eNB as part of handover process.   Why is the handover stage flag required?  It is because, during handover time, there could be more two GTP tunnel context for the same bearer (C-RNTI and LCI combination) - One established by S1 protocol to core network and another one established by X2-AP module to the target eNodeB.  If X2 connection is not available, then S1-handover would be used for Intra-LTE handovers, in which case,  you would have two S1-AP tunnels.   This flag would help in choosing the right GTP-U tunnel context.   Using these three tuples (C-RNTI, LCI and handover flag), GTP-U tunnel context is identified within a sector.  Processing that should happen for downlink traffic during handover is described later.  I am not sure about this, but my impression is that C-RNTI is unique within a sector. Since on GTP-U software instance works on multiple sectors (typically 3),  sector ID needs to be one of the key values to identify the GTU-U tunnel in uplink direction.  If same eNodeB GTP-U is used by multiple operators,  then one more parameter (operator ID) will be another identification parameter required to identify GTP-U tunnel.  As of this writing, I did not get any information that same eNB transport hardware is shared by multiple operators.  If only one operator, then the operator ID is always assumed to be 0.  Finally, we have following parameters to identify the GTP-U tunnel in uplink direction.
• Operator ID (Virtual Instance ID)
• Sector ID
• C-RNTI
• LCI
• Uplink handover flag (If  uplink traffic is handed over to target eNB)  or downlink handover flag (if downlink traffic is handover to the the target eNB).  Note that in non-handover scenarios, both the flags would be set to 0. 
• In downlink side,  packets are given to the GTP -U layer after Ipsec decryption is done.  IPsec layer is expected to pass just the packet.  Packet contains GTP-U header and outer IP header.  GTP-U header contains TEID (Tunnel End Point ID).  Using Source and Destination IP addresses of outer IP header and TEID,  GTP-U tunnel context can be identified.  Key parameters to identify the GTP-U tunnel in the downlink direction are:
• Operator ID (Virtual instance ID)
• Source IP  (from outer IP header of GTP)
• Destination IP (from outer IP header of GTP).
• TEID : This is the ID which was sent to the SGW by the control plane software for SGW to send the packet to the right GTP-U tunnel.

GTP-U Operational concepts in eNodeB:

Working with Path MTU discovery procedures by UE and end points on core network side: 

GTP-U tunneling increases the packet size since it adds GTP-U header and outer IP header to the packet.  If the MTU of the outbound interface is less than the resulting packet size, then packets would need to be fragmented on the sender side and they need to get reassembled before handing over the packets to the received GTP-U layer.  Fragmentation and reassembly not only adds CPU cycles, but also increases the amount of bandwidth that would be used in the backhaul network.  Also, if one fragment is dropped for whatever reason,  the entire set of fragments corresponding to the packet would get dropped as part of reassembly process.  This can lead to more bandwidth on the wireless link too.  I have my own theory on thi which I explained later on.   Anyway to reduce the fragmentation,  PMTU discovery process is normally adopted by any network elements and UE and any other end points are no exception.  GTP-U layer in eNodeB should honor Path MTU discovery process even for IPv4 networks.GTP-U layer also can follow similar approach that are followed by other tunneling protocols.

• GTP-U module in eNodeB:
• GTP-U layer receives the IP packet with DF bit set.
• GTP-U layer knows the amount of header information it is going to add as part of tunneling.
• If the resulting data size is less or equal to  the PMTU value which is already stored in the GTP-U context,  then there will not be fragmentation.  It also need to ensure to put the DF bit in the outer IP header to ensure that it discovers new PMTU value if any.  As part of GTP-U context creation,  it can set its PMTU value from the interface MTU value.
• If the resulting data size is more than the PMTU value stored in the GTU-U context, it generates ICMP Destination unreachable message with the PMTU value  = PMTU value stored in the GTP-U context - Header information GTP-U layer adds.
• When GTP-U module receives the ICMP Destination Unreachable - Path MTU excededed message, it should find out the GTP-U context from the payload of ICMP error message. It should only interpret the ICMP error messages destined to it.
• Update the PMTU value stored in the GTP-U context if it is less than the value in ICMP error message.  This updated value would be used by new packets coming from UE at later time and can be used to generate the ICMP error message itself.

If IPSec packet processing is also going to happen, then it is better to keep the size of headers being added by IPsec too while doing above operations. 

Ok, now to my concerns: Wireless link bandwidth may be more precious compared to backhaul network bandwidth.  By doing PMTU discovery, TCP MSS value would be decreased.  That is more IP packets would be generated for a given amount of data when path MTU discovery process is run.  More packets is equal to more overhead on the wireless link.  So, I believe that PMTU discovery will not be used in IPv4 world.  Having said that, above logic is expected to be implemented by eNodeB.

IPv6 Support:

IPv6 is becoming popular in RAN too.  It is expected that Ipv6 is also supported.  Here rather than Ipv4 outer IP header, IPv6 IP header is used.  Note that internal packet can be IPv4 or IPv6.

HopCount/TTL Decrement:

TTL Decrement operation on the internal packets need not be done by the GTP-U layer.  But I think it should check for TTL and discard the packet if the value is 0.

MBMS (Multmedia Broadcast Multicast Service) - Section 4.2.6 of 3GPP-TS-29.281 :  I don't have much idea on how this works in the control plane.  My guess is that IGMP proxy mechanism is used in eNodeB.  Based on this information, EPC somehow creates the GTP tunnel with all eNBs that require multicast packets.  It appears that EPC sends the TEID (GTP Tunnel ID) via control plane protocols. GTP-U layer gets the request from the control plane protocols to create the GTP Tunnel with the given TEID. Note that in unicast, the receiver of the GTP packets would give the TEID to peer. But in multicast,  sender of the GTP packets give the TEID to the receiver.  I guess it is required this way to ensure that all eNB get the same TEID and only one copy need to be sent from the EPC to eNBs over backhaul network.

My understanding of the multicast packet processing is simple in eNB.  When GTP-U receives the packets and detunnels them (in downlink direction), it needs to inform PDCP that these packets are Multicast packets. L2 stack of the LTE might use this information to send the packets over to Multicast transport channel.  I am not sure whether there is any RAB (Radio Access Bearer) created for multicast packets.  I think it should. If that is the case, then there is no special flag is also required between GTP-U and PDCP.  Does anybody know? If so, somebody inform me which specification talks about this?

Handling of Sequence numbers -  Section 4.3.1 of 3GPP-TS-29.281:

It appears from the description of this section,  only requirement on eNB to ensure to pass the sequence number in the GTP header which comes along with the packet should be sent to the PDCP.  I don't know what PDCP does as it has its own sequence number.  Though this section talks about reordering of received packets, it did not talk about what should be done in case some packets got dropped on the way.  There is no mention of retries and acknowledgments. It makes me believe that this section is not applicable for eNBs.  I would love to hear from others on this.

This section also talked about sending message number (sequence number) along with request signaling messages if there are responses.  GTP Echo-Request is  path management request message to find out whether the peer is active and live.  Peer responds by sending Echo-Response.  Response is expected to have same message number for sender to match the response with request message.

Interfaces with other modules:

3GPP specifications goes into great details on interfaces among wireless network elements - eNB, SGW, MME, UE etc..  For right reasons, 3GPP specifications don't  describe the interfaces among different functions within each network element. It is left to the developers working on particular network elements. Vendors of these equipment may get  some software elements from different software vendors.  In those cases, it becomes easy for integrators if there are standard interfaces. I did not find any software API definition.  I tried to give my understanding of interfaces among different software modules that interact with GTP-U.  In following sections,  I have combined both GTP-U, GTP-U relay entity and UDP into one and I am calling that combination as GTP-U.

GTP-U interfaces with four other software elements:

• Control Plane (S1 and X2) - for setting up and tearing down  the GTP-U tunnels.
• Packet Processing :
• PDCP -  For sending packets to the UE in downlink direction and to receive packets from PDCP in uplink direction.
• IP Forwarding (with and without IPsec) -  to send and receive packets from the EPC or other eNodeBs in handover cases.
• Management plane - for initialization, configuration,  reading statistics, sending alerts/logs etc..

Initialization & Configuration:

• Init():  This function is typically needed to allocate memory for different contexts - GTP tunnel contexts and GTP Path Management contexts.  Note that there could be as many GTP tunnel contexts as number of Radio Access Bearers.  Each UE typically have three SRBs and at least three DRBs - Default, Voice dedicated and Data dedicated bearers.  GTP tunnels are established only for DRBs though.  There could be as many GTP path management contexts as number of S1 peers. X2 peers will not participate in echo request/response path. Parameters can include:
• Maximum number of Virtual Instances
• Maxinum number of sectors
• Maximum number of GTP Tunnel contexts across all Virtual instances and Zones.
• Maximum number of Path Management contexts.
• Configuration:
•  SetEchoConfiguration() :  Echo request/response messages are used to find out the liveness of the peer.  Sender retransmits the echo request to the peer for every X number of seconds (T3_RESPONSE timeout) for Y number (N3-REQUESTS) if it does not receive response.  If it does not receive response for y number times, then it GTP-U module is expected to send the alarm to the control plane.  Parameters
• T-RESPONSE timeout : Default: 60 seconds
• N3-REQUESTS : Default is 5.
• GetEchoConfiguration():  Returns the configuration that was made before.
• SetUDPSourcePortRange():   GTP-U is application on top of UDP.  Destination port of the GTP tunnel is standardized and it is 2152.  But the source port of each GTP-Tunnel to a given destination has to be unique.  Since UDP is also used by control plane, control plane may like to reserve a range for GTP-U. This function can be used to set that range.  COntrol plane is not expected to use ports in these port range for its purpose.  Parameters:
• Min source Port  : Default - 63000
• Max Source Port :  2^16 - 1
• Destination port:  Default 2152 (In case somebody wants to give some other port number during debugging time).
• GetUDPSourcePortRange():  To read the port range that was set using SetUDPSourcePortRange() function. 
• SetHousekeepingTimeOuts():  Parameters include
• Amount of time to wait after forwarding (for handover purposes) tunnels are created and before it gets indication to transfer the forwarded packets to the PDCP layer.  Valid at target eNB.
• Statistics:  There are no specific statistics defined by standards. I suggest to define the statistics for debugging.

Tunnel Establishment:

Tunnels are established by S1 and X2 protocols running in control plane. There are as many GTP tunnels as number of Radio Access Bearers.  It is good to visit the handover procedures before defining the interface for tunnels establishment as well as packet level API with PDCP.

Even though there are different types of handovers,  from GTP-U perspective they are same.  There are two types of handovers -  X2 based handovers and S1-based handovers.  if both source eNB and target eNB belongs to the same MME, then X2 based handover is used.  If there are different MMEs, then S1 handover would be used.  For more details about the handover, please see 3GPP TS 36.300 (Overall architecture) and 23.401.

Handover is applied for each RAB independently.   Typically only RABs that are setup using acknowledged mode do lossless handover. Unacknowledged mode RABs only do seemless handover. Control plane knows which RABs, in turn, which GTP-tunnels require handover and accordingly programs them in the user plane software in both PDCP and GTP-U layers. 

As part of lossless handover source eNB is expected to send unacknowledged downlink packets waiting at the  RLC layer and  new packets that are coming in from SGW.  Ofcourse, first GTP-U needs to send the unacknowledged PDCP packets and then the packets that come from the SGW.  In case of seemless handover only the new packets that are coming in from SGW are transferred to the target eNB.

Since PDCP packets already have sequence number associated with them by the PDCP layer, it is necessary that PDCP SN is sent along with the packet to the target eNB which is expected to send to the UE.   SGW sends the END-MARKET GTP packet to indicating the SGW will not be sending the packet to source eNB.  Target eNB is expected to send the packets with the PDCP SN first to the UE, then the packets which were sent by source eNB which don't have sequence number and then only it should start sending the packets which came from SGW to the target eNB.  Note that X2 or S1 protocol (based on whether X2 or S1 handover) create GTP tunnels as part 'Handover Request Ack'. It is expected that source eNB receives the GTP tunnel ID for sending the pending downlink packets. 

GTP-U also is used to send the uplink packets that were received out of sequence by eNB in PDCP layer.  These packets are also can be sent to the target eNB.  PDCP status transfer message that is sent to the UE as part of handover execution tells the UE the sequence numbers of the packets which were received out-of-order.  UE then accordingly only has to do selective retransmissions and thereby saving air bandwidth.  In any case, source eNB sends these out-of-order received packets to the target eNB with the sequence numbers.  The tunnel it uses to send these packets is new tunnel created for this purpose. This TEID of the tunnel is also sent along with Handover request Ack by the target eNB.

Normally each tunnel has both self TEID and peer TEID.  But in case of tunnels that are setup for forwarding the packets  have peer TEID with respect to source eNB and self TEID with respect to target eNB.

• GTPUCreateTunnel() :  This function is used to create normal tunnels as well as forwarding tunnels during handover.  Control plane creates as many tunnels as number of DRBs (Dedicated Resource Bearers).  Parameters include:
• CP Reference Identification:  Control plane can put its reference which is expected to be sent along with the indications.
• Virtual Instance ID : in case of multiple operators sharing the same eNB hardware.
• Identification parameters in uplink direction:
• Sector ID,  C-RNTI and Logical Channel ID.
• Tunnel Type :  Normal,  Forwarding-downlink-sourceNBside,  forwarding-downlink-targetNBside,  forwarding-uplink-sourceeNB and forwarding-uplink-targeteNB.
• Normal :  Indicates normal tunnel established between eNB and SGW.
• Forwarding-downlink-sourceeNB :  Indicates that this tunnel is being created at source eNB for forwarding pending downlink packets. GTP-U in source eNB would start expecting the PDCP layer to give all downlink packets which were not acknowledged followed by local end-marker  along with the last packet.  GTP-U knows from the end-marker that no more packets should be expected from the local PDCP layer. Even if there are no packets are to be forwarded,  GTP-U expects the PDCP layer to indicate end-marker.
• Forwarding-uplink-sourceeNB:  Indicates that this tunnel is being created at source eNB for forwarding the out-of-order uplink packets.  GTP-U in source enB would start expecting the PDCP layer to give all uplink packes that came in out-of-order followed by local end-marker along with the last packet.  Even if there are no packet to be forwarded,  GTP-U expects PDCP layer to give end-marker. 
• Forwarding-downlink-targeteNB: Indicates that this tunnel is being creates at target eNB for receiving the forwarded downlink packets from source eNB.
• Forwarding-uplink-targeteNB:  Indicates that this tunnel is being creates at target eNB for receiving the out-of-sequence uplink packets at the sourceeNB.
• Identification parameters in downlink direction:
• TEID Self (Only if tunnel type is normal,  forwarding-downlink-targeteNB and forwarding-uplink-targeteNB)
• local IP address (Could be IPv4 or IPv6)
• Peer IP address (Could be IPv4 or IPv6)
•  TEID Peer: Required to send the packets to the peer. Only required if tunnel type is Normal,  forwarding-downlink-sourceeNB and forwarding-uplink-sourceeNB.
• DSCP Marking:  Different RABs are created due to different QoS treatment required.  QoS treatment should be not only limited to MAC scheduler enforcement, but also to be enforced on uplink traffic before sending the packets out onto the backhaul network.  Normally the DSCP value of the IP packets coming from the UE would have right value. In case, they don't send the right value, then it can be set here.  QoS shaping module at the egress level (towards backhaul network) would use the DSCP value of the IP packet to provide differential treatment to the packets. In case of IPsec layer between GTP and QoS,  IPsec is normally configured to copy the DSCP value from inner IP header to IPsec outer IP header there by marked DSCP value is preserved all the way to the QoS module. 
• Active flag: Indicates whether this tunnel is active.  This flag is required due to 'Multiple Preparation' feature.  Source eNB may initiate handover procedure with multiple target eNBs, but only one will be activated eventually.
•  GTPUTargeteNBStartXmitOfForwardedPkts():  This functions will be used by control plane to initiate the transmission of forwarded packets. This function is called on target eNB.  When control plane (RRC) receives the 'RRCReconfigurationComplete' message from the UE by target eNB,  it is expected that this function is called  after the control plane programs the PDCP layer with the information it receives as part of SN-STATUS_TRANSFER.  GTP-U on targeteNB is expected to keep the forwarded packets  until this function is called. It is also expected that GTP-U layer of target eNB sends the packets with sequence number first before sending the forwarded non-seq packts and then the new packets which are coming to target eNB directly from SGW.   Note that when this function is called GTP-U layer of target eNB can send both DL and UL packets to the PDCP layer.  I guess UL packets would be used by PDCP layer for sending status transfer PDCP control message to the UE.  These packets will be eventually would be given back to the GTP layer in order to the GTP tunnel which is created with new SGW (Normal tunnel).  
• Parameters include:
• Identification parameters such as Virtual Instance ID, Sector ID,  C-RNTI, LCI and tunnel type.
• I think that there is no special function required in source eNB from control plane  to initiate handover.  Tunnel creation with appropriate tunnel-type itself is the indication that it can forward the packets. But PDCP user plane layer need to give the packets to the GTP-U layer. 
• GTPUChangeActiveStatus():  This function can be used by control plane to activate or deactivate an particular tunnel. This is required due to 'Multiple preparation' feature.  
• GTPU Do Hairpin Transfer:  This function is called by eNodeB control plane on original S1 tunnel to do the hairpin. This is expected to be done as part of processing HandoverComplete message from target eNodeB.  Once it is done,  GTP-U layer forwarding downlink packets over GTP tunnel established to the target eNB.
•  GTPUTerminateTunnel():  This function terminates the tunnel in the GTP-U.  Control plane (S1 and X2) call this function when it finds that this tunnel is no longer required. Parameters:
• Identification parameters such as Virtual Instance ID, Sector ID,  C-RNTI, LCI and tunnel type.
• GTPUGetTunnelInfo():  This function can be used by control plane to get the information and statistics of a given tunnel.  Parameters:
• Identification parameters such as Virtual Instance ID, Sector ID,  C-RNTI, LCI and tunnel type
• Output:  Statistics counters maintained on per tunnel basis and any state information.
• GTPUGetFirstTunnelInfo() and GTPUGetNextTunnelInfo():  Meant for walking through the tunnels.  Mainly for debugging and SNMP.
• xxxGTPUErrorIndication():  This function is expected to be called by GTP-U to indicate the peer provided error to the control plane (Error indication message received by the GTP-U).
• xxxGTPUPathErrorInd():  This function is expected to be called by GTP-U when it determines the peer is not alive.  As you might have seen, I did not define any function for control plane to initiate the path liveness checks.  Whenever the normal tunnel is created,  GTP-U is expected to create path management context to check the liveness of peer.  It is possible that there are more tunnels to the same peer. In that case, GTP-U should not be creating duplicate path management contexts.  When there are no tunnels to the peer,  path management context can be removed as there is no need to do liveness checks.  One way to do this in implementation is to have 'counter' (atomic) indicating the number of tunnels that have same peer.  This gets incremented whenever new tunnel is established for which path management context already exists.  It gets decremented when the tunnel is terminated.  When the count becomes 0, GTP-U can remove the context.

 Interfacing with PDCP:


GTP-U interfaces with PDCP to

• to send packets to the UE in downlink direction. 
• to get hold of packets from the PDCP in uplink direction 
• to get hold of packets which were not acknowledged in downlink direction. This is required in handover cases.  These packets would be sent by GTP-U to the target eNB.
• to get hold of packets which were received by PDCP from UE (uplink) out-of-order. This is required in handover cases. These packets would be sent by GTP-U to the target eNB.

   

• GTPUSendNormalPktToPDCP():  This function is used to send packets to the UE . Parameters may include
• Identification parameters such as Virtual Instance ID,  Sector ID, C-RNTI, LCI.
• packet in implementation specific buffer
• GTPUSendHOSeqPktToPDCP():  This function is used by GTP-U in target eNodeB. It is used to send the packets  with the sequence number. Parameters include:
• Identification parameters such as Virtual Instance ID,  Sector ID, C-RNTI, LCI.
• Direction of packet (Uplink or downlink) : Note that as part of handover both downlink and uplink packets are sent by source eNB.  Downlink packets are meant to be sent to the UE by PDCP layer.  Uplink packets are expected to be used by PDCP to reorder the packets with newly arrived uplink packets. Note that these uplink packets would be given to the GTP-U layer in sequence. Also note that these packets would be given to normal GTP-U tunnel which was established between target eNB and SGW.
• bValid PDCP SeqNum : Indicates whether the PDCP seq number is valid.
  
• PDCP seq number:   Sequence number is valid in handover scenarios.  This is normally done by the target eNB.  GTP-U implementation gets this from the extension header "PDCP PDU Number".
  
• packet in implementation specific buffer.
• Notes:  GTPUSendNormalPktToPDCP() and GTPUSendHOSeqPktToPDCP() could have been combined into one function. It is good to have two different functions for performance reasons. If combined both,  then number of arguments and preparation of arguments would take some CPU cycles which can be saved by having two different functions.
• GTPURecvNormalPktFromPDCP() :  This function is called by PDCP or some other glue layer to give packets to the GTP-U layer.  Parameters include:
• Identification parameters:  Virtual Instance ID, sector ID, C-RNTI, LCI.
• Note that there could be multiple GTP-U tunnel matching with these parameters. There would be only one tunnel to the SGW, but all other tunnels are X2 tunnels.  Since this function is called to send normal packets, it is expected that this information (normal) is used to select the right tunnel.
• Packet in implementation specific buffer.
• GTPURecvHOPktFromPDCP() :  This function is called by PDCP or some glue layer to give HO packets to the GTP-U layer.  This function is used by PDCP to give both UPLINK and DOWNLINK pending packets.   This function is called by source eNB.  Parameters include:
• Identification parameters:  
• Virtual Instance ID, Sector ID, C-RNTI, LCI
• Type of packet:  Downlink or uplink.
• Sequence number (PDCP sequence number): This will be sent to the peer using PDCP PDU number extension header.
• Some notes:  Due to 'Multiple Preparation' feature, there could be multiple forwarding tunnels in source eNB. But fortunately only one will be active.  It is expected that this 'active' information is used to match the right tunnel.

Some implementation notes:

• There would be two contexts that would need to be maintained in GTP-U - Tunnel contexts and path management contexts.   Tunnel contexts need to be arranged in two different hash tables - One for uplink and another for downlink.  
• Key parameters to the uplink hash table would be:
• Virtual instance ID,  Sector ID, C-RNTI, LCI
• Type (Normal, Handover)
• In case of handover,  Uplink or downlink,  Active/Inactive.
• Key parameters to the downlink hash table would be:
• IP type (IPv4, IPv6)
• Source IP and Destination IP from GTP-U packet.
• TEID from the GTP-U packet
• Some notes: It may be good to have two different hash tables - one for IPv4 and another for IPv6 for performance reasons.  Some thing to think about.

Performance considerations:

• Use RCUs during packet processing. Absolutely no locks during packet processing. Locks may be okay during tunnel establishment and termination.  That is why, RCUs are the best in multicore environments.  
• Multicore processors provide statistics accelerators. Use this functionality to increment/decrement the global counters and counters that may be updated across multiple cores.

Above scribbling is really my notes that was prepared while going through the GTP-U specifications.  I am new to 3GPP and it took me a solid one week to get handle. I hope it would be useful for you and hopefully jump start your study of GTP-U.