LTE GTP User plane eNodeB - Tutorial

This particular tutorial is limited to explaining some concepts related to GTP-U between eNodeB and S1 Peer Gateway.  Also it is limited to giving information from the context of eNodeB.

GTP-U specifications are defined in 3GPP TS 29.281 at www.3gpp.org.

GTP-U in eNodeB tunnels IP packets received from PDCP layer to S1 gateway by tunneling them in GTP tunnel in uplink direction.  In downlink direction, eNodeB detunnels the packets from GTP tunnel and pass the packets onto the PDCP layer to send them to mobiles (UEs).

GTU-U is like any other tunnel protocol adds protocol specific headers and then finally adds the outer IP header to route the packets to peer -  GTP-U header, UDP header and outer IP header to the packet as part of tunneling.  As part of de-tunneling, it removes the GTP-U, UDP and outer IP headers.

As part of S1-AP and X2-AP protocols,  GTP tunnel contexts are created in GTP-U module. 

Identification of GTP-U tunnel in eNodeB:

• PDCP layer in eNodeB gets hold of IP packets that are sent by UE to the core network and passes them to the GTP layer.  In addition to providing the packets, it also provides C-RNTI (Cell Radio Network Temporary Identifier), Logical Channel ID and a downlink handover flag (handover stage) or uplink handover flag. One of the handover flags is set only when the corresponding DRB in PDCP was informed by the RRC in control plane to forward the packets to the target eNB as part of handover process.   Why is the handover stage flag required?  It is because, during handover time, there could be more two GTP tunnel context for the same bearer (C-RNTI and LCI combination) - One established by S1 protocol to core network and another one established by X2-AP module to the target eNodeB.  If X2 connection is not available, then S1-handover would be used for Intra-LTE handovers, in which case,  you would have two S1-AP tunnels.   This flag would help in choosing the right GTP-U tunnel context.   Using these three tuples (C-RNTI, LCI and handover flag), GTP-U tunnel context is identified within a sector.  Processing that should happen for downlink traffic during handover is described later.  I am not sure about this, but my impression is that C-RNTI is unique within a sector. Since on GTP-U software instance works on multiple sectors (typically 3),  sector ID needs to be one of the key values to identify the GTU-U tunnel in uplink direction.  If same eNodeB GTP-U is used by multiple operators,  then one more parameter (operator ID) will be another identification parameter required to identify GTP-U tunnel.  As of this writing, I did not get any information that same eNB transport hardware is shared by multiple operators.  If only one operator, then the operator ID is always assumed to be 0.  Finally, we have following parameters to identify the GTP-U tunnel in uplink direction.
• Operator ID (Virtual Instance ID)
• Sector ID
• C-RNTI
• LCI
• Uplink handover flag (If  uplink traffic is handed over to target eNB)  or downlink handover flag (if downlink traffic is handover to the the target eNB).  Note that in non-handover scenarios, both the flags would be set to 0. 
• In downlink side,  packets are given to the GTP -U layer after Ipsec decryption is done.  IPsec layer is expected to pass just the packet.  Packet contains GTP-U header and outer IP header.  GTP-U header contains TEID (Tunnel End Point ID).  Using Source and Destination IP addresses of outer IP header and TEID,  GTP-U tunnel context can be identified.  Key parameters to identify the GTP-U tunnel in the downlink direction are:
• Operator ID (Virtual instance ID)
• Source IP  (from outer IP header of GTP)
• Destination IP (from outer IP header of GTP).
• TEID : This is the ID which was sent to the SGW by the control plane software for SGW to send the packet to the right GTP-U tunnel.

GTP-U Operational concepts in eNodeB:

Working with Path MTU discovery procedures by UE and end points on core network side: 

GTP-U tunneling increases the packet size since it adds GTP-U header and outer IP header to the packet.  If the MTU of the outbound interface is less than the resulting packet size, then packets would need to be fragmented on the sender side and they need to get reassembled before handing over the packets to the received GTP-U layer.  Fragmentation and reassembly not only adds CPU cycles, but also increases the amount of bandwidth that would be used in the backhaul network.  Also, if one fragment is dropped for whatever reason,  the entire set of fragments corresponding to the packet would get dropped as part of reassembly process.  This can lead to more bandwidth on the wireless link too.  I have my own theory on thi which I explained later on.   Anyway to reduce the fragmentation,  PMTU discovery process is normally adopted by any network elements and UE and any other end points are no exception.  GTP-U layer in eNodeB should honor Path MTU discovery process even for IPv4 networks.GTP-U layer also can follow similar approach that are followed by other tunneling protocols.

• GTP-U module in eNodeB:
• GTP-U layer receives the IP packet with DF bit set.
• GTP-U layer knows the amount of header information it is going to add as part of tunneling.
• If the resulting data size is less or equal to  the PMTU value which is already stored in the GTP-U context,  then there will not be fragmentation.  It also need to ensure to put the DF bit in the outer IP header to ensure that it discovers new PMTU value if any.  As part of GTP-U context creation,  it can set its PMTU value from the interface MTU value.
• If the resulting data size is more than the PMTU value stored in the GTU-U context, it generates ICMP Destination unreachable message with the PMTU value  = PMTU value stored in the GTP-U context - Header information GTP-U layer adds.
• When GTP-U module receives the ICMP Destination Unreachable - Path MTU excededed message, it should find out the GTP-U context from the payload of ICMP error message. It should only interpret the ICMP error messages destined to it.
• Update the PMTU value stored in the GTP-U context if it is less than the value in ICMP error message.  This updated value would be used by new packets coming from UE at later time and can be used to generate the ICMP error message itself.

If IPSec packet processing is also going to happen, then it is better to keep the size of headers being added by IPsec too while doing above operations. 

Ok, now to my concerns: Wireless link bandwidth may be more precious compared to backhaul network bandwidth.  By doing PMTU discovery, TCP MSS value would be decreased.  That is more IP packets would be generated for a given amount of data when path MTU discovery process is run.  More packets is equal to more overhead on the wireless link.  So, I believe that PMTU discovery will not be used in IPv4 world.  Having said that, above logic is expected to be implemented by eNodeB.

IPv6 Support:

IPv6 is becoming popular in RAN too.  It is expected that Ipv6 is also supported.  Here rather than Ipv4 outer IP header, IPv6 IP header is used.  Note that internal packet can be IPv4 or IPv6.

HopCount/TTL Decrement:

TTL Decrement operation on the internal packets need not be done by the GTP-U layer.  But I think it should check for TTL and discard the packet if the value is 0.

MBMS (Multmedia Broadcast Multicast Service) - Section 4.2.6 of 3GPP-TS-29.281 :  I don't have much idea on how this works in the control plane.  My guess is that IGMP proxy mechanism is used in eNodeB.  Based on this information, EPC somehow creates the GTP tunnel with all eNBs that require multicast packets.  It appears that EPC sends the TEID (GTP Tunnel ID) via control plane protocols. GTP-U layer gets the request from the control plane protocols to create the GTP Tunnel with the given TEID. Note that in unicast, the receiver of the GTP packets would give the TEID to peer. But in multicast,  sender of the GTP packets give the TEID to the receiver.  I guess it is required this way to ensure that all eNB get the same TEID and only one copy need to be sent from the EPC to eNBs over backhaul network.

My understanding of the multicast packet processing is simple in eNB.  When GTP-U receives the packets and detunnels them (in downlink direction), it needs to inform PDCP that these packets are Multicast packets. L2 stack of the LTE might use this information to send the packets over to Multicast transport channel.  I am not sure whether there is any RAB (Radio Access Bearer) created for multicast packets.  I think it should. If that is the case, then there is no special flag is also required between GTP-U and PDCP.  Does anybody know? If so, somebody inform me which specification talks about this?

Handling of Sequence numbers -  Section 4.3.1 of 3GPP-TS-29.281:

It appears from the description of this section,  only requirement on eNB to ensure to pass the sequence number in the GTP header which comes along with the packet should be sent to the PDCP.  I don't know what PDCP does as it has its own sequence number.  Though this section talks about reordering of received packets, it did not talk about what should be done in case some packets got dropped on the way.  There is no mention of retries and acknowledgments. It makes me believe that this section is not applicable for eNBs.  I would love to hear from others on this.

This section also talked about sending message number (sequence number) along with request signaling messages if there are responses.  GTP Echo-Request is  path management request message to find out whether the peer is active and live.  Peer responds by sending Echo-Response.  Response is expected to have same message number for sender to match the response with request message.

Interfaces with other modules:

3GPP specifications goes into great details on interfaces among wireless network elements - eNB, SGW, MME, UE etc..  For right reasons, 3GPP specifications don't  describe the interfaces among different functions within each network element. It is left to the developers working on particular network elements. Vendors of these equipment may get  some software elements from different software vendors.  In those cases, it becomes easy for integrators if there are standard interfaces. I did not find any software API definition.  I tried to give my understanding of interfaces among different software modules that interact with GTP-U.  In following sections,  I have combined both GTP-U, GTP-U relay entity and UDP into one and I am calling that combination as GTP-U.

GTP-U interfaces with four other software elements:

• Control Plane (S1 and X2) - for setting up and tearing down  the GTP-U tunnels.
• Packet Processing :
• PDCP -  For sending packets to the UE in downlink direction and to receive packets from PDCP in uplink direction.
• IP Forwarding (with and without IPsec) -  to send and receive packets from the EPC or other eNodeBs in handover cases.
• Management plane - for initialization, configuration,  reading statistics, sending alerts/logs etc..

Initialization & Configuration:

• Init():  This function is typically needed to allocate memory for different contexts - GTP tunnel contexts and GTP Path Management contexts.  Note that there could be as many GTP tunnel contexts as number of Radio Access Bearers.  Each UE typically have three SRBs and at least three DRBs - Default, Voice dedicated and Data dedicated bearers.  GTP tunnels are established only for DRBs though.  There could be as many GTP path management contexts as number of S1 peers. X2 peers will not participate in echo request/response path. Parameters can include:
• Maximum number of Virtual Instances
• Maxinum number of sectors
• Maximum number of GTP Tunnel contexts across all Virtual instances and Zones.
• Maximum number of Path Management contexts.
• Configuration:
•  SetEchoConfiguration() :  Echo request/response messages are used to find out the liveness of the peer.  Sender retransmits the echo request to the peer for every X number of seconds (T3_RESPONSE timeout) for Y number (N3-REQUESTS) if it does not receive response.  If it does not receive response for y number times, then it GTP-U module is expected to send the alarm to the control plane.  Parameters
• T-RESPONSE timeout : Default: 60 seconds
• N3-REQUESTS : Default is 5.
• GetEchoConfiguration():  Returns the configuration that was made before.
• SetUDPSourcePortRange():   GTP-U is application on top of UDP.  Destination port of the GTP tunnel is standardized and it is 2152.  But the source port of each GTP-Tunnel to a given destination has to be unique.  Since UDP is also used by control plane, control plane may like to reserve a range for GTP-U. This function can be used to set that range.  COntrol plane is not expected to use ports in these port range for its purpose.  Parameters:
• Min source Port  : Default - 63000
• Max Source Port :  2^16 - 1
• Destination port:  Default 2152 (In case somebody wants to give some other port number during debugging time).
• GetUDPSourcePortRange():  To read the port range that was set using SetUDPSourcePortRange() function. 
• SetHousekeepingTimeOuts():  Parameters include
• Amount of time to wait after forwarding (for handover purposes) tunnels are created and before it gets indication to transfer the forwarded packets to the PDCP layer.  Valid at target eNB.
• Statistics:  There are no specific statistics defined by standards. I suggest to define the statistics for debugging.

Tunnel Establishment:

Tunnels are established by S1 and X2 protocols running in control plane. There are as many GTP tunnels as number of Radio Access Bearers.  It is good to visit the handover procedures before defining the interface for tunnels establishment as well as packet level API with PDCP.

Even though there are different types of handovers,  from GTP-U perspective they are same.  There are two types of handovers -  X2 based handovers and S1-based handovers.  if both source eNB and target eNB belongs to the same MME, then X2 based handover is used.  If there are different MMEs, then S1 handover would be used.  For more details about the handover, please see 3GPP TS 36.300 (Overall architecture) and 23.401.

Handover is applied for each RAB independently.   Typically only RABs that are setup using acknowledged mode do lossless handover. Unacknowledged mode RABs only do seemless handover. Control plane knows which RABs, in turn, which GTP-tunnels require handover and accordingly programs them in the user plane software in both PDCP and GTP-U layers. 

As part of lossless handover source eNB is expected to send unacknowledged downlink packets waiting at the  RLC layer and  new packets that are coming in from SGW.  Ofcourse, first GTP-U needs to send the unacknowledged PDCP packets and then the packets that come from the SGW.  In case of seemless handover only the new packets that are coming in from SGW are transferred to the target eNB.

Since PDCP packets already have sequence number associated with them by the PDCP layer, it is necessary that PDCP SN is sent along with the packet to the target eNB which is expected to send to the UE.   SGW sends the END-MARKET GTP packet to indicating the SGW will not be sending the packet to source eNB.  Target eNB is expected to send the packets with the PDCP SN first to the UE, then the packets which were sent by source eNB which don't have sequence number and then only it should start sending the packets which came from SGW to the target eNB.  Note that X2 or S1 protocol (based on whether X2 or S1 handover) create GTP tunnels as part 'Handover Request Ack'. It is expected that source eNB receives the GTP tunnel ID for sending the pending downlink packets. 

GTP-U also is used to send the uplink packets that were received out of sequence by eNB in PDCP layer.  These packets are also can be sent to the target eNB.  PDCP status transfer message that is sent to the UE as part of handover execution tells the UE the sequence numbers of the packets which were received out-of-order.  UE then accordingly only has to do selective retransmissions and thereby saving air bandwidth.  In any case, source eNB sends these out-of-order received packets to the target eNB with the sequence numbers.  The tunnel it uses to send these packets is new tunnel created for this purpose. This TEID of the tunnel is also sent along with Handover request Ack by the target eNB.

Normally each tunnel has both self TEID and peer TEID.  But in case of tunnels that are setup for forwarding the packets  have peer TEID with respect to source eNB and self TEID with respect to target eNB.

• GTPUCreateTunnel() :  This function is used to create normal tunnels as well as forwarding tunnels during handover.  Control plane creates as many tunnels as number of DRBs (Dedicated Resource Bearers).  Parameters include:
• CP Reference Identification:  Control plane can put its reference which is expected to be sent along with the indications.
• Virtual Instance ID : in case of multiple operators sharing the same eNB hardware.
• Identification parameters in uplink direction:
• Sector ID,  C-RNTI and Logical Channel ID.
• Tunnel Type :  Normal,  Forwarding-downlink-sourceNBside,  forwarding-downlink-targetNBside,  forwarding-uplink-sourceeNB and forwarding-uplink-targeteNB.
• Normal :  Indicates normal tunnel established between eNB and SGW.
• Forwarding-downlink-sourceeNB :  Indicates that this tunnel is being created at source eNB for forwarding pending downlink packets. GTP-U in source eNB would start expecting the PDCP layer to give all downlink packets which were not acknowledged followed by local end-marker  along with the last packet.  GTP-U knows from the end-marker that no more packets should be expected from the local PDCP layer. Even if there are no packets are to be forwarded,  GTP-U expects the PDCP layer to indicate end-marker.
• Forwarding-uplink-sourceeNB:  Indicates that this tunnel is being created at source eNB for forwarding the out-of-order uplink packets.  GTP-U in source enB would start expecting the PDCP layer to give all uplink packes that came in out-of-order followed by local end-marker along with the last packet.  Even if there are no packet to be forwarded,  GTP-U expects PDCP layer to give end-marker. 
• Forwarding-downlink-targeteNB: Indicates that this tunnel is being creates at target eNB for receiving the forwarded downlink packets from source eNB.
• Forwarding-uplink-targeteNB:  Indicates that this tunnel is being creates at target eNB for receiving the out-of-sequence uplink packets at the sourceeNB.
• Identification parameters in downlink direction:
• TEID Self (Only if tunnel type is normal,  forwarding-downlink-targeteNB and forwarding-uplink-targeteNB)
• local IP address (Could be IPv4 or IPv6)
• Peer IP address (Could be IPv4 or IPv6)
•  TEID Peer: Required to send the packets to the peer. Only required if tunnel type is Normal,  forwarding-downlink-sourceeNB and forwarding-uplink-sourceeNB.
• DSCP Marking:  Different RABs are created due to different QoS treatment required.  QoS treatment should be not only limited to MAC scheduler enforcement, but also to be enforced on uplink traffic before sending the packets out onto the backhaul network.  Normally the DSCP value of the IP packets coming from the UE would have right value. In case, they don't send the right value, then it can be set here.  QoS shaping module at the egress level (towards backhaul network) would use the DSCP value of the IP packet to provide differential treatment to the packets. In case of IPsec layer between GTP and QoS,  IPsec is normally configured to copy the DSCP value from inner IP header to IPsec outer IP header there by marked DSCP value is preserved all the way to the QoS module. 
• Active flag: Indicates whether this tunnel is active.  This flag is required due to 'Multiple Preparation' feature.  Source eNB may initiate handover procedure with multiple target eNBs, but only one will be activated eventually.
•  GTPUTargeteNBStartXmitOfForwardedPkts():  This functions will be used by control plane to initiate the transmission of forwarded packets. This function is called on target eNB.  When control plane (RRC) receives the 'RRCReconfigurationComplete' message from the UE by target eNB,  it is expected that this function is called  after the control plane programs the PDCP layer with the information it receives as part of SN-STATUS_TRANSFER.  GTP-U on targeteNB is expected to keep the forwarded packets  until this function is called. It is also expected that GTP-U layer of target eNB sends the packets with sequence number first before sending the forwarded non-seq packts and then the new packets which are coming to target eNB directly from SGW.   Note that when this function is called GTP-U layer of target eNB can send both DL and UL packets to the PDCP layer.  I guess UL packets would be used by PDCP layer for sending status transfer PDCP control message to the UE.  These packets will be eventually would be given back to the GTP layer in order to the GTP tunnel which is created with new SGW (Normal tunnel).  
• Parameters include:
• Identification parameters such as Virtual Instance ID, Sector ID,  C-RNTI, LCI and tunnel type.
• I think that there is no special function required in source eNB from control plane  to initiate handover.  Tunnel creation with appropriate tunnel-type itself is the indication that it can forward the packets. But PDCP user plane layer need to give the packets to the GTP-U layer. 
• GTPUChangeActiveStatus():  This function can be used by control plane to activate or deactivate an particular tunnel. This is required due to 'Multiple preparation' feature.  
• GTPU Do Hairpin Transfer:  This function is called by eNodeB control plane on original S1 tunnel to do the hairpin. This is expected to be done as part of processing HandoverComplete message from target eNodeB.  Once it is done,  GTP-U layer forwarding downlink packets over GTP tunnel established to the target eNB.
•  GTPUTerminateTunnel():  This function terminates the tunnel in the GTP-U.  Control plane (S1 and X2) call this function when it finds that this tunnel is no longer required. Parameters:
• Identification parameters such as Virtual Instance ID, Sector ID,  C-RNTI, LCI and tunnel type.
• GTPUGetTunnelInfo():  This function can be used by control plane to get the information and statistics of a given tunnel.  Parameters:
• Identification parameters such as Virtual Instance ID, Sector ID,  C-RNTI, LCI and tunnel type
• Output:  Statistics counters maintained on per tunnel basis and any state information.
• GTPUGetFirstTunnelInfo() and GTPUGetNextTunnelInfo():  Meant for walking through the tunnels.  Mainly for debugging and SNMP.
• xxxGTPUErrorIndication():  This function is expected to be called by GTP-U to indicate the peer provided error to the control plane (Error indication message received by the GTP-U).
• xxxGTPUPathErrorInd():  This function is expected to be called by GTP-U when it determines the peer is not alive.  As you might have seen, I did not define any function for control plane to initiate the path liveness checks.  Whenever the normal tunnel is created,  GTP-U is expected to create path management context to check the liveness of peer.  It is possible that there are more tunnels to the same peer. In that case, GTP-U should not be creating duplicate path management contexts.  When there are no tunnels to the peer,  path management context can be removed as there is no need to do liveness checks.  One way to do this in implementation is to have 'counter' (atomic) indicating the number of tunnels that have same peer.  This gets incremented whenever new tunnel is established for which path management context already exists.  It gets decremented when the tunnel is terminated.  When the count becomes 0, GTP-U can remove the context.

 Interfacing with PDCP:


GTP-U interfaces with PDCP to

• to send packets to the UE in downlink direction. 
• to get hold of packets from the PDCP in uplink direction 
• to get hold of packets which were not acknowledged in downlink direction. This is required in handover cases.  These packets would be sent by GTP-U to the target eNB.
• to get hold of packets which were received by PDCP from UE (uplink) out-of-order. This is required in handover cases. These packets would be sent by GTP-U to the target eNB.

   

• GTPUSendNormalPktToPDCP():  This function is used to send packets to the UE . Parameters may include
• Identification parameters such as Virtual Instance ID,  Sector ID, C-RNTI, LCI.
• packet in implementation specific buffer
• GTPUSendHOSeqPktToPDCP():  This function is used by GTP-U in target eNodeB. It is used to send the packets  with the sequence number. Parameters include:
• Identification parameters such as Virtual Instance ID,  Sector ID, C-RNTI, LCI.
• Direction of packet (Uplink or downlink) : Note that as part of handover both downlink and uplink packets are sent by source eNB.  Downlink packets are meant to be sent to the UE by PDCP layer.  Uplink packets are expected to be used by PDCP to reorder the packets with newly arrived uplink packets. Note that these uplink packets would be given to the GTP-U layer in sequence. Also note that these packets would be given to normal GTP-U tunnel which was established between target eNB and SGW.
• bValid PDCP SeqNum : Indicates whether the PDCP seq number is valid.
  
• PDCP seq number:   Sequence number is valid in handover scenarios.  This is normally done by the target eNB.  GTP-U implementation gets this from the extension header "PDCP PDU Number".
  
• packet in implementation specific buffer.
• Notes:  GTPUSendNormalPktToPDCP() and GTPUSendHOSeqPktToPDCP() could have been combined into one function. It is good to have two different functions for performance reasons. If combined both,  then number of arguments and preparation of arguments would take some CPU cycles which can be saved by having two different functions.
• GTPURecvNormalPktFromPDCP() :  This function is called by PDCP or some other glue layer to give packets to the GTP-U layer.  Parameters include:
• Identification parameters:  Virtual Instance ID, sector ID, C-RNTI, LCI.
• Note that there could be multiple GTP-U tunnel matching with these parameters. There would be only one tunnel to the SGW, but all other tunnels are X2 tunnels.  Since this function is called to send normal packets, it is expected that this information (normal) is used to select the right tunnel.
• Packet in implementation specific buffer.
• GTPURecvHOPktFromPDCP() :  This function is called by PDCP or some glue layer to give HO packets to the GTP-U layer.  This function is used by PDCP to give both UPLINK and DOWNLINK pending packets.   This function is called by source eNB.  Parameters include:
• Identification parameters:  
• Virtual Instance ID, Sector ID, C-RNTI, LCI
• Type of packet:  Downlink or uplink.
• Sequence number (PDCP sequence number): This will be sent to the peer using PDCP PDU number extension header.
• Some notes:  Due to 'Multiple Preparation' feature, there could be multiple forwarding tunnels in source eNB. But fortunately only one will be active.  It is expected that this 'active' information is used to match the right tunnel.

Some implementation notes:

• There would be two contexts that would need to be maintained in GTP-U - Tunnel contexts and path management contexts.   Tunnel contexts need to be arranged in two different hash tables - One for uplink and another for downlink.  
• Key parameters to the uplink hash table would be:
• Virtual instance ID,  Sector ID, C-RNTI, LCI
• Type (Normal, Handover)
• In case of handover,  Uplink or downlink,  Active/Inactive.
• Key parameters to the downlink hash table would be:
• IP type (IPv4, IPv6)
• Source IP and Destination IP from GTP-U packet.
• TEID from the GTP-U packet
• Some notes: It may be good to have two different hash tables - one for IPv4 and another for IPv6 for performance reasons.  Some thing to think about.

Performance considerations:

• Use RCUs during packet processing. Absolutely no locks during packet processing. Locks may be okay during tunnel establishment and termination.  That is why, RCUs are the best in multicore environments.  
• Multicore processors provide statistics accelerators. Use this functionality to increment/decrement the global counters and counters that may be updated across multiple cores.

Above scribbling is really my notes that was prepared while going through the GTP-U specifications.  I am new to 3GPP and it took me a solid one week to get handle. I hope it would be useful for you and hopefully jump start your study of GTP-U.