PCIe Endpoint developer considerations

See my earlier post on "PCIe End Point Developer Techniques" on some of the considerations developer should  keep in mind.  This post will give some more expectations from Endpoint.

Interrupt Coalescing:

Endpoints will interrupt the host whenever it puts data in the receive descriptor ring(s) . Also, Endpoint interrupts the host whenever it transmits the packet out from the transmit descriptor ring(s).  Interrupt coalescing is normally used to reduce the number of interruptions to the host.  Interrupt coalescing configuration  by host on Endpoint will result in Endpoint interrupting the host only after filling up certain configured number of descriptors or after some configurable amount of time is passed from previous invocation of interrupt.  Time parameter is required to ensure that host gets the interrupt even if configured number of descriptors are not filled up. This parameter should be chosen carefully to reduce the latency of packets.  Interrupt coalescing also can be configured by host to reduce the number of interrupts when descriptors are removed from the descriptor rings.

As an Endpoint developer, always make sure to provide interrupt coalescing parameter configuration to host for every interrupt that gets generated by the Endpoint.

Now the processors are used in Endpoints,  similar functionality is required in reverse direction.  End Point should be able to indicate the interrupt numbers it would assign to the host. In addition, end point can also have interrupt coalescing parameters  in configuration registers for each interrupt.  Host will read them and use this information to reduce the number of interrupts it invokes on the Endpoint. Since these are registers,  host also can put different configuration values.

As an Endpoint developer, make sure to provide this configuration to the host. If you are developing the host driver,  please ensure to use interrupt coalescing functionality.

Multiple Receive Descriptor Rings and Multiple Transmit Descriptor rings:

 In case of intelligent NIC cards, multiple receive descriptor rings are used to pass different kinds of traffic.  For example,  packets related to configuration & management is given highest priority compared to other data traffic. Within data traffic, voice & video traffic would be given higher priority normally.  Different descriptor rings are used to pass different priority traffic.  Endpoint, upon receiving the packets from wire, classifies the packets and puts them in appropriate rings.   Host driver is expected to select the descriptor ring to dequeue the packet.  Basically, scheduling of rings is used to select the ring.  To enable scheduling, rings are given weightage. Some important rings may also be placed in strict priority.   Scheduler in the host is expected to select the strict priority rings if they have packets. If no packets in strict priority rings, then  a ring is selected from weighted ring set for dequeuing the packet.

Similarly,  multiple transmit descriptor rings are used by the host for multiple reasons. One reason is similar to the reason described above for receive descriptor rings. That is,  multiple descriptor rings might be used to send different priority traffic.   Endpoint software is expected to do scheduling across multiple rings to select the ring and then dequeue the packet for transmission on the wire.  This scheduling also could be similar to the one described above.

Set of descriptor rings meant for different kinds of traffic can be termed as 'Descriptor Ring Group".

As Endpoint developer,  one should provide this kind of flexibility for host to process high priority traffic first over low priority traffic.

Multiple Descriptor Ring Groups:


In Multicore processor environments, avoiding locks is very important for achieving the performance. Endpoint is expected to provide facilities such a way that host never need to do any locking while dequeuing the packets from receive descriptor rings or while en-queuing the packets onto the transmit descriptor rings. That is where,  multiple descriptor groups are required.  If the host has 4 cores, then at least 4 receive and transmit groups are needed to avoid  lock by assigning each group to different core.  To also ensure that right core is woken up upon interrupt,  it is also necessary that each group has its own interrupt which is affined to the appropriate core.  That is, when a core is interrupted, it exactly knowns which group of descriptor rings to look at. Since a group is accessed by only one core, there is no lock required.

Now that  Endpoints are also being implemented on processors, it also becomes important to reduce the locks even on the endpoints.   This makes the problem little more complicated.  Let us assume that host has 4 core processor and endpoint has 8 core processor.  As discussed above, since host has 4 cores, 4 groups are good enough.   That is, 8 cores in the endpoint would be updating only 4 groups.  Since more than one core would be manipulating the group, lock would be required in the endpoint.  To avoid lock in both places, it is required that the number of groups need to be at least maximum of number of cores in host and endpoint.  Descriptor groups would be equally divided across multiple cores on each side.  Even though this formula avoids the lock in both places,  but the traffic distribution can be a problem if the number of groups can't be divided across cores equally. This will lead to some cores getting utilized more than other cores.  If you take an example where the host has 4 cores and endpoint has 7 cores,  then to make the distribution of groups across cores equal on each side,  one should have GCF of 4 and 7. That is, 28 groups are required.  On host side, each core would need to be assigned with 7 groups and on endpoint side, each core is assigned with four groups.

If one core has more than one group, then there could be two challenges.  One is interrupt assignment and other is distribution across groups.   Even though PCIe support MSI and MSI-X, there could always be some practical limitations on the number of interrupts that can be used.  Hence each group can't have its own interrupt.  Since interrupts are used to interrupt the core,   one interrupt per core is good enough.

Producer core is expected to distribute the traffic equally among the groups - Round Robin on packet basis would be one acceptable method. Once the group is chosen,  based on the classification criteria as described in the previous section a descriptor ring  in the group gets selected to place the packet. Consumer core is expected to do similar RR mechanism to select the group to dequeue from.  Once the group is selected, another scheduler as described under the previous section would be used to select the descriptor ring to dequeue the packet from.

As an Endpoint developer,  one should ensure to support group concept to avoid locks.

Command Descriptor rings:


So far we have talked about packet transmit and receive descriptor rings and groups.  Command rings are also required to pass commands to the Endpoint and for end point to respond back.  There are many cases where command rings are required.  For example,  Ethernet endpoint might be required to know information from host such as

• MAC addresses
• Multicast Addresses
• Local IP addresses
• PHY configuration.
• Any other offload specific information.

Unlike packet descriptor rings, here response is expected.  Hence the descriptor should have facility for host to provide both command and response buffers.  Endpoint is expected to act on the command and put the response in the response buffer.

As an endpoint developer,  one should provide facility to send commands and receive responses.

PCIe End Point Developer techniques

Some UTM functions such as Intrusion Prevention and Anti-Malware take significant CPU time. Equipment vendors are looking for ways to offload some processing from the main CPU to NIC cards.  That is how Intelligent NIC cards have born.  Some equipment vendors created their own ASICs or FPGAs to do some offload functions at ingress time and egress time.  Increasingly vendors are showing interest in using Multicore processors in offload cards. Multicore processors approach  have many advantages over ASIC/FPGA approach as typical programming languages can be used to create even offload engines - Faster Time-to-Market,  Image can be upgraded in the field and simple to maintain.  Offload card are attached to main CPU over PCIe and they in essence are replacing  traditional PCIe NIC cards in security equipment devices.

With PCIe cards  increasing becoming processor based,  network programmers are now can program the offload cards using traditional programming languages such as 'C'.  Intelligent NIC cards have Ethernet port attached to it for external connectivity.  Towards CPU, it communicates via PCIe interface. These cards, upon receiving the packets from the Ethernet port, do some operations and send the result packets to the CPU via PCIe interface.  When the packets are given to the card via PCIe, it can do some operations before sending them out on Ethernet port. Basically,  PCIe card as a whole appears as one NIC interface to the host even though it does some offload functions.  In this post, I will not cover typical offload functions that can be done in the offload cards. I will try to do that in my next posts.  In this post, I like to concentrate on the efficient packet transfer over PCIe.

Some brief introduction on PCIe:

Intelligent NIC cards use PCIe End Point functionality to talk to the host CPU.  Host CPU uses the PCI in RC (Root complex) mode.   As you all know,  PCIe devices are arranged in a tree manner.  Root of the tree is RC (host).  Leaves of the trees are  PCIe devices (End points).  In between the leaves,   there can be PCIe switches.   Host uses services provided by the end points.  PCIe switches are used to extend the number of end point devices to the host.  Example of end point devices are:  Ethernet controllers,  SCSI devices, Video devices etc..

Every end point and root complex processors  have PCIe controller hardware block to talk to the PCIe bus.  End points can only talk to the RC, that is, End points can't talk to each other. If they need to talk to each other, RC will act as a relay.  RC can talk to many end points at any time.  In a PCIe tree,  only one RC is possible.  In typical PC system,  x86 is RC and all other PCIe devices are End Points.

RC and end points share required memory mapped IO or memory itself with the other party.  End points indicate the regions of memory or memory mapped IO to the RC via configuration space BAR registers.  Typically,  end point have around 5 BARs.  Each BAR represents in some form both memory address and the size.  Host would map these memory locations in its system address space. In case of x86 hosts, BIOS does the PCIe devices enumeration. As part of enumeration, it maps the device addresses in its space.   Hosts typically keep aside certain system address space to map memory exposed by PCIe devices.  Many host processors today are at the minimum have 36 bit address lines - they can address up to 64 G bytes of address space.  Some address space goes for the DDR and some space in the rest of address space is reserved to map PCIe devices.   Hosts also normally would like to expose its memory to PCIe devices.  There is no standard defined on how the hosts would communicate to the end point  the memory spaces it wants to share.  End Points typically assign some part of DDR memory as the command and response memory.  This memory would be part of the memory space it exposes to the RC host via one of the BAR registers.  One of the commands that EndPoint exposes could be for host drivers to share its memory with the EndPoint.  End point then maps the RC exposed memory in its reserved system address space (End point processor's PCI address space).

PCIe controllers support multiple transactions types  - Most important ones are Memory read,  Memory write,  Configuration Read and Configuration Write.  There are others such as 'Messages',  Memory read and write with Lock. There are some more transaction types related to PCIe internal layers (transaction layer) such as Completion etc..

When the processors (either host or End Point) read or write data from/into the PCIe address space,  PCIe controller gets hold of this operation and creates the PCIe read/write transaction  on to the other side (destination) with the other side's memory address. PCIe transaction message contains the other side address, transaction type and data to be written, if it is write operation.  In case of read operation, there would be one more message that comes from other side with the data.   All this happens automatically.  As far as the processor is concerned, it thinks that it is reading and writing into some address (in this case PCIe address) same way as it read and writes from normal memory.  Internally PCIe transaction would be created.  Normally PCIe controllers have some thing called 'Address Translation Registers'.  Corresponding to each PCIe address space range,  original memory address of other side is stored.  This is normally done whenever other side gives the memory ranges to be mapped.  That is how PCIe controller knows the memory address to be put in the read/write transactions. 

PCIe defines two transaction models called 'Posted' and 'Non Posted'.  Posted transactions are the ones where the transactions are fired and forgotten.  That is, processor does not care whether the transaction was completed.  'Write' transactions are typically posted transactions.   'Read transactions' are by default non-posted as it needs to get the data from the other side's memory address.  Since it is synchronous operation, process simply waits for the result.  Hence the non-posted transactions are expensive.  As I understand read transaction typically even in PCIe Gen2 hardware takes around 1 to 2 Microseconds.  Hence PCIe read transaction for less than 16 bytes should be avoided.  For big size size, it is okay to use PCIe read transactions though.

So, any programming interface exposed by the PCIe card must ensure that the host processor does not do any  PCIe read transactions for data movement (such as packet transfer).   

Descriptor rings:

This post gives one technique whereby both host and End point avoid PCIe read transactions.  Descriptor ring approach is used to send/receive packets and commands between hosts and end points.  Normally PCI devices design the descriptor rings such that hosts never need to do PCIe read transaction either  to get hold of packets or send packets.  As PCIe devices also being implemented on processors for doing flexible offload mechanisms, it is necessary that PCIe read transactions should be avoided even in End Point implementations.

Descriptor rings consists of set of descriptors. Each descriptor contains some control & status bits and pointers to buffers holding the packets or commands/results.

For every descriptor ring, there is one consumer and producer.  In case of Ethernet based PCIe device,  on transmit side,  host is producer and Ethernet controller end point is consumer.  For received packets,  Ethernet controller is producer and host is consumer.  That is,  at least two descriptor rings are used  in Ethernet Controller end point -  Transmit descriptor ring and Receive descriptor ring.  In addition to these two rings,  there may be command rings too to send command and receive responses from the End point.

Extending the Ethernet example,  Ethernet driver in the host typically initializes both packet transfer (receive and transmit)  rings.  Host Ethernet driver normally initializes the descriptors in the descriptor ring with buffers to receive packets.  Ethernet Controller when it receives the packet from the wire puts it in the buffers in the receiver descriptor ring and invokes the interrupt on the host.  On transmit side too,  host puts the buffers of the packets in the descriptors and informs the end point.  In case of processor based end points,  even interrupt on that processor would be invoked by host.

Descriptor ring consists of descriptors in a ring with Producer Index and Consumer Index.  Producer Index (PI) points to the descriptors that can be written into and Consumer Index (CI) points to the ring from where consumer is expected to read descriptors.  Since there should be a way to find out whether the ring is full or empty, normally one descriptor is left empty.  That is, when writing to descriptor pointed by PI makes the PI equal to CI, then that descriptor is not used by producer until CI moves up.

To avoid PCIe read transactions, descriptor ring buffer has to be part of consumer processors' memory space.  PI and CI indexes would be there in both spaces, but the consumer space would be exposed to the producer.  Producer PI and CI need not be exposed to the consumer.  Here is the flow :

Initialization time:

• Consumer Descriptor ring is part of the memory it exposes to the producer.
• Consumer Descriptor rings PI and CI are also part of the memory it exposes to the producer.
• Initially, both PI  and CI are set to 0. 

Producer :

Producer has got the buffer to write into.

If  (PI + 1 )%ring Size != CI ) 

{

• Write the buffer in the descriptor pointed by the PI.
• Write the status and control bits accordingly.
• Increment the local PI:  PI = (PI + 1) %ring size;
• Write the new PI into other side PI..
• Generate Interrupt (MSI interrupt is also write transaction) - ofcourse, it needs to honor coalescing criteria.

}

Consumer flow would be:

if ( PI !=  CI )

{

• Read from the descriptor that is indexed by local CI and process
• Update local CI by (CI + 1 ) % ring size;
• Write  the new CI into other side CI, that is update Producer's CI value   
• Generate interrupt (if needed) to indicate producer that the descriptor is consumed.

}

If you see above, producer or consumer never do any read operation on other side memory. Any read you see above is on the local memory.  

If you take Ethernet Controller is an example, then the receive descriptor ring is defined in the host memory and transmit descriptor ring is defined in the endpoint memory.

Note that the packet buffers are always expected to be part of the host memory to ensure that no changes are required in host TCP/IP stack or host applications.  Due to this, endpoint would issue PCI read transaction on the packet buffer using address found in the descriptor. To reduce the read latency across multiple packets, it is good to get hold of multiple buffer points from multiple descriptors and initialize the multiple DMA controllers (if the device has them).  Since the packet content is expected to be in terms of hundreds of bytes and multiple packets can be read at the same time (if multiple DMA controllers are available),   read latency would be amortized to small value on per packet basis.  Read transaction on transmit packets in case of  Ethernet controllers is unavoidable,   but it is necessary that the descriptors reads should never result into PCIe read transaction.

Another point to note is by the time  interrupt  reaches the consumer, it is necessary that the consumers PI is updated.  Otherwise, there could be race condition where interrupt goes first and nothing was found to be read.  If PI was updated after the interrupt is seen by the peer, some filled descriptors will never be read until another interrupt is generated due to new descriptor being updated by producer.  That race condition is very dangerous as next descriptor update might happen after long time.  Since PCIe MSI interrupt is write transaction, there will not be any race condition as long as PI update is issued before invoking the interrupt.