Corundum by alexforencich
GitHub repository: https://github.com/ucsdsysnet/corundum
Corundum is an open-source, high-performance FPGA-based NIC. Features include a high performance datapath (256 bit AXI), 10G Ethernet, PCI express gen 3, a custom, high performance, tightly-integrated PCIe DMA engine, many (1000+) transmit, receive, completion, and event queues, MSI interrupts, multiple interfaces, multiple ports per interface, per-port transmit scheduling including high precision TDMA, checksum offloading, and native IEEE 1588 PTP timestamping. A Linux driver is included that integrates with the Linux networking stack. Development and debugging is facilitated by an extensive simulation framwork that covers the entire system from a simulation model of the driver and PCI express interface on one side to the Ethernet interfaces on the other side.
Corundum has several unique architectural features. First, transmit, receive, completion, and event queue states are stored efficiently in block RAM or ultra RAM, enabling support for thousands of individually-controllable queues. These queues are associated with interfaces, and each interface can have multiple ports, each with its own independent scheduler. This enables extremely fine-grained control over packet transmission. Coupled with PTP time synchronization, this enables high precision TDMA.
Corundum currently supports Xilinx Ultrascale and Ultrascale Plus series devices. Desgins are included for the following FPGA boards:
- Alpha Data ADM-PCIE-9V3 (Xilinx Virtex Ultrascale Plus XCVU3P)
- Exablaze ExaNIC X10 (Xilinx Kintex Ultrascale XCKU035)
- Xilinx VCU108 (Xilinx Virtex Ultrascale XCVU095)
- Xilinx VCU118 (Xilinx Virtex Ultrascale Plus XCVU9P)
Completion queue manager module. Stores device to host queue state in block RAM or ultra RAM.
Event mux module. Enables multiple event sources to feed the same event queue.
Event queue module. Responsible for writing event queue entries into host memory.
Interface module. Contains the event queues, interface queues, and ports.
Port module. Contains the transmit and receive engines
Queue manager module. Stores host to device queue state in block RAM or ultra RAM.
Receive checksum computation module. Computes 16 bit checksum of Ethernet frame payload to aid in IP checksum offloading.
Receive engine. Manages receive descriptor dequeue and fetch via DMA, packet reception, data writeback via DMA, and completion enqueue and writeback via DMA. Handles PTP timestamps for inclusion in completion records.
TDMA bit error ratio test channel module. Controls PRBS logic in Ethernet PHY and accumulates bit errors. Can be configured to bin error counts by TDMA timeslot.
TDMA bit error ratio test module. Wrapper for a tdma_scheduler and multiple instances of tdma_ber_ch.
TDMA scheduler module. Generates TDMA timeslot index and timing signals from PTP time.
Transmit checksum computation and insertion module. Computes 16 bit checksum of frame data with specified start offset, then inserts computed checksum at the specified position.
Transmit engine. Manages receive descriptor dequeue and fetch via DMA, packet data fetch via DMA, packet transmission, and completion enqueue and writeback via DMA. Handles PTP timestamps for inclusion in completion records.
Round-robin transmit scheduler. Determines which queues from which to send packets.
Round-robin TDMA transmit scheduler. Determines which queues from which to send packets. Contains a tdma_scheduler instance to control configuration based on PTP time.
cpl_queue_manager.v : Completion queue manager event_mux.v : Event mux event_queue.v : Event queue interface.v : Interface port.v : Port queue_manager.v : Queue manager rx_checksum.v : Receive checksum offload rx_engine.v : Receive engine tdma_ber_ch.v : TDMA BER channel tdma_ber.v : TDMA BER tdma_scheduler.v : TDMA scheduler tx_checksum.v : Transmit checksum offload tx_engine.v : Transmit engine tx_scheduler_rr.v : Round robin transmit scheduler tx_scheduler_tdma_rr.v : Round robin TDMA transmit scheduler
Running the included testbenches requires MyHDL and Icarus Verilog. Make sure that myhdl.vpi is installed properly for cosimulation to work correctly. The testbenches can be run with a Python test runner like nose or py.test, or the individual test scripts can be run with python directly.
tb/axi.py : MyHDL AXI4 master and memory BFM tb/axil.py : MyHDL AXI4 lite master and memory BFM tb/axis_ep.py : MyHDL AXI Stream endpoints tb/eth_ep.py : MyHDL Ethernet frame endpoints tb/ip_ep.py : MyHDL IP frame endpoints tb/mqnic.py : MyHDL mqnic driver model tb/pcie.py : MyHDL PCI Express BFM tb/pcie_us.py : MyHDL Xilinx Ultrascale PCIe core model tb/pcie_usp.py : MyHDL Xilinx Ultrascale Plus PCIe core model tb/ptp.py : MyHDL PTP clock model tb/udp_ep.py : MyHDL UDP frame endpoints tb/xgmii_ep.py : MyHDL XGMII endpoints
Corundum internally uses the following libraries:
Copyright 2019, The Regents of the University of California. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE REGENTS OF THE UNIVERSITY OF CALIFORNIA ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OF THE UNIVERSITY OF CALIFORNIA OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. The views and conclusions contained in the software and documentation are those of the authors and should not be interpreted as representing official policies, either expressed or implied, of The Regents of the University of California.