NVLink is a wire-based serial, multi-lane, near-range communications link developed by Nvidia for connecting processors in high-performance computing systems. Unlike PCI Express, a device can consist of multiple NVLinks, and devices can use mesh networking to communicate instead of a central hub or switch. The protocol was first announced in March 2014 and uses a proprietary high-speed signaling interconnect (NVHS). It is primarily used to transfer data and control code between CPUs and GPUs, and between GPUs and GPUs, enabling high-bandwidth, low-latency communication for compute-intensive workloads such as Artificial intelligence and Machine learning.
NVLink specifies a point-to-point connection with data rates of 20, 25, and 50 Gbit/s per differential pair for versions 1.0, 2.0, and 3.0 and higher, respectively. For NVLink 1.0 and 2.0, eight differential pairs form a sub-link, and two sub-links (one for each direction) form a link. Starting with NVLink 3.0, only four differential pairs form a sub-link. For NVLink 2.0 and higher, the total data rate for a sub-link is 25 GB/s, and the total data rate for a link is 50 GB/s. Each V100 GPU supports up to six links, providing up to 300 GB/s in total bidirectional bandwidth. Announced on May 14, 2020, NVLink 3.0 increased the data rate per differential pair from 25 Gbit/s to 50 Gbit/s while decreasing the number of pairs per link from 8 to 4. With 12 links for an Ampere-based A100 GPU, this brings total bandwidth to 600 GB/s. The Hopper GPU microarchitecture, announced in March 2022, has 18 NVLink 4.0 links, enabling 900 GB/s total bandwidth. NVLink 6, used in Nvidia's Vera Rubin NVL72 platform, provides 3.6 TB/s of bidirectional bandwidth per GPU, and an NVL72 rack uses nine NVLink 6 switches to provide 260 TB/s of total scale-up bandwidth.
Topology and switching
For small numbers of GPUs, the NVLink lanes on a single device are sufficient for an all-to-all mesh connectivity. To accommodate higher GPU counts, NVLink since 2018 uses a packet-switched architecture, where a central switch can serve up to 32 two-lane ports. The NVSwitch for NVLink 4.0 can produce some simple computation of its own (e.g., sum, broadcast) to reduce the need for communication, thanks to the SHARP accelerator. This allows scaling beyond direct GPU-to-GPU connections, enabling larger systems like the DGX series and NVL72 racks.
Performance characteristics
Real-world performance is determined by applying different data transmission overhead costs, such as 128b/130b line code, link control characters, transaction headers, buffering capabilities, and DMA usage on the computer side. These physical limitations usually reduce the data rate to between 90 and 95 percent of the transfer rate. NVLink benchmarks show an achievable transfer rate of about 35.3 Gbit/s (host to device) for a 40 Gbit/s (2 sub-lanes uplink) NVLink connection towards a P100 GPU in a system driven by IBM POWER8 CPUs.
Use with plug-in boards
For various versions of plug-in boards, a small number of high-end gaming and professional graphics GPU boards expose extra connectors for joining them into an NVLink group. A similar number of slightly varying, relatively compact, PCB-based interconnection plugs exist. Typically only boards of the same type will mate together due to physical and logical design. For some setups, two identical plugs need to be applied to achieve full data rate. The typical plug is U-shaped with a fine grid edge connector on each end stroke facing away from the viewer. The width of the plug determines how far apart the plug-in cards need to be seated, with known plug widths of 3 to 5 slots depending on board type. The interconnect is often referred to as Scalable Link Interface (SLI) from 2004 for its structural design, even though modern NVLink-based designs are technically different. Reported devices include Quadro GP100 (up to 2 bridges, 160 GB/s), Quadro GV100 (up to 2 bridges, 200 GB/s), GeForce RTX 2080 and 2080 Ti with single bridges, GeForce RTX 3090 with a unique bridge, and Quadro RTX 5000, 6000, and 8000 with single bridges achieving up to 100 GB/s.
Service software and programming
For the Tesla, Quadro, and Grid product lines, the NVML-API (Nvidia Management Library API) offers functions for programmatically controlling aspects of NVLink on Windows and Linux, such as component evaluation, status/error querying, and performance monitoring. The NCCL library (Nvidia Collective Communications Library) enables developers to implement powerful Deep learning and computation-hungry applications atop NVLink. The Nvidia Control panel's "3D Settings" page and the CUDA sample application "simpleP2P" use such APIs. On Linux, the command-line application with sub-command "nvidia-smi nvlink" provides advanced information and control.
History
On April 5, 2016, Nvidia announced that NVLink would be implemented in the Pascal-microarchitecture-based GP100 GPU, as used in Nvidia Tesla P100 products. With the introduction of the DGX-1 high-performance computer, it became possible to have up to eight P100 modules in a single rack system connected to up to two host CPUs. The carrier board allows for a dedicated board for routing NVLink connections, with each P100 requiring 800 pins (400 for PCIe plus power, and another 400 for NVLink). Since then, NVLink has evolved through multiple generations, with each iteration increasing bandwidth and link count, and it remains a key component in Nvidia's Generative AI and Large language model training infrastructure.