A GPU is specialized for compute-intensive, highly parallel computations.
In order to achieve this goal, compared to a CPU, more transistors are devoted to data processing rather than data caching and flow control.
Single Instruction, Multiple Threads (SIMT)
Nvidia introduced SIMT in the Tesla GPU microarchitecture with the G80 chip.
ATI Technologies, now AMD, released a competing product slightly later on May 14, 2007 (the TeraScale 1-based "R600" GPU chip).
iGPU vs dGPU
iGPU = integrated GPU (integrated into the CPU, that is)
dGPU = dedicated (or discrete?) GPU
Libraries
NCCL
NCCL (NVIDIA Collective Communication Library, pronounced as «Nickel») provides primitives for inter-GPU communication.
See also torch.cuda.nccl
cuDNN
NVIDIA CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks and provides highly tuned implementations for standard routines, such as
- forward and backward convolution,
- attention,
- matmul,
- pooling and
- normalization.
cuBLAS
NVIDIA cuBLAS is a GPU-accelerated library for accelerating AI and HPC applications.
BLAS stands for Basic Linear Algebra Subprograms.
NPP
NPP is a library of over 5000 primitives for image and signal processing.
NPP supports
- color conversion,
- image compression,
- filtering,
- thresholding, and
- image manipulation.
NPP++ seems to be a C++ extension for NPP.
npp stands for Nvidia performance primitives.
NVML
The NVIDIA Management Library (NVML) is a C-based programmatic interface for monitoring and managing various states within Data Center GPUs.
See also the python library
pyNVML.
nvcc
nvcc is the CUDA compiler.
nvprof
See also /usr/local/cuda/bin/nvprof.
NVTX
NVIDIA Tools Extension Library is an API for annotating source code to provide contextual information to developer tools.
See also torch.cuda.nvtx.
NVIDIA Container Toolkit
The
NVIDIA Container Toolkit is a collection of libraries and utilities that allows containerized applications to interact with the host's GPUs by injecting the user mode driver libraries (e.g.
libcuda.so, required device nodes (presumably
/dev/nvidia-*?) and utilities such as
nvidia-smi into the container as it starts.
It currently includes
- Container Runtime (
nvidia-container-runtime)
- Container Toolkit CLI (
nvidia-ctk)
- CDI Hooks (
nvidia-cdi-hook)
- Container Runtime Hook (
nvidia-container-runtime-hook)
- Container CLI (
nvidia-container-cli)
- Container Library (
libnvidia-container1)
RAPIDS
The Nvidia RAPIDS data science framework is a collection of libraries which are designed to have a similar look and feel to working in Python yet with optimized Cuda primitives and high-bandwith GPU memory under the hood.
APIs
| cuDF | cuDF is a pandas like GPU DataFrame library (built on the Apache Arrow columnar memory format). |
| dask-cuDF | Dask-cuDF extends Dask where necessary to allow its DataFrame partitions to be processed using cuDF GPU DataFrames instead of Pandas DataFrames. Dask-cuDF is used to leverage multiple gpus and multiple nodes for more performance at larger scales |
| cuML | cuML is scikit-learn like suite of machine learning libraries. |
| cuGraph | Graph analytics like NetworkX |
| cuxfilter | onnector library |
| Java + cuDF | Java bindings for the cuDF library. |
| cuCIM | GPU accelerated I/O, computer vision & image processing primitives |
| cuVS | GPU-accelerated vector search and clustering. |
| KvikIO | A Python and C++ library for high performance file IO using GPUDirect Storage (GDS). |
| RAFT | Fundamental widely-used algorithms and primitives for vector search, machine learning, and information retrieval. |
| Dask-CUDA | Various utilities to improve deployment and management of Dask workers on CUDA-enabled systems. |
| RMM | RAPIDS Memory Manager, used for all device memory allocations in cuDF (C++ and Python) and other RAPIDS libraries. |
| RapidsMPF | RAPIDS Multi-Process Foundation (rapidsmpf) is a collection of multi-GPU, distributed memory algorithms written in C++ and exposed to Python. |
| UCXX | The Python interface for UCX. |
| nvForest | A lightweight RAPIDS library for fast inference for decision tree models on NVIDIA GPUs and CPUs. |
Libraries
| libcudf | A C/C++ CUDA library for implementing standard dataframe operations. |
| libcuml | A C/C++ CUDA library for cuML. |
| libkvikio | A C++ header-only library providing bindings to cuFile, which enables GPUDirect Storage (GDS). |
| librapidsmpf | A C++ collection of multi-GPU, distributed memory algorithms. |
| libucxx | An object-oriented C++ interface for UCX, with native support for Python bindings. |
| rapids-cmake | A collection of CMake modules that are useful for all CUDA RAPIDS projects. |
Docker images
Nvidia provides docker images for CUDA on the docker hub under
nvidia/cuda.
The Dockerfiles for the images are open-source and licensed under 3-clause BSD.
base | Includes the CUDA runtime (cudart, libcudart.so) |
runtime | Builds on base and includes the CUDA math libraries and NCCL. A runtime image that also includes cuDNN is available. Some images may also include TensorRT. |
devel | Builds on runtime and includes headers, static librariesx and, development tools for building CUDA images. These images are particularly useful for multi-stage builds. |