CUDA

CUDA
Original authorsIan Buck
John Nickolls
DeveloperNvidia
ReleaseFebruary 16, 2007; 19 years ago (2007-02-16)
Stable release
13.4.1Edit this on Wikidata / 9 September 2026; 2 months ago (9 September 2026)
Written inC
Operating systemWindows, Linux
PlatformSupported GPUs
TypeGPGPU
LicenseProprietary
Websitedeveloper.nvidia.com/cuda-zone Edit this at Wikidata

CUDA (Compute Unified Device Architecture) is a proprietary parallel computing platform and application programming interface (API) developed by Nvidia that allows software to use certain types of graphics processing units (GPUs) for accelerated general-purpose processing, significantly broadening their utility in artificial intelligence, scientific and high-performance computing. CUDA was created in 2004 and was officially released in 2007. When introduced, the name was an acronym for Compute Unified Device Architecture, but Nvidia later dropped the initial meaning of the acronym and now rarely expands it.

CUDA is both a software layer that manages data, giving direct access to the GPU and CPU as necessary, and a library of APIs that enable parallel computation. In addition to drivers and runtime kernels, the CUDA platform includes compilers, libraries and developer tools to help programmers.

CUDA is written in the C programming language, but is designed to work with other programming languages including C++, Fortran, Python and Julia. This accessibility makes it easier for specialists in parallel programming to use GPU resources, in contrast to prior APIs like Direct3D and OpenGL, which require advanced skills in graphics programming. CUDA-powered GPUs support programming frameworks such as OpenMP, OpenACC and OpenCL.

Graphics processing unit

The graphics processing unit (GPU) is a specialized processor. It addresses the demands of real-time high-resolution, compute-intensive tasks such as 3D graphics. By 2012, GPUs had evolved into highly parallel multi-core systems allowing efficient manipulation of large blocks of data. This design is more effective than general-purpose central processing unit (CPUs) for algorithms in situations where processing large blocks of data is done in parallel, such as:

History

CUDA traces to the early 2000s, when Ian Buck, a computer science PhD student at Stanford University, began experimenting with using GPUs for purposes beyond rendering graphics. Buck had become interested in GPUs during his undergraduate studies at Princeton University, initially through video gaming. After graduation, he interned at Nvidia, gaining deeper exposure to GPU architecture. At Stanford, he built an 8K gaming rig using 32 GeForce graphics cards, originally to push the limits of graphics performance in games like Quake and Doom. However, his interests shifted toward exploring the potential of GPUs for general-purpose parallel computing.

To that end, Buck developed Brook, a programming language designed to enable general-purpose computing on GPUs. His work attracted support from Nvidia and the Defense Advanced Research Projects Agency (DARPA). In 2004, Nvidia hired Buck and paired him with John Nickolls, then director of architecture for GPU computing. Together, they began transforming Brook into CUDA. CUDA was officially released in 2007.

CUDA became central to the company's strategy of positioning GPUs as versatile hardware for scientific applications. By 2015, CUDA's development increasingly focused on accelerating machine learning and artificial neural network workloads.

Ontology

The following table offers an approximate summary of the CUDA ontology.

Ontology of CUDA framework
memory
(hardware)
memory (code, or variable scoping) computation
(hardware)
computation
(code syntax)
computation
(code semantics)
RAM non-CUDA variables host program one routine call
VRAM,
GPU L2 cache
global, const, texture device grid simultaneous call of the same subroutine on many processors
GPU L1 cache local, shared SM ("streaming multiprocessor") block individual subroutine call
warp = 32 threads SIMD instructions
GPU L0 cache,
register
thread (aka. "SP", "streaming processor", "cuda core", but these names are now deprecated) analogous to individual scalar ops within a vector op

Programming abilities

Example of CUDA processing flow
  1. Copy data from main memory to GPU memory
  2. CPU initiates the GPU compute kernel
  3. GPU's CUDA cores execute the kernel in parallel
  4. Copy the resulting data from GPU memory to main memory

The CUDA platform is accessible to software developers through CUDA-accelerated libraries, compiler directives such as OpenACC, and extensions to industry-standard programming languages including C, C++, Fortran and Python. C/C++ programmers can use 'CUDA C/C++', compiled to PTX with nvcc (Nvidia's LLVM-based C/C++ compiler) or by clang itself. Fortran programmers can use 'CUDA Fortran', compiled with the PGI CUDA Fortran compiler from The Portland Group.[needs update] Python programmers can use the cuPyNumeric library to accelerate applications on Nvidia GPUs.

In addition to libraries, compiler directives, CUDA C/C++ and CUDA Fortran, the CUDA platform supports other computational interfaces, including the Khronos Group's OpenCL, Microsoft's DirectCompute, OpenGL Compute Shader and C++ AMP. Third party wrappers are also available for Python, Perl, Fortran, Java, Ruby, Lua, Common Lisp, Haskell, R, MATLAB, IDL, Julia, and native support in Mathematica.

In the computer game industry, GPUs are used for graphics rendering, and for game physics calculations (physical effects such as debris, smoke, fire, fluids); examples include PhysX and Bullet. CUDA has also been used to accelerate non-graphical applications in computational biology, cryptography and other fields by an order of magnitude or more.

CUDA provides both a low level API (CUDA Driver API, non single-source) and a higher level API (CUDA Runtime API, single-source). The initial CUDA SDK was made public on 15 February 2007, for Microsoft Windows and Linux. Mac OS X support was later added in version 2.0, which supersedes the beta released February 14, 2008. CUDA works with all Nvidia GPUs from the G8x series onwards, including GeForce, Quadro and the Tesla line. CUDA is compatible with most standard operating systems.

CUDA 8.0 comes with the following libraries (for compilation & runtime, in alphabetical order):

CUDA 8.0 comes with these other software components:

CUDA 9.0–9.2 comes with these other components:

CUDA 10 comes with these other components:

CUDA 11.0–11.8 comes with these other components:

Advantages

CUDA has several advantages over traditional general-purpose computation on GPUs (GPGPU) using graphics APIs:

Limitations

Example

This example code in C++ loads a texture from an image into an array on the GPU:

void foo()
{
    cudaArray* cu_array = nullptr;

    // Allocate array
    cudaChannelFormatDesc description = cudaCreateChannelDesc<float>();
    cudaMallocArray(&cu_array, &description, width, height);

    // Copy image data to array
    cudaMemcpy2DToArray(cu_array, 0, 0,
                        image, width * sizeof(float),
                        width * sizeof(float), height,
                        cudaMemcpyHostToDevice);

    // Resource description (what the texture points to)
    cudaResourceDesc resDesc = {};
    resDesc.resType           = cudaResourceTypeArray;
    resDesc.res{.array{.array = cu_array}};

    // Texture description (sampling parameters)
    cudaTextureDesc texDesc = {};
    texDesc.addressMode      = {cudaAddressModeClamp, cudaAddressModeClamp};
    texDesc.filterMode       = cudaFilterModePoint;
    texDesc.readMode         = cudaReadModeElementType;
    texDesc.normalizedCoords = 0; // use unnormalized integer coordinates

    // Create the texture object
    cudaTextureObject_t texObj = 0;
    cudaCreateTextureObject(&texObj, &resDesc, &texDesc, nullptr);

    // Run kernel - pass the texture object as a parameter
    dim3 blockDim(16, 16, 1);
    dim3 gridDim((width  + blockDim.x - 1) / blockDim.x,
                 (height + blockDim.y - 1) / blockDim.y, 1);
    kernel<<<gridDim, blockDim>>>(texObj, d_data, height, width);

    // Clean up
    cudaDestroyTextureObject(texObj);
    // cudaFreeArray(cu_array); // when you are done with the array
}

__global__ void kernel(cudaTextureObject_t texObj,
                       float* odata, int height, int width)
{
    int x = blockIdx.x * blockDim.x + threadIdx.x;
    int y = blockIdx.y * blockDim.y + threadIdx.y;

    if (x < width && y < height)
    {
        float c = tex2D<float>(texObj, x, y);
        odata[y * width + x] = c;
    }
}

Below is an example given in Python that computes the product of two arrays on the GPU. The unofficial Python language bindings can be obtained from PyCUDA.

import numpy
import pycuda.autoinit

from numpy.typing import NDArray, float32
from pycuda.compiler import SourceModule
from pycuda.driver import Function, In, Out

mod: SourceModule = SourceModule(
    """
__global__ void multiply_them(float* dest, float* a, float* b) {
    const int i = threadIdx.x;
    dest[i] = a[i] * b[i];
}
"""
)

multiply_them: Function = mod.get_function("multiply_them")

a: NDArray[float32] = numpy.random.randn(400).astype(numpy.float32)
b: NDArray[float32] = numpy.random.randn(400).astype(numpy.float32)

dest: NDArray[float32] = numpy.zeros_like(a)
multiply_them(Out(dest), In(a), In(b), block=(400, 1, 1))

print(dest - a * b)

Additional Python bindings to simplify matrix multiplication operations can be found in the program pycublas.

 
import numpy

from pycublas import CUBLASMatrix

A: CUBLASMatrix = CUBLASMatrix(numpy.mat([[1, 2, 3], [4, 5, 6]], numpy.float32))
B: CUBLASMatrix = CUBLASMatrix(numpy.mat([[2, 3], [4, 5], [6, 7]], numpy.float32))
C: CUBLASMatrix = A * B
print(C.np_mat())

while CuPy directly replaces NumPy:

import cupy

from cupy.typing import NDArray, float64

a: NDArray[float64] = cupy.random.randn(400)
b: NDArray[float64] = cupy.random.randn(400)

dest: NDArray[float64] = cupy.zeros_like(a)

print(dest - a * b)

GPUs supported

Note on notation: compute capability X.Y is also written as SMXY or sm_XY (e.g. 10.3 as SM103 or sm_103) in professional Nvidia software and the code Nvidia has contributed to LLVM.

Below is a table of supported CUDA compute capabilities based on the CUDA SDK version and microarchitecture, listed by code name:

Note: CUDA SDK 10.2 is the last official release for macOS, as support will not be available for macOS in newer releases.

CUDA compute capability by version with associated GPU semiconductors and GPU card models (separated by their various application areas):

* – OEM-only products

  1. ↑ CUDA Toolkit 13.0 renamed the SM101 for Thor GPUs to SM110.

Version features and specifications

Note: A GPU with a higher compute capacity is able to execute PTX code meant for a GPU of a lower range of compute capacities. However, it is possible to compile CUDA code into a form that only works on one family (same "X") of GPUs; if existing code is compiled this way, recompilation will be needed for it to work on a newer GPU.

Feature support (unlisted features are supported for all compute capabilities) Compute capability (version)
1.0, 1.11.2, 1.32.x3.03.23.5, 3.7, 5.x, 6.x, 7.0, 7.27.58.x9.0, 10.x, 12.x
Warp vote functions (__all(), __any()) No Yes
Warp vote functions (__ballot()) No Yes
Memory fence functions (__threadfence_system())
Synchronization functions (__syncthreads_count(), __syncthreads_and(), __syncthreads_or())
Surface functions
3D grid of thread blocks
Warp shuffle functions No Yes
Unified memory programming
Funnel shift No Yes
Dynamic parallelism No Yes
Uniform Datapath No Yes
Hardware-accelerated async-copy No Yes
Hardware-accelerated split arrive/wait barrier
Warp-level support for reduction ops
L2 cache residency management
DPX instructions for accelerated dynamic programming No Yes
Distributed shared memory
Thread block cluster
Tensor memory accelerator (TMA) unit
Feature support (unlisted features are supported for all compute capabilities) 1.0, 1.11.2, 1.32.x3.03.23.5, 3.7, 5.x, 6.x, 7.0, 7.27.58.x9.0, 10.x, 12.x
Compute capability (version)

Data types

Floating-point types

Data type Supported vector types Bits Comments
Storage Length
(complete vector)
Used Length
(single value)
Sign Exponent Mantissa
E2M1 = FP4 e2m1x2 / e2m1x4 8 / 16 4 1 2 1
E2M3 = FP6 variant e2m3x2 / e2m3x4 16 / 32 6 1 2 3
E3M2 = FP6 variant e3m2x2 / e3m2x4 16 / 32 6 1 3 2
UE4M3 ue4m3 8 7 0 4 3 Used for scaling
(E2M1 only)
E4M3 = FP8 variant e4m3 / e4m3x2 / e4m3x4 8 / 16 / 32 8 1 4 3
E5M2 = FP8 variant e5m2 / e5m2x2 / e5m2x4 8 / 16 / 32 8 1 5 2 Exponent/range of FP16,
fits into 8 bits
UE8M0 ue8m0x2 16 8 0 8 0 Used for scaling
(any FP4 or FP6 or FP8 format)
FP16 f16 / f16x2 16 / 32 16 1 5 10
BF16 bf16 / bf16x2 16 / 32 16 1 8 7 Exponent/range of FP32,
fits into 16 bits
TF32 tf32 32 19 1 8 10 Exponent/range of FP32,
mantissa/precision of FP16
FP32 f32 / f32x2 32 / 64 32 1 8 23
FP64 f64 64 64 1 11 52

Version support

Data type Basic Operations Supported since
Atomic Operations Supported since
for global memory
Supported since
for shared memory
8-bit integer
signed/unsigned
loading, storing, conversion 1.0 N/a N/a
16-bit integer
signed/unsigned
general operations 1.0 atomicCAS() 3.5
32-bit integer
signed/unsigned
general operations 1.0 atomic functions 1.1 1.2
64-bit integer
signed/unsigned
general operations 1.0 atomic functions 1.2 2.0
any 128-bit trivially copyable type general operations No atomicExch, atomicCAS 9.0
16-bit floating point
FP16
addition, subtraction,
multiplication, comparison,
warp shuffle functions, conversion
5.3 half2 atomic addition 6.0
atomic addition 7.0
16-bit floating point
BF16
addition, subtraction,
multiplication, comparison,
warp shuffle functions, conversion
8.0 atomic addition 8.0
32-bit floating point general operations 1.0 atomicExch() 1.1 1.2
atomic addition 2.0
32-bit floating point float2 and float4 general operations No atomic addition 9.0
64-bit floating point general operations 1.3 atomic addition 6.0

Note: Any missing lines or empty entries do reflect some lack of information on that exact item.

Tensor cores

FMA per cycle per tensor core Supported since 7.0 7.2 7.5 Workstation 7.5 Desktop 8.0 8.6 Workstation 8.7 8.6 Desktop 8.9 Desktop 8.9 Workstation 9.0 10.0 10.1 12.0
Data Type For dense matrices For sparse matrices 1st Gen (8x/SM) 1st Gen? (8x/SM) 2nd Gen (8x/SM) 3rd Gen (4x/SM) 4th Gen (4x/SM) 5th Gen (4x/SM)
1-bit values (AND) 8.0 as
experimental
No No 4096 2048 8192 No
1-bit values (XOR) 7.5–8.9 as
experimental
No 1024 No
4-bit integers 8.0–8.9 as
experimental
256 1024 512 legacy mma.sync No
4-bit floating point FP4 (E2M1) 10.0 No 4096 tbd 512
6-bit floating point FP6 (E3M2 and E2M3) 10.0 No 2048 tbd
8-bit integers 7.2 8.0 No 128 128 512 256 1024 2048 256
8-bit floating point FP8 (E4M3 and E5M2) with FP16 accumulate 8.9 No 256
8-bit floating point FP8 (E4M3 and E5M2) with FP32 accumulate 128 128
16-bit floating point FP16 with FP16 accumulate 7.0 8.0 64 64 64 256 128 512 1024 128
16-bit floating point FP16 with FP32 accumulate 32 64 128 64
16-bit floating point BF16 with FP32 accumulate 7.5 8.0 No 64
32-bit (19 bits used) floating point TF32 speed tbd (32?) 128 32 64 256 512 32
64-bit floating point 8.0 No No 16 speed tbd 32 16 tbd

Note: Any missing lines or empty entries do reflect some lack of information on that exact item.

Tensor Core Composition 7.0 7.2, 7.5 8.0, 8.6 8.7 8.9 9.0
Dot Product Unit Width in FP16 units (in bytes) 4 (8) 8 (16) 4 (8) 16 (32)
Dot Product Units per Tensor Core 16 32
Tensor Cores per SM partition 2 1
Full throughput (Bytes/cycle) per SM partition 256 512 256 1024
FP Tensor Cores: Minimum cycles for warp-wide matrix calculation 8 4 8
FP Tensor Cores: Minimum Matrix Shape for full throughput (Bytes) 2048
INT Tensor Cores: Minimum cycles for warp-wide matrix calculation No 4
INT Tensor Cores: Minimum Matrix Shape for full throughput (Bytes) No 1024 2048 1024

FP64 Tensor Core Composition 8.0 8.6 8.7 8.9 9.0
Dot Product Unit Width in FP64 units (in bytes) 4 (32) tbd 4 (32)
Dot Product Units per Tensor Core 4 tbd 8
Tensor Cores per SM partition 1
Full throughput (Bytes/cycle) per SM partition 128 tbd 256
Minimum cycles for warp-wide matrix calculation 16 tbd
Minimum Matrix Shape for full throughput (Bytes) 2048

Technical specifications

Technical specifications Compute capability (version)
1.0 1.1 1.2 1.3 2.x 3.0 3.2 3.5 3.7 5.0 5.2 5.3 6.0 6.1 6.2 7.0 7.2 7.5 8.0 8.6 8.7 8.9 9.0 10.x 12.x
Maximum number of resident grids per device
(concurrent kernel execution, can be lower for specific devices)
1 16 4 32 16 128 32 16 128 16 128
Maximum dimensionality of grid of thread blocks 2 3
Maximum x-dimension of a grid of thread blocks 65535 231 − 1
Maximum y-, or z-dimension of a grid of thread blocks 65535
Maximum dimensionality of thread block 3
Maximum x- or y-dimension of a block 512 1024
Maximum z-dimension of a block 64
Maximum number of threads per block 512 1024
Warp size 32
Maximum number of resident blocks per multiprocessor 8 16 32 16 32 16 24 32
Maximum number of resident warps per multiprocessor 24 32 48 64 32 64 48 64 48
Maximum number of resident threads per multiprocessor 768 1024 1536 2048 1024 2048 1536 2048 1536
Number of 32-bit regular registers per multiprocessor 8 K 16 K 32 K 64 K 128 K 64 K
Number of 32-bit uniform registers per multiprocessor No 2 K

Maximum number of 32-bit registers per thread block 8 K 16 K 32 K 64 K 32 K 64 K 32 K 64 K 32 K 64 K
Maximum number of 32-bit regular registers per thread 124 63 255
Maximum number of 32-bit uniform registers per warp No 63

Amount of shared memory per multiprocessor
(out of overall shared memory + L1 cache, where applicable)
16 KiB 16 / 48 KiB (of 64 KiB) 16 / 32 / 48 KiB (of 64 KiB) 80 / 96 / 112 KiB (of 128 KiB) 64 KiB 96 KiB 64 KiB 96 KiB 64 KiB 0 / 8 / 16 / 32 / 64 / 96 KiB (of 128 KiB) 32 / 64 KiB (of 96 KiB) 0 / 8 / 16 / 32 / 64 / 100 / 132 / 164 KiB (of 192 KiB) 0 / 8 / 16 / 32 / 64 / 100 KiB (of 128 KiB) 0 / 8 / 16 / 32 / 64 / 100 / 132 / 164 KiB (of 192 KiB) 0 / 8 / 16 / 32 / 64 / 100 KiB (of 128 KiB) 0 / 8 / 16 / 32 / 64 / 100 / 132 / 164 / 196 / 228 KiB (of 256 KiB) 0 / 8 / 16 / 32 / 64 / 100 KiB (of 128 KiB)
Maximum amount of shared memory per thread block 16 KiB 48 KiB 96 KiB 48 KiB 64 KiB 163 KiB 99 KiB 163 KiB 99 KiB 227 KiB 99 KiB
Number of shared memory banks 16 32
Amount of local memory per thread 16 KiB 512 KiB
Constant memory size accessible by CUDA C/C++
(1 bank, PTX can access 11 banks, SASS can access 18 banks)
64 KiB
Cache working set per multiprocessor for constant memory 8 KiB 4 KiB 8 KiB
Cache working set per multiprocessor for texture memory 16 KiB per TPC 24 KiB per TPC 12 KiB 12 – 48 KiB 24 KiB 48 KiB 32 KiB 24 KiB 48 KiB 24 KiB 32 – 128 KiB 32 – 64 KiB 28 – 192 KiB 28 – 128 KiB 28 – 192 KiB 28 – 128 KiB 28 – 256 KiB
Maximum width for 1D texture reference bound to a CUDA
array
8192 65536 131072
Maximum width for 1D texture reference bound to linear
memory
227 228 227 228 227 228
Maximum width and number of layers for a 1D layered
texture reference
8192 × 512 16384 × 2048 32768 x 2048
Maximum width and height for 2D texture reference bound
to a CUDA array
65536 × 32768 65536 × 65535 131072 x 65536
Maximum width and height for 2D texture reference bound
to a linear memory
65000 x 65000 65536 x 65536 131072 x 65000
Maximum width and height for 2D texture reference bound
to a CUDA array supporting texture gather
N/a 16384 x 16384 32768 x 32768
Maximum width, height, and number of layers for a 2D
layered texture reference
8192 × 8192 × 512 16384 × 16384 × 2048 32768 x 32768 x 2048
Maximum width, height and depth for a 3D texture
reference bound to linear memory or a CUDA array
20483 40963 163843
Maximum width (and height) for a cubemap texture reference N/a 16384 32768
Maximum width (and height) and number of layers
for a cubemap layered texture reference
N/a 16384 × 2046 32768 × 2046
Maximum number of textures that can be bound to a
kernel
128 256
Maximum width for a 1D surface reference bound to a
CUDA array
Not
supported
65536 16384 32768
Maximum width and number of layers for a 1D layered
surface reference
65536 × 2048 16384 × 2048 32768 × 2048
Maximum width and height for a 2D surface reference
bound to a CUDA array
65536 × 32768 16384 × 65536 131072 × 65536
Maximum width, height, and number of layers for a 2D
layered surface reference
65536 × 32768 × 2048 16384 × 16384 × 2048 32768 × 32768 × 2048
Maximum width, height, and depth for a 3D surface
reference bound to a CUDA array
65536 × 32768 × 2048 4096 × 4096 × 4096 16384 × 16384 × 16384
Maximum width (and height) for a cubemap surface reference bound to a CUDA array 32768 16384 32768
Maximum width and number of layers for a cubemap
layered surface reference
32768 × 2046 16384 × 2046 32768 × 2046
Maximum number of surfaces that can be bound to a
kernel
8 16 32
Maximum number of instructions per kernel 2 million 512 million
Maximum number of Thread Blocks per Thread Block Cluster No 16 8
Technical specifications 1.0 1.1 1.2 1.3 2.x 3.0 3.2 3.5 3.7 5.0 5.2 5.3 6.0 6.1 6.2 7.0 7.2 7.5 8.0 8.6 8.7 8.9 9.0 10.x 12.x
Compute capability (version)

Multiprocessor architecture

Architecture specifications Compute capability (version)
1.0 1.1 1.2 1.3 2.0 2.1 3.0 3.2 3.5 3.7 5.0 5.2 5.3 6.0 6.1 6.2 7.0 7.2 7.5 8.0 8.6 8.7 8.9 9.0 10.x 12.x
Number of ALU lanes for INT32 arithmetic operations 8 32 48 192 128 128 64 128 128 64 64 64 128
Number of ALU lanes for any INT32 or FP32 arithmetic operation N/a N/a
Number of ALU lanes for FP32 arithmetic operations 64 64 128 128
Number of ALU lanes for FP16x2 arithmetic operations No 1 128 128 64
Number of ALU lanes for FP64 arithmetic operations No 1 16 by FP32 4 by FP32 8 8 / 64 64 4 32 4 32 2 32 2 64 2
Number of Load/Store Units 4 per 2 SM 8 per 2 SM 8 per 2 SM / 3 SM 8 per 3 SM 16 32 16 32 16 32
Number of special function units for single-precision floating-point transcendental functions 2 4 8 32 16 32 16
Number of texture mapping units (TMU) 4 per 2 SM 8 per 2 SM 8 per 2 / 3SM 8 per 3 SM 4 4 / 8 16 8 16 8 4
Number of ALU lanes for uniform INT32 arithmetic operations No 2
Number of tensor cores No 8 (1st gen.) 0 / 8 (2nd gen.) 4 (3rd gen.) 4 (4th gen.)
Number of raytracing cores No 0 / 1 (1st gen.) No 1 (2nd gen.) No 1 (3rd gen.) No
Number of SM Partitions = Processing Blocks 1 4 2 4
Number of warp schedulers per SM partition 1 2 4 1
Max number of new instructions issued each cycle by a single scheduler 2 1 2 2 1
Size of unified memory for data cache and shared memory 16 KiB 16 KiB 64 KiB 128 KiB 64 KiB SM + 24 KiB L1 (separate) 96 KiB SM + 24 KiB L1 (separate) 64 KiB SM + 24 KiB L1 (separate) 64 KiB SM + 24 KiB L1 (separate) 96 KiB SM + 24 KiB L1 (separate) 64 KiB SM + 24 KiB L1 (separate) 128 KiB 96 KiB 192 KiB 128 KiB 192 KiB 128 KiB 256 KiB
Size of L3 instruction cache per GPU 32 KiB use L2 Data Cache
Size of L2 instruction cache per Texture Processor Cluster (TPC) 8 KiB
Size of L1.5 instruction cache per SM 4 KiB 32 KiB 32 KiB 48 KiB 128 KiB 32 KiB 128 KiB ~46 KiB 128 KiB
Size of L1 instruction cache per SM 8 KiB 8 KiB
Size of L0 instruction cache per SM partition only 1 partition per SM No 12 KiB 16 KiB? 32 KiB
Instruction Width 32 bits instructions and 64 bits instructions 64 bits instructions + 64 bits control logic every 7 instructions 64 bits instructions + 64 bits control logic every 3 instructions 128 bits combined instruction and control logic
Memory Bus Width per Memory Controller in bits 64 ((G)DDR) 32 ((G)DDR) 512 (HBM) 32 ((G)DDR) 512 (HBM) 32 ((G)DDR) 512 (HBM) 32 ((G)DDR) 512 (HBM) 32 ((G)DDR)
L2 Cache per Memory Controller 16 KiB 32 KiB 128 KiB 256 KiB 1 MiB 512 KiB 128 KiB 512 KiB 256 KiB 128 KiB 768 KiB 64 KiB 512 KiB 4 MiB 512 KiB 8 MiB 5 MiB 6.25 MiB 8 MiB
Number of Render Output Units (ROP) per memory controller (or per GPC in later models) 4 8 4 8 16 8 12 8 4 16 2 8 16 16 per GPC 3 per GPC 16 per GPC
Architecture specifications 1.0 1.1 1.2 1.3 2.0 2.1 3.0 3.2 3.5 3.7 5.0 5.2 5.3 6.0 6.1 6.2 7.0 7.2 7.5 8.0 8.6 8.7 8.9 9.0 10.x 12.x
Compute capability (version)

For more information, read the Nvidia CUDA C++ Programming Guide.

Usages

Comparison with competitors

CUDA competes with other GPU computing stacks: Intel OneAPI and AMD ROCm.

Whereas Nvidia's CUDA is closed-source, Intel's OneAPI and AMD's ROCm are open source.

Intel OneAPI

oneAPI is an initiative based in open standards, created to support software development for multiple hardware architectures. The oneAPI libraries must implement open specifications that are discussed publicly by the Special Interest Groups, offering the possibility for any developer or organization to implement their own versions of oneAPI libraries.

Originally made by Intel, other hardware adopters include Fujitsu and Huawei.

Unified Acceleration Foundation (UXL)

Unified Acceleration Foundation (UXL) is a new technology consortium working on the continuation of the OneAPI initiative, with the goal to create a new open standard accelerator software ecosystem, related open standards and specification projects through Working Groups and Special Interest Groups (SIGs). The goal is to offer open alternatives to Nvidia's CUDA. The main companies behind it are Intel, Google, ARM, Qualcomm, Samsung, Imagination, and VMware.

AMD ROCm

ROCm is an open source software stack for graphics processing unit (GPU) programming from Advanced Micro Devices (AMD).

See also