NVIDIA CUDA enables graphics cards to do far more than render images. It allows them to perform complex scientific and AI-related calculations at high speed. Without CUDA and similar acceleration platforms, many recent advances in generative AI would be difficult to achieve at today’s pace and scale.

Key Takeaways

Key takeaways:

  • CUDA is NVIDIA’s proprietary platform for using the computing power of NVIDIA GPUs for general-purpose calculations (GPGPU).
  • The CUDA hierarchy splits a large computation into threads, blocks and grids; streaming multiprocessors execute them in groups of 32 threads called warps.
  • For AI and LLMs, CUDA delivers the massive parallelisation that makes training and inference practical.
  • Since the Volta architecture, Tensor Cores have further accelerated matrix operations, and specialised libraries scale workloads across multiple GPUs.
  • An ecosystem of libraries such as cuBLAS, cuDNN, NCCL, TensorRT and cuFFT, plus tools such as the CUDA Toolkit, NVCC and Nsight, means most developers rarely need to write CUDA code themselves.
  • Frameworks such as PyTorch and TensorFlow build on CUDA in the background, so CUDA remains the technical foundation of today’s AI systems.

What is CUDA?

The acronym CUDA originally stood for ‘Compute Unified Device Architecture’, but NVIDIA now uses it primarily as the name of its proprietary platform for parallel computing on graphics cards. It allows developers to harness the computing power of NVIDIA GPUs for general-purpose calculations. These are no longer limited to traditional graphics tasks, but are known as General-Purpose GPU Computing (GPGPU).

Unlike a CPU, which is designed for a smaller number of complex tasks, a GPU contains thousands of smaller processing units. These can perform many similar calculations simultaneously, making GPUs particularly suitable for highly parallel workloads. CUDA provides the required programming interfaces, compilers and runtime environments to enable software to take advantage of this parallel processing capability.

Developers can access CUDA through programming languages such as C++, Python or Fortran. The actual calculation code runs on the GPU as a so-called kernel function, while the CPU manages the overall program flow. CUDA also provides APIs and runtime functions that enable data transfers between CPU and GPU memory and simplify memory management through features such as Unified Memory.

Over time, CUDA has developed from a developer platform into a comprehensive software ecosystem. Many widely used tools, including PyTorch, TensorFlow and NVIDIA RAPIDS, use CUDA in the background to accelerate calculations on NVIDIA GPUs. This means users can benefit from GPU acceleration without having to write every CUDA operation themselves.

Note

CUDA is closely tied to NVIDIA hardware because it is a proprietary platform developed and supported exclusively for CUDA-compatible NVIDIA GPUs. Applications that rely directly on CUDA therefore cannot easily run on GPUs from other manufacturers, such as AMD or Intel. This close integration between hardware and software is considered one of the reasons for CUDA’s high performance and widespread adoption in AI and high-performance computing.

Technical foundations of the CUDA hierarchy

The core idea behind CUDA is the division of complex computations into many smaller tasks. Rather than processing operations sequentially, CUDA distributes them across thousands of parallel execution units. This organisation is structured through the CUDA hierarchy.

The smallest unit is a thread. Threads execute the same kernel code while processing different data. Depending on conditional statements, they may also follow different execution paths. For example, each thread can process an individual element of a matrix.

Multiple threads are combined into a block. Threads within the same block can communicate with one another and share access to a fast memory area known as shared memory. This allows them to work together efficiently on a task.

Multiple blocks are combined into a grid. Each kernel launch creates a grid consisting of one or more blocks. The CUDA runtime automatically assigns these blocks to the available processing units on the GPU.

At the hardware level, blocks are executed by streaming multiprocessors (SMs). Each SM can handle multiple blocks simultaneously, helping to maximise hardware utilisation. Within an SM, threads are further grouped into sets of 32 called warps. These warps are scheduled together and execute in parallel when their threads follow the same instruction path. This warp-based execution model is a key factor behind the high performance of many CUDA applications.

Importance for AI and LLMs

Progress in the field of AI is largely driven by deep learning models that require enormous numbers of mathematical operations. Training and inference of LLMs mainly involve matrix multiplications, vector operations and tensor calculations. GPUs and NVIDIA CUDA are specifically designed to accelerate these highly parallel workloads.

CUDA provides not only high computational performance, but also the software foundation behind many major AI frameworks. This enables developers to train and run complex models without having to implement every individual GPU operation themselves.

Massive parallelisation for neural networks

Neural networks consist of millions or even billions of parameters. During training, huge numbers of mathematical operations have to be carried out. Many of these calculations are independent of each other and can be executed in parallel. This is exactly where the strength of CUDA lies. Instead of using a few CPU cores, thousands of GPU cores can work on the same task at the same time. Matrix multiplications, activation functions and gradient calculations can thus be massively accelerated. Depending on the model, hardware and implementation, this can reduce training time from months to days or weeks.

Tensor Cores and AI acceleration

Since the Volta architecture, NVIDIA has integrated special computing units called Tensor Cores. These were developed specifically for AI workloads and can execute matrix operations significantly faster than classic CUDA cores. CUDA enables software libraries to use this hardware automatically. In many cases, developers do not have to address Tensor Cores directly. Instead, frameworks such as PyTorch or TensorFlow use the relevant CUDA libraries in the background. Tensor Cores deliver a substantial performance boost, especially for deep learning, generative AI and LLMs.

Note

With CUDA Toolkit 13.0, offline compilation and library support for Maxwell, Pascal and Volta architectures have been discontinued. Systems using these GPU architectures should continue to use the CUDA 12.x series.

Scaling across multiple GPUs

Very large LLMs can exceed the memory capacity of a single GPU. Depending on model size, numerical precision and available GPU memory, they may need to be distributed across multiple GPUs. NVIDIA CUDA provides the foundation for this scaling approach. Specialised libraries enable fast communication between GPUs within a single server or across entire GPU clusters. This allows datasets, models and computations to be distributed across many accelerators.

Foundation for modern AI frameworks

Most developers do not interact with CUDA code directly. Instead, they rely on frameworks such as PyTorch or TensorFlow, which automatically convert many operations into CUDA-based processes. This allows users to take advantage of GPU acceleration without needing to manage the underlying hardware details. However, CUDA remains the core technical layer behind this functionality. When a tensor in PyTorch is transferred to an NVIDIA GPU, the calculations are ultimately executed through CUDA libraries and kernels.

The CUDA ecosystem of libraries and tools

CUDA is more than just a programming interface. Over time, NVIDIA has developed a comprehensive ecosystem of libraries, development tools and optimisation frameworks around the platform. This ecosystem is one of the key factors behind the widespread use of NVIDIA CUDA in AI. Most developers no longer write custom CUDA code themselves. Instead, they use specialised libraries that are already optimised for modern NVIDIA GPUs, enabling high performance without having to handle every aspect of GPU optimisation manually.

Important CUDA libraries

A key part of the CUDA ecosystem is its range of specialised libraries, which provide commonly used computational functions that are already highly optimised for CUDA-compatible GPUs. This simplifies development by allowing programmers to use existing, efficient implementations instead of creating them from scratch. The most important libraries include:

  • cuBLAS: cuBLAS stands for CUDA Basic Linear Algebra Subroutines. It is a GPU-optimised implementation of the BLAS library for linear algebra operations. It provides functions for matrix multiplications, vector operations and other common mathematical calculations.
  • cuDNN: cuDNN stands for CUDA Deep Neural Network Library. It provides highly optimised functions for deep learning and neural network operations and is widely used by frameworks such as PyTorch and TensorFlow.
  • NCCL: NCCL (NVIDIA Collective Communications Library) handles communication between multiple GPUs. It enables the fast exchange of parameters and intermediate results during the training of large models and is particularly important for distributed training of large LLMs.
  • TensorRT: TensorRT is designed to optimise and accelerate trained AI models for inference. The library analyses models and improves their execution performance on NVIDIA GPUs.
  • cuFFT: cuFFT provides GPU-accelerated Fourier transform functions. It is mainly used in scientific computing, signal processing and simulation applications.

Important CUDA tools

In addition to its libraries, the CUDA ecosystem includes a range of development and analysis tools that help developers create, debug and optimise GPU-accelerated applications:

  • CUDA Toolkit: The CUDA Toolkit is the foundation of CUDA development. It includes compilers, libraries, runtime components and documentation.
  • NVCC: NVCC is NVIDIA’s official CUDA compiler. It converts CUDA code into executable GPU code and automatically separates the CPU and GPU components of a program. Developers typically use NVCC when building their own CUDA applications.
  • Nsight Systems: Nsight Systems is a performance analysis tool for complex applications. It provides insights into how the CPU, GPU and memory interact during program execution, helping developers identify bottlenecks and optimisation opportunities.
  • Nsight Compute: Nsight Compute enables detailed analysis of individual CUDA kernels. Developers can use it to examine memory access patterns, resource utilisation and GPU efficiency. The tool is often used to optimise particularly demanding applications.
  • CUDA-GDB: CUDA-GDB extends the GNU debugger with GPU debugging capabilities. It allows developers to analyse CUDA applications and investigate errors directly within GPU code, simplifying the development of CUDA-based software.
Note

The CUDA Toolkit can be installed on Ubuntu 24.04 using NVIDIA’s official package repositories. Alternatively, local installation packages and runfile installers are available for specific use cases. For most users, NVIDIA recommends using the package manager, as it simplifies the management of CUDA components and future updates.

Why CUDA remains central to AI development

CUDA has developed from a platform for GPU programming into one of the most important software foundations in modern computing. Its architecture enables efficient use of the massive parallel processing capabilities of CUDA-compatible NVIDIA GPUs and provides the technical foundation for numerous AI, deep learning and HPC applications. In particular, CUDA has become a key connection between software and hardware in the development of large language models and generative AI. Understanding the basics of NVIDIA CUDA is therefore essential for anyone who wants to understand how modern AI systems are trained and operated.

Go to Main Menu