What is High Bandwidth Memory (HBM) and why is it important for AI GPUs?

High Bandwidth Memory (HBM) is a 3D-stacked DRAM technology that sits directly on the GPU package and uses extremely wide memory interfaces to enable data rates of several terabytes per second. By combining high bandwidth, short signal paths and low power consumption, HBM has become a key technology for AI accelerators and large language models.

Key Takeaways ⓘ

The key facts about High Bandwidth Memory at a glance:

• HBM is 3D-stacked DRAM placed directly beside the GPU on the same package, which keeps signal paths very short.
• Instead of relying on high clock rates, HBM uses extremely wide memory interfaces with thousands of parallel connections.
• From HBM2 to HBM3e, bandwidth rose from around 256 GB/s to more than 1.2 TB/s per stack.
• HBM4 doubles the interface width to 2048 bit per stack and reaches around 2 TB/s according to JEDEC.
• Through-silicon vias and microbump bonding connect the stacked memory dies; the short distance to the GPU cuts energy use per transmitted bit.
• Compared with GDDR6 and GDDR7, HBM delivers significantly more bandwidth per watt in a smaller footprint.
• For large language models, memory bandwidth is often the limiting factor rather than compute power alone.

What is High Bandwidth Memory (HBM)?

HBM stands for ‘High Bandwidth Memory’ and describes a specialised memory architecture used in high-performance computers, GPUs and AI accelerators. Unlike conventional graphics memory such as GDDR, HBM is not placed separately on the board but directly beside the GPU in the same package. This reduces electrical signal paths and improves data transfer efficiency. HBM also uses extremely wide memory interfaces with thousands of parallel connections, allowing large amounts of data to be transferred simultaneously.

AI models rely on this high memory bandwidth because billions of parameters need to be continuously transferred between the GPU and memory. HBM therefore combines high performance with comparatively low energy consumption. It is particularly important for training and inference of large LLMs, where memory bandwidth can have a significant impact on overall performance.

The evolution from HBM2 to HBM3e

The development of High Bandwidth Memory (HBM) focuses primarily on continuously increasing memory bandwidth. The key factor here is the so-called pin speed, i.e. the data rate per line.

The key development steps of HBM include:

• HBM2: Established HBM as memory for HPC and AI accelerators
• HBM2e: Improved bandwidth and higher memory capacities
• HBM3: Significantly higher transfer rates for AI systems
• HBM3e: Optimised variant with up to 9.6 Gbps pin speed
• HBM4: New generation with doubled interface width

HBM2e typically achieved transfer rates of around 3.2 to 3.6 Gbps per pin, depending on the implementation. With HBM3, data rates increased significantly to around 6.4 Gbps. HBM3e further raises pin speeds to over 9 Gbps, with even higher data rates possible depending on the implementation. As a result, HBM3e can deliver more than 1.2 TB/s per stack.

This development is important for AI architectures such as the NVIDIA Hopper platform. GPUs like the NVIDIA H100 and NVIDIA H200 use HBM3 and HBM3e respectively to train and run large language models efficiently. Without this memory bandwidth, GPU compute units would often have to wait for data.

Comparison of HBM generations

HBM4 and the move to a 2048-bit interface

HBM4 is considered one of the biggest technological leaps so far in the development of High Bandwidth Memory. The most important innovation in HBM is the doubling of the memory interface width from 1024 to 2048 bit per stack. This allows significantly more data to be transmitted in parallel.

While earlier generations became faster mainly through higher clock speeds, HBM4 now combines high pin speeds with vastly more parallel data lines and very high bandwidths. Several HBM4 stacks on a single AI accelerator can thus enable total data rates in the double-digit terabytes range.

This is particularly crucial for LLMs and GenAI. Large transformer models move enormous volumes of data between compute units, activations and model weights. Even state-of-the-art GPUs today are often limited by memory bandwidth rather than by sheer computing power alone.

Another important advance concerns the so-called base die. This lowest layer of a High Bandwidth Memory stack controls communication with the GPU interface. With HBM4, the base die becomes significantly more important: it includes more complex logic for control, signal routing and energy efficiency. Depending on the manufacturer and customer requirements, it can also be produced using more advanced logic processes.

Additionally, the new HBM improves:

• energy efficiency per transmitted bit
• signal stability at high data rates
• scalability for multi-die GPUs
• memory connectivity for large AI clusters

3D stacking and TSV technology behind HBM performance

The high performance of High Bandwidth Memory is based on a specialised 3D architecture. Multiple memory dies are stacked vertically and directly connected to each other in a process known as 3D stacking.

Communication between the individual memory layers takes place via so-called through-silicon vias (TSVs). These microscopic vertical connections run through the silicon and enable extremely fast data transfer between the stacked memory chips.

The dies are also connected using microbump bonding. These tiny contact points create dense electrical connections within the stack, forming a compact high-speed memory system.

The key advantage, however, is the physical proximity to the GPU. HBM is placed directly on the same package as the AI accelerator, meaning data does not have to travel long distances across the mainboard.

This helps reduce several challenges:

• lower signal loss
• lower latency
• less electromagnetic interference
• lower energy consumption per transmitted bit

Energy efficiency is particularly important. Conventional memory uses comparatively more energy to transmit signals over longer distances. High Bandwidth Memory reduces this demand and achieves very low pJ/bit values (picojoules per bit). This is crucial for data centres, where AI systems require enormous amounts of electricity.

HBM vs. GDDR6/7 and how HBM overcomes the limits of conventional VRAM

Conventional graphics memory such as GDDR6 or GDDR7 is primarily designed for discrete GPUs and cost-sensitive high-bandwidth applications like gaming, workstations and certain compute scenarios. The focus there is mainly on high clock rates and cost-efficient memory solutions. For current AI models, however, this concept is increasingly reaching its limits, because GDDR works with comparatively narrow memory interfaces and compensates for this with very high frequencies. High Bandwidth Memory takes the opposite approach, using extremely wide interfaces at lower clock rates. This is precisely how HBM achieves significantly higher efficiency.

These differences are particularly evident in:

• memory bandwidth
• energy consumption
• space requirements
• scalability
• thermal load

GDDR memory is placed around the GPU on the PCB rather than directly alongside it. This requires longer data paths and can increase energy consumption. At very high frequencies, signal loss and heat generation can also become greater challenges. HBM is much more compact: multiple memory stacks are positioned directly beside the GPU on a shared interposer. This enables extremely high bandwidths within a small footprint.

For AI applications, memory bandwidth is a critical factor. Large language models with hundreds of billions of parameters constantly move vast amounts of data. If the GPU can process data faster than the memory can provide it, a bottleneck occurs. This challenge is becoming increasingly important as modern AI accelerators continue to increase in computing power faster than conventional memory technologies can scale. HBM helps address this limitation.

Energy efficiency is another important factor. In large AI data centres, reducing power consumption is increasingly important. Compared with conventional GDDR memory, HBM delivers significantly more bandwidth per watt.

However, this does not mean that GDDR will disappear. GDDR6 and GDDR7 remain practical and cost-effective solutions for gaming graphics cards and many other applications. For high-end AI accelerators and large HPC systems, however, HBM has become the leading memory technology.

HBM and the next generation of AI infrastructure

HBM has evolved from a specialist technology into a key foundation of AI infrastructure. Without the enormous memory bandwidth provided by High Bandwidth Memory, many current large language models and HPC applications would be far more difficult to operate efficiently.

This development also shows that memory technologies are becoming just as important as GPU computing power itself. AI accelerators are increasingly constrained by factors such as memory bandwidth, energy efficiency and data movement. Manufacturers such as SK Hynix, Samsung Electronics and Micron Technology therefore play a key role in the AI market.

HBM4 represents the next generation of HBM technology and, following the publication of the JEDEC JESD270-4 standard, reached volume production in 2026. NVIDIA moved its Vera Rubin platform into full production at the end of May 2026, Micron is manufacturing the HBM4 memory designed for it in high-volume production, and Samsung Electronics and SK Hynix are qualified as additional suppliers.