What is Intel Gaudi 3? A portrait of the AI ac­cel­er­a­tor

The Intel Gaudi 3 is a powerful AI ac­cel­er­a­tor designed specif­i­cal­ly for demanding AI workloads. Gaudi 3 is man­u­fac­tured using the 5-nanometer process, has 64 tensor cores and offers twice as much FP8 per­for­mance and four times the AI computing power of its pre­de­ces­sor. This makes Intel’s Gaudi 3 ideal for inference tasks and training large AI models.

What are the per­for­mance features of Intel Gaudi 3?

With Gaudi 3, Intel is setting new standards in terms of per­for­mance and energy ef­fi­cien­cy. The AI ac­cel­er­a­tor is based on the ar­chi­tec­ture of Gaudi 2, but offers sig­nif­i­cant­ly more computing power, a higher memory bandwidth and better energy ef­fi­cien­cy. The following overview sum­ma­rizes the most important per­for­mance features of Intel Gaudi 3:

• FP8 computing power: The Gaudi 3 achieves an FP8 computing power of 1.835 PFLOPS. Its pre­de­ces­sor achieved just over 0.8 PFLOPS, which means that the per­for­mance for FP8 cal­cu­la­tions has more than doubled.
• BF16 computing power: In BF16 cal­cu­la­tions, the Intel Gaudi 3 also achieves 1.835 PFLOPS, which rep­re­sents a fourfold increase in computing power compared to the Gaudi 2.
• Network bandwidth: Bi-di­rec­tion­al network bandwidth has been doubled to 1200 gigabits per second, enabling faster com­mu­ni­ca­tion between nodes in AI cluster systems.
• HBM capacity and bandwidth: With its HBM memory of 128 gigabytes, the Gaudi 3 offers 50 percent more memory bandwidth than the previous gen­er­a­tion. The HBM bandwidth of 3.7 terabytes per second cor­re­sponds to an increase of 33 percent.

The Intel Gaudi 3 has two compute dies (special computing units) that contain 64 tensor processor cores and 8 MMEs (matrix mul­ti­pli­ca­tion engines for parallel pro­cess­ing). The 24 RDMA NIC ports, each with 200 gigabits per second, ensure fast com­mu­ni­ca­tion via stan­dard­ized Ethernet networks.

What are the ad­van­tages and dis­ad­van­tages of Intel Gaudi 3?

Using an AI ac­cel­er­a­tor of the Gaudi 3 gen­er­a­tion has various ad­van­tages. The most important of these include:

• High computing power: With 1,835 PFLOPS of FP8 and BF16 per­for­mance, Intel’s Gaudi 3 offers tremen­dous per­for­mance similar to the level of the much more expensive NVIDIA H100. According to an Intel press release, the in-house AI ac­cel­er­a­tor even out­per­forms the NVIDIA flagship in some areas.
• High energy ef­fi­cien­cy: The Gaudi 3 AI ac­cel­er­a­tors are man­u­fac­tured using the 5-nanometer process (by TSMC), which enables a higher power density. This reduces power con­sump­tion and lowers operating costs in data centers.
• Cost-effective AI scal­a­bil­i­ty: With Intel Gaudi 3, systems can be flexibly scaled ver­ti­cal­ly and hor­i­zon­tal­ly, which is par­tic­u­lar­ly ben­e­fi­cial for complex de­ploy­ments.
• Support for open standards: As Gaudi 3 supports open standards, the AI ac­cel­er­a­tors can be flexibly in­te­grat­ed into existing IT in­fra­struc­tures. This makes companies more in­de­pen­dent in their choice of AI platforms.

However, the AI ac­cel­er­a­tors also have notable dis­ad­van­tages. Although the Intel Gaudi 3 has first-class per­for­mance, the high-end chips from NVIDIA offer even better per­for­mance on the whole. Why does this matter? Because companies active in the AI field have so far tended to opt for the most powerful rather than the most cost-efficient solution. As a result, the Intel Gaudi 3 is less common than AI ac­cel­er­a­tors from NVIDIA, whose ecosystem benefits from broad support from AI de­vel­op­ment teams.

Which areas of ap­pli­ca­tion is Intel Gaudi 3 best suited to?

Intel Gaudi 3 was developed specif­i­cal­ly for compute-intensive AI workloads and is par­tic­u­lar­ly suitable for inference tasks that require high parallel pro­cess­ing and memory bandwidth. Typical workloads include text gen­er­a­tion with large language models (LLMs), image gen­er­a­tion and speech synthesis. Thanks to its high inference speed and optimized FP8 ar­chi­tec­ture, Gaudi 3 enables powerful and energy-efficient pro­cess­ing of gen­er­a­tive AI models. However, there are other areas of ap­pli­ca­tion. These include:

• Basic training of large AI models: Gaudi 3 makes it possible to process large data sets ef­fi­cient­ly. The AI ac­cel­er­a­tors are therefore ideal for training AI models — such as neural networks for machine learning or trans­former models such as GPT and LLaMA — from scratch.
• Image pro­cess­ing and computer vision: Thanks to its high computing power, the Intel Gaudi 3 is able to process complex image data in real time. This also makes the AI ac­cel­er­a­tor suitable for ap­pli­ca­tions such as security sur­veil­lance or in­dus­tri­al au­toma­tion.
• GPU servers and AI clusters in data centers: The Intel Gaudi 3 can be used for GPU servers to provide the computing power required for AI training and inference tasks.

What are the possible al­ter­na­tives to Intel Gaudi 3?

There are various AI ac­cel­er­a­tors that can be con­sid­ered as al­ter­na­tives to Intel Gaudi 3. One of the best-known al­ter­na­tive options and com­peti­tor products is the NVIDIA H100. While the Intel ac­cel­er­a­tor is ideal for inference ap­pli­ca­tions, the H100 offers high-end per­for­mance for AI and data science use cases. Another fre­quent­ly chosen Gaudi 3 al­ter­na­tive is the NVIDIA A30, which combines a high level of per­for­mance with an af­ford­able price.