GPU

Infrastructure
About 1 min read

A processor specialized in parallel computing through thousands of cores, serving as the core infrastructure that determines the training and inference performance of AI models. Optimized for deep learning matrix operations beyond graphics processing, it has recently emerged as the standard for running Large Language Models (LLMs) by integrating High Bandwidth Memory (HBM).

Also known as
Graphics Processing UnitGraphics Processing Unit

Detailed explanation

Thanks to its parallel architecture that processes countless simple computations simultaneously, the Graphics Processing Unit (GPU) performs AI model training and inference tens of times faster than a CPU. Currently, NVIDIA's Hopper (H100, H200) and next-generation Blackwell (B200) architectures lead the market, with AMD's Instinct MI300 series standing as a prominent challenger. In modern AI workloads, video RAM (VRAM) capacity and memory bandwidth are just as critical as raw computation speed (FLOPS). If VRAM is insufficient, the model cannot be loaded, and if bandwidth is low, the inference speed (tokens/second) drops. Users must choose between on-premise deployment or specialized GPU cloud services (such as Lambda, RunPod, etc.) based on model size and budget.

Why it matters in tool selection

The GPU is the most common bottleneck in AI performance. The VRAM capacity of the selected GPU determines the parameter size of the models that can be run, while the memory bandwidth determines the user response speed. In particular, latest processes like Blackwell are highly power-efficient compared to previous generations, playing a crucial role in reducing total cost of ownership (TCO).

Four things to check when choosing a GPU

  • VRAM capacity: Can the size of the model (e.g., Llama-3 70B) fit onto a single GPU?
  • Memory bandwidth (e.g., HBM3e): Does the token generation speed per second meet service requirements?
  • Interconnect technology (e.g., NVLink): Can multiple GPUs be connected and scaled without data loss?
  • Software ecosystem: Is library support smooth within the CUDA environment?

Examples of optimal GPUs by use case

For LLM training and large-scale inference, the NVIDIA H200 (141GB) or B200 (192GB) is the standard. For building cost-effective servers for small-to-medium models, the L40S or A6000 Ada is preferred, while consumer flagship GPUs like the RTX 4090 or 5090 are widely used in local development environments.

Confusing terms

TPU (Tensor Processing Unit)

A dedicated accelerator built by Google, optimized for TensorFlow/JAX computations, available only on Google Cloud.

NPU (Neural Processing Unit)

A low-power processor designed to efficiently handle AI computations on edge devices like smartphones and laptops.

Related terms

TPUNPU