AI Hardware: How Many GPUs Do You Need?

Discussions of AI development hardware often focus on GPUs, and with good reason: GPUs do the bulk of the work that trains a deep-learning model. And the bigger the model, the more GPUs you need.

But GPUs are only part of the AI hardware story. You also need immense amounts of fast memory; fast input-output circuitry to move data between the GPUs and memory (and between memory and data storage); and most crucially, CPUs to coordinate the whole process.

In a nutshell, you need both GPUs and CPUs in AI development because a GPU won’t do anything unless a CPU tells it to. In this article, we discuss the role of each type of processor and the relationship between them plus considerations for configuring an AI development environment with the right number of each type.

GPUs vs. CPUs in AI

First, let’s review the differences between a GPU and a CPU.

As its name implies, GPUs (graphical processing units) were developed to perform the intense mathematical operations associated with rendering on-screen graphics. Whether you’re watching YouTube videos or playing Red Dead Redemption (or Minecraft, for that matter), the GPU does the heavy lifting that calculates the value of every pixel on the screen in every frame of the video. In high-end gaming computers, the GPU is often the most expensive component.

In a rare instance of serendipity in computing, it turns out that GPUs perform repeated multiply-and-accumulate operations on vast arrays of numbers, and they do this well. Multiply-and-accumulate is the basic function needed to calculate and recalculate the weight values for each connection between neurons in an artificial neural network.

The Central Processing Unit (CPU) is the primary component of a computer that controls the flow of data and instructions. It's also known as the central or main processor, microprocessor, or processor.

Can CPUs perform the same operations that GPUs do? Yes—but because GPUs are optimized for this task, CPUs  are much slower at it. Moreover, CPUs have other things to do, such as run the operating system and the software that coordinates the overall AI model training process and parcels out tasks to the GPUs.

CPUs are also better than GPUs at tasks such as data preprocessing, which is often necessary before feeding data to the GPUs.

So, in AI development, you need both GPUs and CPUs. In general, you need more GPUs than CPUs.

Typical AI Hardware Environments

AI development hardware environments are divided into two basic categories: workstations and servers.

Workstations are standalone desktop or laptop computers that contain an AI-optimized GPU (such as AMD’s Ryzen AI GPU) as well as a high-performance CPU, which can have multiple CPU cores that operate independently.

Server-based AI hardware is deployed in a data center. Multiple server-based GPUs can join forces to address AI development tasks together. Large companies with ample resources deploy AI hardware in their own data centers to power their AI initiatives.

Those without the infrastructure, budget, or expertise to deploy AI server hardware are turning to cloud-based AI development platforms, such as TensorWave.

Optimal Configurations for Different AI Tasks

How do you know how many GPUs and CPUs you need for a given AI task? There are no hard-and-fast rules, but we can provide some guidelines according to the type of task:

• Natural language processing (NLP), including large language models (LLMs): Depending on the model size (measured by the number of weights, or parameters, to be determined), training an NLP model can involve both CPUs and GPUs. As the models grow, the number of processors must also grow, but the ratio of CPUs to GPUs becomes smaller.

• Machine learning: In general, machine-learning algorithm training requires much more GPU capacity than CPU capacity because of the amount of data that must be processed in training. If data preprocessing is needed, more CPUs may be necessary.

• Image recognition, as a special case of machine learning, requires even more GPU power than machine-learning algorithms that process, for instance, financial transaction data. Again, the number of CPUs needed will depend on the amount of data preprocessing needed.

• Video and audio generation: Generative AI that synthesizes speech, music, images, or video (or some combination) is perhaps the most GPU-intensive AI application today. Both training and inference require GPU resources.

• Drones and other autonomous vehicles: Autonomous vehicles, especially self-driving cars that must interact with other vehicles, pedestrians, cyclists, animals, weather, and other unpredictable entities, must be trained to recognize and react properly to every situation—which means huge quantities of training data and vast numbers of GPUs for training. Responding appropriately to a situation requires lightning-fast inference using only on-board resources (both CPUs and GPUs).

TensorWave’s Role in Hardware Configuration

The bottom line is this: Choosing a hardware configuration is as much art as science, and it depends on experience. Different situations call for different numbers and ratios of CPUs and GPUs. A flexible and scalable environment is the best approach.

TensorWave’s AI cloud development platform, based on the AMD MI300X GPU, offers this solution. With guidance from TensorWave’s expert consultants, along with AMD’s ROCm software to help optimize the environment, you can spend more time on development and testing and less time fiddling with the hardware configuration.

To learn how TensorWave can help you right-size a hardware environment for your backlog of AI projects, contact us today.