PostHeaderSection

Symbol

Global NavBar

PostContentSection

Global Footer

Blog Detail Page

Ryder Hankins

Everyone loves talking about GPUs. They are the star of the show. But at Beyond Summit 2026, Quentin Anthony made the case that the real work happens in everything surrounding it. "In order to keep GPUs going 'brrr,'" he told the crowd, "everything else has to not keep them bottlenecked and keep the tokens fed."

GPU Interconnects at Scale: What Zyphra Learned Training on 1,024 AMD GPUs

Everyone loves talking about GPUs. They are the star of the show. But at Beyond Summit 2026, Quentin Anthony made the case that the real work happens in everything surrounding it. "In order to keep GPUs going 'brrr,'" he told the crowd, "everything else has to not keep them bottlenecked and keep the tokens fed."

Quentin is VP of Engineering at Zyphra, an open-source AI lab training large models on AMD hardware. He came to the topic with real battle scars: Zyphra recently pre-trained their 8B ZAYA1-Base model on a cluster of 1,024 MI300X GPUs and published a paper documenting what they learned. His talk drew directly from that experience, offering a look at what it actually takes to squeeze full bandwidth out of a point-to-point GPU topology at scale.


Why Topology Is Not a Background Detail

Most practitioners in the field have been quietly assuming NVIDIA's NVLink architecture as the baseline. And for good reason: NVLink routes every link through a switch, using a crossbar to reconfigure connections on the fly. The practical upshot is that whether you're communicating between two GPUs or all eight, you get the full link bandwidth. Quentin put it plainly: "You can kind of turn your brain off" when designing parallelism strategies on NVLink. Any degree of tensor parallelism or expert parallelism works.

AMD's Infinity Fabric is fundamentally different. Rather than routing through a switch, it uses a point-to-point topology; direct copper links between GPUs on the baseboard. Each link tops out at 64 GBs. To reach the advertised ~450 GBs that brings AMD in line with NVLink performance, you have to use all the links simultaneously. That means collectives need to span all eight GPUs in the node, not just two or four.

The consequences show up quickly in practice. Run ring attention with fewer than eight GPUs on a point-to-point topology and performance drops noticeably. Tensor parallel degrees of 2 or 4 that work fine on switched hardware leave most of your intra-node bandwidth idle on AMD — not because the hardware is inferior, but because the communication pattern isn't using what's available. Once you know that, you can fix it.


Co-Design, Not Workarounds

Rather than framing this as a limitation to work around, Quentin presented it as a design input. Understanding the hardware deeply enough changes the decisions you make from the start. A practice he calls co-design.

"There is no bad hardware; there's just hardware that you don't understand deeply yet. You just need to look at things in terms of flop bytes per dollar and then maximize your flop bytes."


Expert Parallelism at Degree Eight

For ZAYA1-Base, the team could have used an expert parallel degree of two with a smaller batch size and technically fit the model in memory. Instead, they chose to push expert parallelism to eight and expand the batch size accordingly. The result was meaningfully better performance, not because of any algorithmic breakthrough, but because they matched the parallelism strategy to what the hardware actually delivers.


Tree Attention for Context Parallelism

The second major co-design was around long-context inference. Quentin and his team at Zyphra’s goal is to reach one million tokens and beyond. The standard approach, ring attention, has two problems at that scale.

First, scaling across multiple nodes means traffic has to travel over much slower inter-node connections: a significant drop in speed compared to communication within a single node. Second, even within a node, the ring pattern only uses a fraction of the available links, leaving most of the direct GPU-to-GPU connections sitting completely unused.

Zyphra's solution was to rethink how attention is distributed across GPUs entirely. Instead of passing data around a ring, their approach, which they call tree attention, uses a different communication pattern that puts all available connections to work. The result is full bandwidth utilization within a node, and the ability to scale to much longer contexts across nodes without the same slowdown


The Bigger Picture: Compute Is Outrunning Bandwidth

Quentin zoomed out to close his talk with a trend that frames everything else. Looking at the ratio of compute (FLOPS) to bandwidth across GPU generations, the number keeps climbing. What was roughly balanced in older hardware has roughly doubled by the MI300X and H100 generation, and is on track to double again with the next wave.

That ratio matters because it means keeping GPUs fed requires proportionally more bandwidth per generation. And bandwidth is not keeping pace. 

"To keep that compute fed,we need to be increasingly creative. We need to increasingly change our parallelism schemes and change our software stacks to adjust to the hardware we're getting."

Quentin’s central message is simple: understanding your hardware isn’t optional; it’s foundational. Topology, bandwidth, and parallelism strategy aren't implementation details to sort out later; they're the foundation everything else is built on. Get that right, and the hardware will deliver.


Want to Dive Even Deeper?

Watch the full video here:


TensorWave Cloud: Engineered for AI Velocity

TensorWave is the industry’s premier AMD-exclusive cloud provider, delivering memory optimized AI infrastructure for training, fine-tuning and inference. 

Our platform combines the raw computational power of AMD's latest GPU architectures with innovative liquid cooling technology and ultra-high-speed networking, creating an optimized environment for the most demanding AI workloads. From large language model training to real-time inference, TensorWave provides the performance, scalability, and cost-efficiency that modern AI initiatives demand.

Ready to get started? Connect with a sales engineer. 

GPU Interconnects at Scale: What Zyphra Learned Training on 1,024 AMD GPUs

Product

Solutions

Resources

Company

Product

Resources

Solutions

Company