How AT&T, TensorWave, and the creators of the Transformer architecture are redefining what's possible with domain-specific AI models running on AMD GPUs at unprecedented scale.

Executive Summary

The landscape of AI infrastructure is undergoing a fundamental transformation. While the industry has been fixated on pre-training ever-larger foundation models, a quiet revolution has been taking place in post-training and reinforcement learning that promises to democratize AI development. In a landmark masterclass at Talent Arena, Greg Diamos (ScalarLM Architect and pioneer of GPU deep learning), Farbod Tavakkoli (Data Scientist at AT&T), and Ashish Vaswani (lead author of "Attention Is All You Need") demonstrated how organizations can build and train domain-specific AI models from scratch using entirely open-source software at scales previously reserved for frontier labs.

The result? A 240x speedup on 256 AMD MI325 GPUs, proving that the future of AI infrastructure isn't just about NVIDIA, and the future of AI development isn't just about pre-training.

Access the masterclass here and read below for our deep dive.

The Genesis: From CUDA to Collective Intelligence

To understand where AI infrastructure is headed, we must first understand where it came from. Greg Diamos's journey through the evolution of deep learning infrastructure provides a unique window into the challenges that have shaped our current technological landscape.

In 2012, while at NVIDIA as the software architect for CUDA, Diamos faced a fundamental challenge: how do you make GPUs programmable for the massively parallel workloads that deep learning demands? The solution required rethinking how computation itself is distributed across thousands of processing cores. This work culminated in the development of collectives: the communication primitives that enable distributed training across multiple GPUs. These collectives eventually became part of libraries like NVIDIA's NCCL and AMD's RCCL, forming the backbone of modern distributed deep learning.

The breakthrough wasn't just technical but fundamentally philosophical. The team discovered that you could shrink the numerical representation of neural network weights through mixed precision training, achieving massive efficiency gains while maintaining model accuracy. When NVIDIA released Tensor Cores in 2018 with hardware acceleration for these formats, the industry had its proof point: you could train faster and more efficiently by thinking differently about numerical precision.

But perhaps the most profound insight came from the language model scaling laws discovered at Baidu in 2017. The premise was elegant in its simplicity: invest more compute in training, and models get smarter predictably. They don't just improve at existing tasks but learn entirely new abilities. This insight, later championed by Ilya Sutskever as "the driver of all progress in AI in the last decade," forms the theoretical foundation for why we continue to build ever-larger training clusters.

The Post-Training Revolution: Beyond Pre-Training

While pre-training has dominated headlines, the real frontier has shifted. Today's most capable open models like DeepSeek R1 and the latest Qwen 3.5 releases use reinforcement learning (RL) to achieve their remarkable capabilities. The process works aligned to the model below:

The model thinks—generating reasoning trajectories that explore different approaches to solving a problem
The model evaluates—using reward signals to determine which approaches worked
The model learns—baking successful patterns back into its weights through backpropagation

This isn't just theoretical. The Qwen 3.5 benchmarks show a clear progression: as models are exposed to more reinforcement learning training environments, their performance on complex tasks improves dramatically. The key insight is that inference and training must be unified, you can't efficiently do one without the other.

This is where the industry's current infrastructure falls short. Most organizations have separate teams, systems, and workflows for training and inference. When you're doing reinforcement learning at scale, where individual training steps need to publish new models and incorporate feedback from inference on those models within seconds, this separation becomes a fatal bottleneck. Deployments that should take milliseconds instead take days or weeks.

Introducing ScalarLM: The Eighth Generation

ScalarLM represents the culmination of over a decade of iterative development in training and inference infrastructure. Building on the lessons learned from seven previous generations at Baidu, Apple, AI Fund, MLCommons, and multiple startups, ScalarLM takes a clean-slate approach to unifying training and inference.

The architecture is elegant in its integration:

The Key Innovation: Unifying Inference and Training

The breakthrough in ScalarLM isn't just about gluing existing tools together but fundamentally about creating a seamless feedback loop between inference and training. Here's how it works:

For Training: You provide input-output pairs (prompts and completions), potentially with reward scores. The system handles tokenization, shuffling, and distribution across hundreds of GPUs automatically.

For Inference: The system maintains a persistent queue that can handle hundreds of thousands of requests. This isn't just about batching but it's about flow control that prevents the cluster from being overwhelmed while maintaining high utilization.

The result is that you can write code like this:

This simplicity is deceptive. Under the hood, ScalarLM is managing distributed training across hundreds of GPUs, automatically converting models between Megatron and vLLM formats, handling checkpoint management, and orchestrating all the complex networking required for high-performance training.

The Hardware: TensorWave's AMD MI300 Cluster

One of the most compelling aspects of this work is the hardware platform. TensorWave has built one of the world's largest AMD GPU clusters, with pods containing 2,000+ MI300 GPUs. These aren't just AMD alternatives to NVIDIA but they represent a fundamentally different approach to AI infrastructure.

The AMD MI300X offers:

192GB of HBM3 memory (vs. 80GB on H100)
5.3 TB/s memory bandwidth
Superior memory capacity for large models

The MI325X pushes this further with:

256GB of HBM3e memory
6 TB/s memory bandwidth

This memory advantage is critical for two reasons:

Larger models on fewer GPUs: You can fit bigger models or larger batch sizes on each GPU
Longer context windows: More memory means more KV cache, which means longer conversations and documents

The hardware specs are key but the software stack makes it usable.

The AMD Software Journey: From "Not Ready" to Production

TensorWave's CEO Darrick Horton was candid about AMD's early challenges: "It's very publicly documented that AMD training performance was not ideal in 2024." The hardware was good, but the software stack lagged behind NVIDIA's mature CUDA ecosystem.

The difficulty in creating a training stack is orders of magnitude larger than creating an inference stack. It's not just about running computations but coordinating thousands of GPUs, managing network communication, handling failures, and optimizing every layer of the stack.

Over the past 12-16 months, AMD has dramatically improved its ROCm software stack, and ScalarLM has been designed to be hardware agnostic from the ground up. The same code that runs on NVIDIA H100s can run on AMD MI300s with minimal changes.

This matters for the industry. NVIDIA's dominance has created supply constraints and pricing pressure. A viable alternative means more choice, more availability, and ultimately more innovation.

Open Telco Models: A Real-World Application

AT&T's Farbod Tavakkoli brought the technical discussion into the real world with the Open Telco Models project. This isn't a research demo but a very powerful production system with 30+ trained models, all open-source with open weights, open datasets, and a complete open training stack.

The telecommunications industry presents unique challenges for AI:

Root cause analysis: Understanding why network issues occur
Technical embeddings: Representing telecom-specific knowledge accurately
Operational intelligence: Making real-time decisions about network operations

These aren't problems that general-purpose foundation models solve well. They require domain-specific training on domain-specific data. The Open Telco Models project demonstrates how organizations can take open-source foundation models and adapt them for their specific needs.

The process follows the reinforcement learning paradigm:

Start with a pre-trained model (like rnj-1, the first US-trained open-weight model trained on AMD)
Define domain-specific tasks and reward functions
Run inference to generate reasoning trajectories
Evaluate trajectories against reward functions
Update the model through backpropagation
Repeat at scale

The Scale Achievement: 241x Speedup on 256 GPUs

The headline result from the masterclass was a 241x speedup on 256 AMD MI325 GPUs for post-training workloads. But what does this actually mean?

Speedup in distributed training is measured against a single GPU baseline. If training takes 241 hours on one GPU, and 1 hour on 256 GPUs, that's a 241x speedup. This isn't just about raw performance but an execution model for efficiency.

Linear scaling (256x on 256 GPUs) is the theoretical maximum. Achieving 241x means the system is operating at 93.75% efficiency, an extraordinary achievement for post-training workloads that involve complex coordination between training and inference.

This efficiency comes from several factors:

Minimized communication overhead: Smart data parallelism that reduces the amount of information exchanged between GPUs
Efficient tensor parallelism: Splitting model tensors across GPUs only when necessary
Unified memory management: Avoiding unnecessary data movement between training and inference
Flow control: Keeping GPUs busy without overwhelming the system

Deep Dive: How ScalarLM Achieves Its Performance

Distribution Strategies

ScalarLM inherits and builds on Megatron-LM's three distribution strategies:

Data Parallelism: The most widely used strategy. Each GPU gets a different chunk of training data, trains a separate copy of the model, and then gradients are synchronized across all GPUs. Modern implementations keep weights synchronized during training using efficient all-reduce operations.

Tensor Parallelism: Used when models don't fit in a single GPU's memory. Individual tensors are split across GPUs, with communication coordinated during forward and backward passes.

Pipeline Parallelism: Splits the model by layer, with each GPU handling different layers. Less efficient than other methods but sometimes necessary for very deep models.

The key insight is that data parallelism is usually the best starting point. It's simple, efficient, and works well for most post-training scenarios. Tensor and pipeline parallelism add complexity and should be used only when necessary.

The Flow Control Innovation

One of ScalarLM's most important innovations is its approach to flow control. When you're running reinforcement learning at scale, you need to:

Generate thousands or millions of reasoning trajectories
Submit them to the inference cluster
Wait for results
Update the model
Repeat

The challenge is that submitting too many requests at once overwhelms the cluster (leading to the kind of outages that affected services like Claude), while submitting too few leaves GPUs underutilized.

ScalarLM solves this with a network-style flow control mechanism that automatically right-sizes the request rate based on cluster capacity. Users can submit 200,000+ prompts in one shot from their laptop, and the system handles the complexity of processing them efficiently.

SLURM + Kubernetes: Best of Both Worlds

The infrastructure layer combines SLURM (for high-performance networking) with Kubernetes (for container orchestration). This is a key architectural decision that solves a real problem in the industry.

SLURM isn't just a job scheduler but the system that sets up MPI (Message Passing Interface) and the high-performance network stack. Setting up RDMA (Remote Direct Memory Access) networking manually is, in Diamos's words, "a nightmare" historically requiring teams of engineers and millions of dollars.

Kubernetes provides the container management that modern software development requires. By running SLURM inside Kubernetes containers, ScalarLM gets the best of both worlds: the high-performance networking essential for distributed training, and the container management essential for software development.

The GRPO Algorithm: Thinking Multiple Times

The masterclass provided a deep dive into GRPO (Group Relative Policy Optimization), the algorithm popularized by DeepSeek that powers many of today's most capable models.

The concept is intuitive:

Present a question to the model
Generate multiple reasoning paths (the model "thinks" about different approaches)
Evaluate each path using a reward function
Update the model to prefer paths that worked

The reward function can be simple or complex. Diamos offered a straightforward example: if you're training a model to write code, the "compile button" is a reward function. If the code compiles, positive reward. If it crashes, negative reward.

This simplicity is powerful. You don't always need complex reward models but sometimes the outcome itself is a sufficient signal. And because modern language models can read their own outputs, they can often determine success or failure without explicit scoring.

Practical Implementation: From Zero to Training

The masterclass demonstrated how simple the user experience can be. Here's what a typical workflow looks like:

Step 1: Define Your Data

Step 2: Train

Step 3: Use

Under the hood, ScalarLM handles:

Tokenization using HuggingFace tokenizers
Shuffling for stochastic gradient descent
Distribution across available GPUs
Checkpoint management
Automatic format conversion between training and inference

The Security Advantage: No External Dependencies

A surprising benefit of ScalarLM's architecture is its security posture. Unlike systems that require cloud object storage, managed databases, and external load balancers, ScalarLM requires only:

GPU compute nodes
A high-performance interconnect
A shared filesystem

This enables deployment in secure enclaves with no external network dependencies. For organizations handling sensitive data—healthcare, finance, defense—this is a significant advantage.

The team made a key observation: training datasets for LLMs are actually quite small. A dataset like Alpaca (52K examples) might be 15-20 MB compressed. Even large datasets of hundreds of thousands of examples are only a few GB. A modern GPU cluster that can train a 70B parameter model can easily handle storing and moving a few GB of training data without external object storage.

The Rnj-1 Connection: Ashish Vaswani's Contribution

A special guest at the masterclass was Ashish Vaswani, lead author of the "Attention Is All You Need" paper that introduced the Transformer architecture. Vaswani has been working on Rnj-1, an open-weight language model trained in the US on AMD GPUs.

Rnj-1 represents an important milestone: it's among the first open-weight models trained primarily on AMD infrastructure, demonstrating that the AMD ecosystem is mature enough for serious AI development. The collaboration between the Transformer's creator and the TensorWave/ScalarLM team underscores the open, collaborative nature of this work.

Lessons Learned: Ten Years of Infrastructure Development

Greg Diamos shared hard-won lessons from a decade of building training infrastructure:

Training and inference are both essential, but most frameworks focus on one or the other. This separation is artificial and harmful for modern RL-based training.
SLURM and Kubernetes have different worldviews, but pieces of both are useful. The future lies in combining them effectively.
A model exchange format has not yet emerged. HuggingFace is the closest thing we have, and building on it enables maximum compatibility.
The main dependency is the GPU. Everything else can be simplified away, leading to more secure, more portable systems.
Data parallelism is usually the answer. Complex distribution strategies add overhead. Start simple and scale.

Future Directions: What's Next for ScalarLM

The ScalarLM team has ambitious plans for v1.0 and beyond:

Performance optimizations and kernel-level benchmarks on modern GPUs including MI300X and H200
Integration with SLINKY for improved SLURM-Kubernetes integration
Expanded model support as new architectures emerge
AILuminate benchmark integration for standardized evaluation

The long-term vision is to enable LLM agents that train and deploy themselves—systems that can learn from their own experiences and improve over time without manual intervention.

The Bigger Picture: Democratizing AI Infrastructure

The Open Telco Models project and ScalarLM represent something larger than any single technical achievement. They represent a democratization of AI infrastructure that has profound implications:

For Organizations: You don't need a frontier lab's budget to train capable models. You can use open-source software, open weights, and open datasets to build systems tailored to your domain.

For the Industry: AMD provides a viable alternative to NVIDIA, creating competition that benefits everyone. The software ecosystem around AMD is maturing rapidly.

For Society: Open models and open infrastructure mean more transparency, more auditability, and more distributed capability. The future of AI doesn't have to be controlled by a handful of companies.

Getting Started: Explore the Technology

ScalarLM: Available at scalarlm.com with full source code on GitHub
TensorWave AI Cloud: Access to the latest AMD accelerators, and AI cloud services at tensorwave.com
Open Telco Models: 30+ open-source models for telecommunications applications
Documentation: Comprehensive docs at docs.scalarlm.com

Key Takeaways for the technical summary

The Paradigm Shift

The AI industry is transitioning from pre-training focused development to post-training and reinforcement learning. The most capable models today don't just predict the next token but they think, reason, and learn from their own outputs. This shift demands infrastructure that unifies training and inference, not separates them.

The Hardware Reality

AMD GPUs are production-ready for AI training. The combination of TensorWave's infrastructure and ScalarLM's software stack demonstrates that organizations have a viable alternative to NVIDIA. AMD's superior memory capacity (192-256GB vs. 80GB on H100) provides tangible benefits for large model deployment.

The Software Innovation

ScalarLM represents the 8th generation of distributed training infrastructure, learning from a decade of development at Baidu, NVIDIA, Apple, and MLCommons. Its key innovations include:

Innovation	Problem Solved	Impact
Unified Training + Inference	RL requires seamless handoffs between training and inference	Eliminates deployment bottlenecks
Flow Control	Thundering herd from massive parallel inference	8-10x throughput improvement
SLURM + Kubernetes	HPC networking vs. container management	Best of both worlds
Hardware Agnostic Design	Vendor lock-in and supply constraints	Code runs on AMD and NVIDIA
Secure Enclave Architecture	External dependencies create attack surface	No databases, object stores, or cloud services required

The Performance Achievement

241x speedup on 256 AMD MI325 GPUs for post-training workloads—representing 93.75% scaling efficiency. This proves that large-scale distributed training on AMD is not just possible, but competitive with NVIDIA infrastructure.

The Democratization impact of the Open Telco Models project demonstrates

Domain-specific AI is achievable without frontier-lab resources
Open-source software can power production AI systems
Organizations can build competitive AI capabilities on open infrastructure

What This Means for AI Infrastructure Leaders

Evaluate AMD seriously: The software ecosystem has matured significantly in 2024-2025
Invest in post-training capabilities: Pre-training is table stakes; differentiation comes from RL-based training
Unify your training and inference stacks: Separate systems create friction that modern RL workflows cannot tolerate
Consider the full TCO: AMD's price/performance and availability advantages matter for large-scale deployments
Build on open standards: HuggingFace, vLLM, and Megatron-LM form a mature ecosystem that reduces lock-in

References

Vaswani, A., et al. "Attention Is All You Need." NeurIPS 2017.
Diamos, G., et al. "Mixed Precision Training." ICLR 2018.
DeepSeek R1 Technical Report.
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.
vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention.
TensorWave AMD MI300X Benchmarks.
ScalarLM Documentation and Source Code.

Building Domain-Specific AI at Scale: A Deep Dive into Open Telco Models and the Future of Post-Training Infrastructure