PostHeaderSection

Symbol

Global NavBar

PostContentSection

Global Footer

Blog Detail Page

Byron Anderson

Explore how Modular is building a hardware-agnostic AI software stack with Mojo and MAX to reduce GPU fragmentation and enable portable AI workloads.

Breaking GPU Lock-In with Modular Mojo and MAX

Beyond Summit 2026 challenged one of AI's biggest assumptions: that the future of AI infrastructure has to be built around a single vendor. Formerly known as Beyond CUDA, this year's event focused on the industry's shift toward a more open AI ecosystem. One session explored that transition from the software perspective. In this blog, we look back at “Breaking Free from GPU Fragmentation”, presented by Mostafa Hagog, VP of Engineering at Modular.


Mission Beyond CUDA 

Hagog opened by talking about what Modular’s goal is in their current market position: to fix fragmented AI software infrastructure and to unlock the full potential of existing hardware. Hagog states, “It's 15 years old. It's time for something new.”

Rather than tying developers to a single hardware ecosystem, Modular is building a hardware agnostic software stack that enables AI models to run across GPUs, CPUs, and other AI accelerators without requiring code rewrites or sacrificing performance.

The goal is to give developers the flexibility to deploy across hardware platforms while avoiding vendor lock in. Hagog noted, “We wanted to make it available for everybody everywhere and able to run on any hardware. We're not just talking about GPUs and CPUs, but future AI accelerators as well.” Modular's vision extends well beyond compilers and kernels. The company aims to build the entire AI software stack, from compilers and libraries to developer tools, models, serving infrastructure, and ultimately the cloud. By managing the full stack, developers can move between hardware platforms more easily while maintaining performance and portability.


Modular Framework 

The framework that Modular uses is threefold: Mojo, their programming language; MAX, similar to vLLM or SGLang; and their cloud layer. Each layer serves as a distinct part of the pipeline, with Mojo giving developers a single language to write, MAX compiling and serving the code, and the cloud layer deploying it at scale. Together, they're designed to replace the patchwork of vendor-specific tools that CUDA-based stacks require.

Mojo is positioned as the equivalent of ROCm or CUDA, but doesn’t require rewrites to be compatible across numerous hardware systems. Hagog mentions, “Mojo was designed in principle to be able to be portable but also to solve the performance problem.” Mojo, originally designed for CPUs, later expanded to GPUs and has now continued to evolve alongside Modular’s stack. He says Modular often finds that Mojo kernels outperform external kernels and integrate more easily with the stack, because they work natively with fusion, graph analysis, and parameterization. Mojo 1.0 is currently set to stabilize later in 2026.

MAX, used for serving generative AI models, offers wide compatibility and high performance, with much of its codebase published openly under Modular's Community License. Hagog notes, Modular built MAX because Mojo kernels are designed to understand symbolic tensors and the graph they are part of. Their containers, free from PyTorch and vendor libraries, are extremely small, only around 500 to 600 megabytes. Hagog emphasizes that there is “no magic” in performance and hardware-specific specialization is still necessary. The key is being portable by default while specializing when needed. MAX is also “hackable” so it can integrate vendor libraries and make them look like Mojo kernels. Hagog stated that it’s useful for shortening the time to market and that Modular’s business goal is to give customers the best performance as quickly as possible.


Cloud Layer

The cloud layer is needed for scaling across many nodes, where optimizations like distributed KV caching and routing become important. The cloud layer maintains a cluster-wide view of where KV cache currently lives across nodes. Because it knows which node already holds the relevant context, it can route incoming requests there directly instead of defaulting to round-robin or load-only routing. This structure avoids redundant recomputation, provides a faster time-to-first-token, and improves GPU utilization across the cluster, since hardware is no longer sitting idle re-deriving something another node has already computed. The difference is an optimization that doesn't just work well on a single GPU, but holds up across clusters scaled to entire data centers. Hagog said, “The cloud layer is not just the gateway; it's also about all the optimizations that come when you scale into hundreds of nodes.” 

Hagog also mentioned that Modular partnered with the Google Brain team to bring Gemma 4 to multiple hardware platforms at their day-zero launch, including AMD’s MI355 GPUs. “I would say that the container was ready at the time the post was published. We were able to achieve better throughput than a framework that came with the model launch on two platforms with a small team of two people for a few days,” Hagog said. He emphasizes that they’re not just offering an idea, but a deployable, functional solution. 


What Users and Engineers Actually Need

Modular builds with user needs at the forefront rather than designing the stack in isolation and adjusting afterward. Hagog highlighted this, saying, "When we design the language, we design it by asking the kernel engineers, 'Is this actually good for you? Is this useful?' We do the same thing at the different layers of the stack. We try to build that stack from the perspective of the user and the use cases that customers care about." This bottom-up approach extends to reliability: systems are built to run reliably 24/7, without users having to worry about optimizations that crash in production. 

The Gemma 4 partnership is a proof point of this philosophy. The Modular team hit day-zero readiness with just two engineers in a couple of days; a pace that would be hard to replicate on a stack locked to a single vendor's configuration. Hagog also noted, “We were able to achieve better throughput than a framework that came with the model,” highlighting that Modular's portability doesn't come at the cost of performance.

This strategy to build infrastructure rivals the norm of building spec first and having user feedback treated as an afterthought. It's part of why the stack is designed around engineer’s actual workflows rather than theoretical use cases. In practice, this means fewer surprises for the teams deploying it; the stack is designed to hold up under real production conditions, not just in a demo.


Conclusion 

Modular’s full-stack alternative to the CUDA-era infrastructure gives users portability across hardware, faster production performance than current alternatives, and a seamless process all the way from kernels to cloud serving. If Hagog's session made one thing clear, it's that Modular isn't pitching a theoretical alternative to CUDA… they're already shipping it, one production deployment at a time. The shift away from single-vendor lock-in is gaining real momentum, and innovators like Modular are rewriting the framework, shaping the infrastructure decisions that will determine how flexible platforms are tomorrow. As we evaluate what comes next for AI infrastructure, one thing is clear: fragmentation's days may be numbered.


TensorWave Cloud: Engineered for AI Velocity

TensorWave is the industry’s premier AMD-exclusive cloud provider, delivering memory optimized AI infrastructure for training, fine-tuning and inference. Our platform combines the raw computational power of AMD's latest GPU architectures with innovative liquid cooling technology and ultra-high-speed networking, creating an optimized environment for the most demanding AI workloads. From large language model training to real-time inference, TensorWave provides the performance, scalability, and cost-efficiency that modern AI initiatives demand.Ready to get started? Connect with a sales engineer.

Breaking Free from GPU Fragmentation

Product

Solutions

Resources

Company

Product

Resources

Solutions

Company