PostHeaderSection

Symbol

Global NavBar

PostContentSection

Global Footer

Blog Detail Page

Ryder Hankins

At Beyond Summit 2026, Makora co-founder and CEO Waleed Atallah laid out a five-layer framework for achieving 100x performance improvements in AI deployment, and why the answer isn't more engineers. It's better search.

Stop Designing AI Systems. Start Searching for Them.

At Beyond Summit 2026, Makora co-founder and CEO Waleed Atallah laid out a five-layer framework for achieving 100x performance improvements in AI deployment, and why the answer isn't more engineers. It's better search.

Twelve months ago at GTC, Jensen Huang put up a slide crediting nearly 30 years of CUDA library development as the treasure of NVIDIA's company. At this year's GTC, that slide was gone. 

The reason, as Atallah explained at Beyond Summit 2026, is straightforward: generating kernel code has changed forever.

Kernel engineers were rare, expensive, and mostly already working at NVIDIA or OpenAI. That bottleneck has now largely been resolved, through domain-specific languages like Triton and Mojo, compiler-based approaches, and most significantly, coding agents. With that barrier lowered, Atallah argued that AI deployment has become something different entirely. It is now a systems problem.


Five Layers, One Search Problem

Makora builds tooling to automate AI performance engineering end-to-end. To frame the opportunity, Atallah broke the optimization surface into five distinct layers, each compounding on the last.

The foundation is hardware. Atallah was direct: the future of AI hardware is heterogeneous, and that is no longer a prediction; it is already happening. Different parts of an AI pipeline are best suited to different specialized chips, and getting the combination right is one of the highest-leverage paths to large performance gains.

Above that sits kernel optimization. Makora's automated kernel generation has produced results across a range of hardware targets over the past year: generated primitive ops running two to five times faster on Tenstorrent, three times faster FlashAttention on Qualcomm, and GEMM kernels beating cuBLAS on NVIDIA H100 in certain configurations.

The third layer is the serving engine. vLLM, SGLang, TRT-LLM, Dynamo; tuning at this level alone, Atallah said, yields another 30 to 40% performance improvement. The fourth is the algorithm itself, where Makora has, since Atallah’s Beyond Summit session, published a new technique they are calling Sequential Monte Carlo Speculative Decoding. The fifth is co-design across all of the above simultaneously.


A New Speculative Decoding Technique Built for Heterogeneous Hardware

The algorithmic layer is where Atallah spent the most time. Makora's forthcoming Sequential Monte Carlo Speculative Decoding method replaces the exact inference used in standard speculative decoding with a probabilistic algorithm designed specifically for disaggregated inference settings. In practice, this means the draft model and the decode model can run on entirely different hardware architectures; a small speculator running on a chip optimized for high token throughput, feeding into a larger decode model running elsewhere.

The utilization result stands out. Because of how asynchronously the speculator and decode model operate, Makora's method achieves 99% GPU utilization. Conventional speculative decoding approaches typically land between 40 and 60%.

"Because of the nature of how asynchronous the speculator model and the decode model is, we actually achieve 99% GPU utilization… In our testing versus other speculative decoding methods, we see something like 40 to 60% utilization"


Why the Real Gains Come From Co-Design

Atallah was clear that the five layers are not meant to be optimized in isolation. The compounding effect of getting all of them right together is where the 100x figure comes from. A 20% gain is good. A 2x gain is worth celebrating. But reaching 100x requires treating the entire stack as a single search space and navigating it systematically.

The challenge is that for humans, the search space is simply too large. No team has the bandwidth or breadth of expertise to exhaustively explore every hardware combination, kernel configuration, serving setting, and algorithmic variant simultaneously. Makora's answer is a generative AI-based evolutionary search algorithm that operates across all five layers at once, identifies high-leverage optimization candidates, tests them, combines winners, and iterates.

"Pretty soon we won't design systems at all. We will simply search for them."


What's Next

Makora published the Sequential Monte Carlo Speculative Decoding paper last week. The evolutionary search algorithm is already in use across production deployments, with results across AMD, NVIDIA, Google TPU, Qualcomm, and Tenstorrent hardware. As the heterogeneous hardware landscape continues to expand, the search space only grows, and so does the ceiling for what automated optimization can find within it.


Go Deeper

Watch Waleed Atallah's full session here:


TensorWave Cloud: Engineered for AI Velocity

TensorWave is the industry’s premier AMD-exclusive cloud provider, delivering memory optimized AI infrastructure for training, fine-tuning and inference. 

Our platform combines the raw computational power of AMD's latest GPU architectures with innovative liquid cooling technology and ultra-high-speed networking, creating an optimized environment for the most demanding AI workloads. From large language model training to real-time inference, TensorWave provides the performance, scalability, and cost-efficiency that modern AI initiatives demand.

Ready to get started? Connect with a sales engineer.

Stop Designing AI Systems. Start Searching for Them.

Product

Solutions

Resources

Company

Product

Resources

Solutions

Company