PostHeaderSection

Symbol

Global NavBar

PostContentSection

Global Footer

Blog Detail Page

Beirit Harvey

See how TensorWave is improving GPU health checks, recovery, and developer tools as it builds toward ClusterMAX Platinum and a more reliable AMD cloud.

From Silver Toward Platinum: Building a Better AMD Cloud

A reliable AI cloud needs to handle the moments when something goes wrong. During TensorWave’s ClusterMAX 3.0 evaluation, SemiAnalysis put that ability to the test by deliberately introducing a failure. Our health checks caught it immediately, and the platform prompted the reviewers to replace the unhealthy node with a healthy one.

Health checks and recovery proved challenging for several providers in this round. For TensorWave, they were a strength: SemiAnalysis described our monitoring and autoremediation as “exemplary,” while recommending that recovery happen automatically without an approval click.

Behind that response was the team’s work to rebuild TensorWave’s GPU health checks and recovery workflows. We spoke with Larry Singh, founding engineer at TensorWave, about what changed, what the evaluation taught his team, and where they’re focusing next.

In our recap of the results, we shared the progress behind TensorWave’s Silver rating. Here, we take a closer look at the engineering work behind that progress and the next steps toward a more reliable experience for the teams building on our cloud.


What Happens When a GPU Node Fails?

That question became a major focus for the team preparing for this evaluation.

TensorWave already had GPU health checks, but Singh and the team saw room to make them more comprehensive. They rebuilt the checks from the ground up to cover more hardware failure scenarios, then developed a workflow for replacing unhealthy nodes.

The injected failure gave reviewers a chance to experience that workflow firsthand. The system detected the issue and offered a replacement, connecting a health alert to an action the customer could take.

When we asked Singh which improvement stood out most for customers, he pointed to this work. Broader checks and a clearer recovery process also gave the team a stronger foundation for operating at greater scale.

It’s the kind of engineering that becomes especially important when something goes wrong. A customer needs to know what happened and what they can do next. Making that process easier took a substantial rebuild behind the scenes.


A Spreadsheet With a Bigger Purpose

The plan started somewhere less dramatic: a shared spreadsheet.

The team broke down SemiAnalysis’s published evaluation criteria line by line. In weekly reviews, they worked through what TensorWave already supported, what needed attention, and which capabilities depended on further development in the AMD ecosystem.

The useful question was what those gaps meant for customers. Which changes would make the platform easier to use? Which would help the teams running workloads on it every day?

“We went through the criteria line by line, identified the gaps, and focused on improvements that would make the biggest difference for our customers while helping us move forward in the ratings.”

— Larry Singh, Founding Engineer at TensorWave

That overlap shaped the priorities. The evaluation supplied a detailed set of expectations; the team’s job was to turn them into improvements people could use.


From a Requirement to an Everyday Tool

The new command-line interface, or CLI, is a good example of an improvement that customers could use. 

TensorWave hadn’t offered one in the previous round. Building it addressed an evaluation requirement and gave customers another way to interact with the platform. It also found users inside TensorWave: networking and DevOps teams now use the CLI in their own work.

The collaboration extended to planning. Software engineering brought MLOps and DevOps into the process of reviewing requirements, choosing features, and figuring out how everything would fit together.

Singh described this as an important change from the previous round. More of the people responsible for building and operating the platform were involved in shaping its improvements. Working toward a better rating became a shared effort across teams.


Help Us Build Toward Platinum

Silver recognizes the progress we've made. Platinum is the goal we're working toward, and reaching it will take more of the work Singh described: understanding what users need, finding gaps, and building improvements that hold up in practice.

For developers, that's an opportunity to help shape an AMD-exclusive AI cloud as it grows. The work spans the tools customers use to interact with the platform and the systems that help our teams detect and respond to hardware problems. It calls for engineers who enjoy working across teams and following a problem from an initial requirement through to a working solution.

If those are the kinds of problems you want to solve, we'd love to build with you. Check out open positions

From Silver Toward Platinum: Building a Better AMD Cloud

Product

Solutions

Resources

Company

Product

Resources

Solutions

Company