Are GPU’s really a critical resource in an AI-powered world?
I’m here at #SC26 this week in Chicago and one insight doesn’t bode well for #GPU vendors like NVIDIA. The newest number one #supercomputer on the #top500 list doesn’t use GPU’s despite being designed for AI HPC workloads. When one really optimizes HPC architecture for AI, one simply doesn’t want to use 64 digit floating point precision (FP64) operations – the kind of math functionality that classic GPU’s were built to offload from core processing.
When you optimize code for AI processing, 4, 6 or 8 digit precision (FP4/6/8) is plenty because AI model learning as embedded in node weights isn’t going to be any more accurate at higher precisions. And when you look at the difference in impact on memory, core processing, data storage and transfer bandwidth, a system can handle a lot more FP4 numbers for the cost of one FP64 number. This gain may not be fully 16 times better, although in some cases it could conceivable be even more if you also custom build an FP4 sized pipeline/cache/etc.
Which is exactly what a the big AI hyperscalars are all doing with their custom AI accelerators. For example,Microsoft is working on Maia, Google has TPUs, AWS has Trainium, and Meta has MTIA. These aren’t actually GPU’s, but accelerators in a more data handling sense. In fact, what I learned today was that AI workloads benefit from more cores and vector functionality, matrix engine units, optimized memory structures (HBM and Fast RAM) and the like. And all these companies are building their own custom hardware to provide those things.
Now we could next examine in detail the impact of the big players going off and building their own non-standard architectures, but even that may not matter much in the long run. The number one new supercomputer running at exascale (China’s LineShine) is reportedly just CPU without either GPUs or accelerators. Who would have thought that commodity infrastructure deployed cheaply at great scale would beat closed system customized hardware?
So will this burst an AI bubble? No, as it actually enables AI tokens to become more affordable and perhaps more democratically distributed. But in the near term this idea that you don’t really need GPUs for AI may very well shake up who all is seen as the current AI vendor darlings.
