Forget megabucks Nvidia GPUs, apparently all you need to run an LLM is a Pentium II CPU from 1997

The catch? It's kinda slow, as in 20,000 times slower than a GPU.

Conventional wisdom says you need a mountain of Nvidia GPUs at about $50,000 a pop to have a chance of running the latest AI models. But apparently not. EXO Labs (via Indian Defence Review) claims to have got the Llama 2 LLM up and running on a Windows 98 box circa 1997 courtesy of a mere Pentium II processor. Hurrah! The catch? It’s running about 20,000 times slower than on a modern GPU. Haroo.

Apparently, Exo Labs picked up the machine for just under $120 on eBay, following which perhaps the biggest headache was getting peripherals to work, what with the legacy PS2 ports and just a single USB input.

Indeed, getting the required files onto the machine was a serious headache. Then there was compiling the files in a format which was compatible with the Pentium II’s ancient instruction set.

Anywho, with the code and hardware sorted, it was time to run Llama 2. Reportedly, the 260K parameter version of the model achieved 39.31 tokens per second on the Pentium II, while the larger 15M parameter version hit just 1.03 tokens per second.

They even tried a partial data model run using a one billion parameter version of Llama 3.2 which returned a glacial 0.0093 tokens per second. To put that into context, there are references to the one billion parameter 3.2 model hitting 40 tokens per second on Arm CPUs and 200 tokens per second on GPUs.

In other words, it’s running about 20,000 times slower on the Pentium II. But hey, it’s running. The comparison isn’t perfect, there are all kinds of variables in terms of how the models are set up. But that 20,000 times figure probably gives the right idea in terms of the performance delta in rough order of magnitude terms.

Indeed, while it’s impressive to get a modern LLM running on such an old CPU, the performance gap is a reminder that speed matters. Actually, it’s a bit like 3D gaming.

Compiled correctly, you’d could no doubt get Cyberpunk 2077 running in full path-traced mode on a Pentium II at 4K. But you’d probably be looking a frame rate similar to the P II’s 0.0093 tokens per second performance. At which point, it’s all a bit academic.

But maybe it would be fun to watch the pixels being rendered, one-by-one. On the other hand, completing a benchmark run could take years. Perhaps we’ll leave all that, for now.