Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPU

Source: Hacker News

Question

Hi everyone, I’m kinda involved in some retrogaming and with some experiments I ran into the following question: “It would be possible to run transformer models bypassing the CPU/RAM, connecting the GPU to the NVMe?”

Solution Overview

This is the result of that question itself and some weekend vibecoding (the linked library repository is in the README as well). It seems to work, even on consumer GPUs, and should work even better on professional ones.

Further Reading

• Comments on Hacker News — 233 points, 55 comments