Micron Reportedly Explores Near-GPU NAND Flash to Run Bigger LLMs

Micron is reportedly exploring the idea of developing high-endurance NAND Flash modules that would be positioned closer to the GPU. The company is said to be investigating ways to bring NAND Flash closer to handle workloads that don't require the latency and bandwidth of traditional HBM or regular DRAM modules. This architecture, currently referred to as "near-GPU NAND," would serve as a middle ground between density and durability. This concept would allow GPUs to access a pool of NAND Flash that has a lower density than conventional TLC or QLC NAND, but instead focuses on improving I/O speeds, bandwidth, and read times. It would sit between GPU memory and regular system storage, acting as a very fast buffer, which would be ideal for the inference of massive LLMs.