[Paper] An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU

Source: arXiv - 2603.16428v1

Overview

Fine‑tuning today’s massive language models still feels like a luxury reserved for organizations with multi‑GPU clusters. The new system SlideFormer shows that, with clever co‑design of CPU, GPU, and storage, you can fine‑tune 100‑plus‑billion‑parameter models on a single consumer‑grade GPU (e.g., RTX 4090). The paper demonstrates up to 6× larger model support and 8× larger batch sizes without sacrificing speed, opening the door for a much broader community of developers to customize LLMs locally.

Key Contributions

• Sliding‑window GPU engine – treats GPU memory as a moving window, overlapping GPU compute with CPU‑side parameter updates and multi‑tier I/O to keep the device busy.
• Heterogeneous memory management – a unified scheme that dynamically shuttles tensors between GPU, CPU RAM, and fast NVMe, cutting peak GPU memory roughly in half.
• Optimized Triton kernels – custom kernels that accelerate the most memory‑bound operations (e.g., gradient accumulation, optimizer steps) and integrate advanced prefetching.
• Single‑GPU fine‑tuning of 123B+ models – demonstrates practical fine‑tuning of state‑of‑the‑art LLMs on a single RTX 4090 (and comparable AMD GPUs).
• Performance gains – 1.4×–6.3× higher throughput vs. existing single‑GPU baselines, while keeping CPU/GPU memory usage low and maintaining >95 % of theoretical hardware utilization.

Methodology

1. Sliding‑window abstraction – Instead of loading the entire model into GPU memory, SlideFormer partitions the model into “chunks.” While one chunk is being processed on the GPU, the next chunk is streamed in from CPU RAM or NVMe, and the previous chunk’s gradients are offloaded for optimizer updates. This pipelining hides data‑transfer latency.
2. Asynchronous engine – A lightweight scheduler runs on the CPU, issuing non‑blocking copy commands and kernel launches. It balances three streams: (a) forward/backward compute, (b) optimizer‑step updates, and (c) I/O prefetch/post‑fetch.
3. Heterogeneous memory pool – The system maintains a global pool that tracks where each tensor lives (GPU, host RAM, or NVMe). When memory pressure spikes, less‑used tensors are evicted to the next slower tier, and hot tensors are promoted back.
4. Triton‑based kernels – The authors rewrote critical kernels (e.g., fused Adam, gradient accumulation) in Triton, allowing fine‑grained control over thread block sizes and memory access patterns, which yields up to 2× speedups on those ops.
5. Evaluation harness – Experiments were run on RTX 4090 (NVIDIA) and Radeon 7900 XTX (AMD) across models ranging from 7B to 123B parameters, measuring throughput (tokens/second), peak memory, and hardware utilization.

Results & Findings

• Memory reduction: Peak GPU memory shrank by ~45‑55 % across all model sizes; CPU RAM usage also dropped by ~30 % thanks to smarter eviction.
• Throughput: For the largest 123B model, SlideFormer achieved 5.6 tps on a single RTX 4090, a regime where prior tools would either crash or require multi‑GPU sharding.
• Utilization: Both NVIDIA and AMD GPUs stayed above 95 % of their theoretical FLOP capacity, indicating the asynchronous pipeline kept the hardware busy.
• Batch size scaling: The system supported batch sizes up to 8× larger than baseline, which is crucial for downstream tasks that benefit from larger mini‑batches (e.g., instruction‑tuning).

Practical Implications

• Democratized fine‑tuning: Small AI labs, startups, or even solo developers can now adapt cutting‑edge LLMs without investing in expensive GPU clusters.
• Cost‑effective prototyping: Training runs that previously required multi‑node setups can be executed on a single workstation, cutting cloud‑GPU expenses dramatically.
• Edge‑centric workflows: The heterogeneous memory scheme can be adapted for on‑device inference where RAM is limited, enabling larger models on consumer hardware.
• Integration path: Since SlideFormer builds on existing PyTorch/Triton stacks, developers can plug it into familiar pipelines (e.g., Hugging Face Trainer) with minimal code changes.
• Accelerated research cycles: Faster iteration on domain‑specific fine‑tuning means quicker feedback loops for applications like legal document analysis, biomedical text generation, or customized chat assistants.

Limitations & Future Work

• I/O bottleneck on slower storage: The system’s performance degrades if the underlying NVMe drive cannot sustain the required throughput; high‑end SSDs are still a prerequisite.
• Model‑specific tuning: Some kernel optimizations are tuned for transformer‑style LLMs; extending to other architectures (e.g., diffusion models) may need additional work.
• Scalability ceiling: While SlideFormer pushes the single‑GPU limit, models beyond ~150B parameters still exceed the memory budget even with aggressive paging.
• Future directions: The authors suggest exploring compression techniques (e.g., quantization, activation checkpointing) in tandem with the sliding window, and adding support for distributed‑CPU offloading to further stretch a single GPU’s capabilities.

Authors

• Ruijia Yang
• Zeyi Wen

Paper Information

• arXiv ID: 2603.16428v1
• Categories: cs.DC, cs.AI
• Published: March 17, 2026
• PDF: Download PDF