[Paper] A Survey on Split Learning for LLM Fine-Tuning: Models, Systems, and Privacy Optimizations

Source: arXiv - 2604.24468v1

Overview

Fine‑tuning large language models (LLMs) is the key to turning generic AI into domain‑specific assistants, but the compute‑heavy process is out of reach for many smaller companies. Split learning—where a client runs the front part of a model and a server runs the back—offers a way to share the heavy lifting without exposing raw data. This survey is the first to map the rapidly growing literature on split‑learning‑based LLM fine‑tuning, organizing it along model, system, and privacy dimensions and pointing out where the field can go next.

Key Contributions

• Unified training pipeline – Introduces a fine‑grained, end‑to‑end workflow that isolates the “split points,” communication steps, and optimization knobs common to all split‑learning approaches.
• Three‑axis taxonomy – Systematically classifies 70+ recent works into:
  1. Model‑level optimizations (e.g., adapter insertion, low‑rank factorization, dynamic split selection).
  2. System‑level efficiency (e.g., bandwidth‑aware scheduling, GPU/CPU co‑placement, asynchronous pipelines).
  3. Privacy preservation (e.g., differential privacy, secure aggregation, gradient‑masking, attack‑defense benchmarks).
• Comparative analysis – Provides side‑by‑side tables that compare methods on metrics such as communication overhead, fine‑tuning speed, accuracy loss, and privacy guarantees.
• Benchmark suite recommendation – Suggests a set of open‑source datasets, model sizes, and evaluation protocols to make future research reproducible.
• Research roadmap – Highlights open challenges (e.g., cross‑device heterogeneity, composable privacy budgets) and promising directions (e.g., federated‑split hybrids, hardware‑aware split points).

Methodology

1. Literature collection – The authors performed a systematic search across major AI conferences and pre‑print servers, filtering for works that explicitly combine split learning with LLM fine‑tuning.
2. Pipeline abstraction – They broke down the fine‑tuning process into nine reusable components (model partitioning, forward/backward data exchange, loss computation, optimizer sync, etc.).
3. Taxonomy construction – Each paper was mapped onto the three axes by examining its primary contribution (e.g., a new adapter design → model‑level; a communication‑compression scheme → system‑level; a privacy‑preserving protocol → privacy).
4. Empirical cross‑comparison – Where possible, the survey aggregates reported numbers (e.g., FLOPs saved, latency reduction, privacy ε values) into comparative charts.
5. Critical synthesis – The authors discuss trade‑offs (e.g., higher privacy often means more communication) and identify gaps where no existing work currently operates.

Results & Findings

Overall, the survey finds that split learning can bring LLM fine‑tuning within reach of edge devices or small enterprises when combined with lightweight model adapters and smart system scheduling, while still offering provable privacy guarantees.

Practical Implications

• For SaaS AI platforms – Providers can expose a “split‑fine‑tune” API that runs the heavy transformer layers in the cloud while letting customers keep their proprietary data on‑premise. This reduces compliance risk (e.g., GDPR, HIPAA) without sacrificing model performance.
• For startups & SMEs – By adopting adapter‑based split learning, a team with a single GPU can fine‑tune a 70 B‑parameter LLM through a modest‑size server, cutting cloud compute bills by >70 %.
• Tooling impact – The taxonomy points to concrete building blocks (e.g., PyTorch Lightning split modules, gRPC‑based activation streaming) that can be packaged into open‑source libraries, accelerating adoption.
• Security posture – The privacy‑focused findings give product teams a checklist for threat modeling (e.g., activation inference attacks) and a set of mitigations (DP noise, secure aggregation) that can be integrated into CI pipelines.
• Hardware roadmap – System‑level insights suggest that future NICs with on‑the‑fly compression or programmable pipelines could further shrink the communication bottleneck, making split learning viable even over 4G/5G links.

Limitations & Future Work

• Scope limited to fine‑tuning – The survey does not cover split learning for pre‑training or inference‑time serving, which may have different constraints.
• Benchmark heterogeneity – Reported numbers come from disparate hardware setups and datasets, making apples‑to‑apples comparison imperfect.
• Privacy evaluation gaps – Most works evaluate privacy under a single threat model; broader adversarial settings (e.g., colluding servers) remain under‑explored.
• Future directions highlighted by the authors include:
  • Dynamic, data‑driven split‑point selection that adapts to network conditions.
  • Hybrid federated‑split frameworks that combine the strengths of both paradigms.
  • Standardized privacy accounting across multiple fine‑tuning rounds.
  • Co‑design of hardware accelerators and split‑learning protocols to minimize latency and energy consumption.

Authors

• Zihan Liu
• Yizhen Wang
• Rui Wang
• Xiu Tang
• Sai Wu

Paper Information

• arXiv ID: 2604.24468v1
• Categories: cs.CR, cs.CL, cs.DC, cs.LG
• Published: April 27, 2026
• PDF: Download PDF