[Paper] Pruning as Evolution: Emergent Sparsity Through Selection Dynamics in Neural Networks

Source: arXiv - 2601.10765v1

Overview

The paper “Pruning as Evolution: Emergent Sparsity Through Selection Dynamics in Neural Networks” reframes network pruning as a natural, continuous selection process rather than a post‑hoc, rule‑based cleanup step. By treating groups of parameters (neurons, filters, attention heads, etc.) as evolving populations, the authors show that sparsity can emerge organically during standard gradient training—opening a path toward lighter models without dedicated pruning schedules.

Key Contributions

• Evolutionary framing of pruning – Introduces a formal model where each parameter group has a “population mass” that evolves under selection pressure derived from local learning signals.
• Continuous selection dynamics – Derives differential equations governing mass evolution, eliminating the need for discrete pruning events or external importance metrics.
• Empirical validation on a scaled MLP – Demonstrates that the evolutionary process reproduces dense‑model accuracy (≈98 % on MNIST) and yields predictable accuracy‑sparsity trade‑offs when hard‑pruned after training.
• Sparsity without explicit schedules – Shows that a simple training loop can produce 35–50 % sparsity automatically, simplifying pipelines that currently require multi‑stage pruning‑retraining loops.

Methodology

1. Population definition – The network is partitioned into populations (e.g., each hidden neuron). Each population (i) carries a scalar mass (m_i) that scales its output contribution.

2. Fitness estimation – During back‑propagation, the gradient of the loss w.r.t. a population’s output serves as a proxy for its fitness: higher gradient magnitude → higher fitness, indicating that the population is currently useful for reducing loss.

3. Selection dynamics – The authors adopt a replicator‑type differential equation:

   $$\dot{m}_i = m_i \bigl( f_i - \bar{f} \bigr)$$

   where (f_i) is the fitness of population (i) and (\bar{f}) is the average fitness across all populations. Populations with below‑average fitness shrink, while high‑fitness ones grow.

4. Mass normalization – To keep the total capacity bounded, masses are periodically renormalized (e.g., L1‑norm constraint), ensuring the network does not simply inflate all masses.

5. Hard pruning – After training, any population whose mass falls below a small threshold is removed, yielding a sparse architecture. No extra pruning epochs or mask‑learning phases are required.

The entire process plugs into a standard training loop: compute forward pass, back‑propagate, update weights, compute fitness, update masses, renormalize, repeat.

Results & Findings

Key takeaways

• The evolutionary dynamics naturally drive many neurons toward negligible mass, making them easy to prune.
• Accuracy degrades gracefully as sparsity increases, mirroring classic pruning curves but without any explicit pruning schedule.
• Different variants of the selection dynamics (e.g., alternative fitness definitions) produce slightly different sparsity‑accuracy curves, suggesting a tunable “selection pressure” knob for developers.

Practical Implications

• Simplified pipelines – Teams can drop the multi‑stage prune‑retrain‑fine‑tune workflow. A single training run already yields a ready‑to‑prune model.
• Dynamic model sizing – By adjusting the mass‑renormalization strength or the fitness scaling factor, developers can steer the model toward a desired size on‑the‑fly, useful for edge‑device deployment where memory budgets vary.
• Hardware‑aware training – Since the method works at the granularity of neurons/filters, it aligns well with structured sparsity that modern accelerators (e.g., NVIDIA Ampere’s sparse tensor cores, Intel’s DL Boost) can exploit without costly unstructured mask handling.
• Potential for continual learning – The population view naturally accommodates adding new neurons (mass injection) or removing stale ones, offering a framework for models that must adapt over time without full retraining.
• Reduced hyper‑parameter burden – No need to tune pruning thresholds, schedule epochs, or regularization weights dedicated to sparsity; the only new knobs are the fitness‑to‑mass mapping and renormalization rate.

Limitations & Future Work

• Scale of experiments – Validation is limited to a modest MLP on MNIST; behavior on large CNNs, Transformers, or language models remains untested.
• Fitness proxy simplicity – Using raw gradient magnitude may be noisy for deeper networks; more robust fitness estimators (e.g., moving averages, second‑order information) could improve stability.
• Hard pruning threshold – The final cut‑off is still a manual hyper‑parameter; automating its selection (e.g., via a target mass budget) is an open question.
• Interaction with other regularizers – How the evolutionary dynamics coexist with dropout, batch norm, or weight decay needs systematic study.

The authors suggest extending the framework to structured pruning of convolutional filters and attention heads, exploring adaptive selection pressures, and integrating the approach into large‑scale training libraries (e.g., PyTorch Lightning, TensorFlow Keras).

Authors

• Zubair Shah
• Noaman Khan

Paper Information

• arXiv ID: 2601.10765v1
• Categories: cs.NE
• Published: January 14, 2026
• PDF: Download PDF