[Paper] Pruning as a Game: Equilibrium-Driven Sparsification of Neural Networks

Source: arXiv - 2512.22106v1

Overview

The paper “Pruning as a Game: Equilibrium‑Driven Sparsification of Neural Networks” reframes network pruning as a strategic game among the model’s own components (weights, neurons, filters). Instead of imposing sparsity with hand‑crafted importance scores or regularizers, the authors let sparsity emerge naturally when each component’s “participation level” reaches an equilibrium where staying active is no longer beneficial. This game‑theoretic lens yields a simple, interpretable pruning algorithm that matches the sparsity‑accuracy trade‑offs of state‑of‑the‑art methods.

Key Contributions

• Game‑theoretic formulation: Models parameter groups as players in a continuous non‑cooperative game, where each chooses a participation scalar.
• Equilibrium‑driven sparsity: Shows that at Nash equilibrium, dominated players (i.e., redundant parameters) collapse to zero without any external pruning rule.
• Simple algorithm: Derives an end‑to‑end training routine that jointly updates weights and participation variables, eliminating the need for separate importance scoring or post‑hoc thresholding.
• Theoretical insight: Provides provable conditions under which dominated players are guaranteed to be pruned, offering a principled explanation for why pruning works.
• Empirical validation: Demonstrates competitive sparsity‑accuracy curves on standard benchmarks (e.g., CIFAR‑10/100, ImageNet subsets) with a lightweight implementation.

Methodology

1. Players & Strategies – Each parameter group (e.g., a filter) is a player. Its strategy is a continuous scalar (p_i \in [0,1]) representing how much it participates in the forward pass.
2. Utility Function – The utility balances two terms:
   • Contribution: How much the player improves the loss (e.g., gradient‑based signal).
   • Cost: A penalty for redundancy/competition with other players, modeled as a smooth function of the collective participation vector (\mathbf{p}).
3. Equilibrium Computation – The authors derive the first‑order optimality condition for a Nash equilibrium, which leads to a closed‑form update for (p_i). When the marginal benefit of staying active falls below the cost, the optimal (p_i) becomes zero.
4. Training Loop – During each mini‑batch:
   • Forward pass uses the current participation mask (\mathbf{p}).
   • Backward pass updates both the raw weights and the participation scalars via gradient descent on the joint loss.
   • After convergence, any (p_i) that is exactly zero (or below a tiny epsilon) is permanently removed, yielding a sparse model.

The whole process is a single‑stage training routine—no separate “pre‑train → prune → fine‑tune” phases are required.

Results & Findings

• The equilibrium‑driven method consistently reaches 70‑80% sparsity with less than 1% absolute drop in accuracy.
• Compared to magnitude‑based pruning and L1‑regularization baselines, the proposed approach attains similar or better trade‑offs while using fewer hyper‑parameters (no need to tune pruning thresholds).
• Ablation studies show that the participation variables converge quickly (within a few epochs) and that the final sparsity pattern is stable across random seeds, indicating robustness.

Practical Implications

• One‑shot pruning: Developers can integrate the algorithm into the normal training pipeline, avoiding the cumbersome multi‑stage prune‑then‑fine‑tune workflow.
• Hardware‑friendly sparsity: Since participation scalars become exact zeros, the resulting masks are binary and can be directly leveraged by sparse matrix libraries or specialized accelerators.
• Interpretability: The participation values give a continuous importance score that is theoretically grounded, making it easier to audit which parts of a model are truly essential.
• Reduced hyper‑parameter burden: No need to manually set pruning ratios, thresholds, or schedule regularization strengths—equilibrium dynamics handle it automatically.
• Potential for adaptive inference: Because participation can be recomputed on‑the‑fly, one could envision dynamic sparsification where a model trims itself further at inference time based on resource constraints.

Limitations & Future Work

• Scale: Experiments are limited to moderate‑size models and datasets; the paper does not yet demonstrate performance on full‑scale ImageNet or transformer architectures.
• Computation overhead: Jointly optimizing participation variables adds a small constant factor to each training step, which may be noticeable for very large models.
• Game design choices: The specific form of the cost/competition term influences the equilibrium; exploring alternative utility functions could yield even better sparsity patterns.
• Extension to structured pruning: While the current formulation works for arbitrary parameter groups, applying it to more complex structures (e.g., entire attention heads) remains an open direction.

Overall, treating pruning as an equilibrium problem offers a fresh, theory‑backed avenue for building leaner neural networks—an approach that could streamline model deployment pipelines and inspire new research at the intersection of game theory and deep learning.

Authors

• Zubair Shah
• Noaman Khan

Paper Information

• arXiv ID: 2512.22106v1
• Categories: cs.AI
• Published: December 26, 2025
• PDF: Download PDF