[Paper] Drawback of Enforcing Equivariance and its Compensation via the Lens of Expressive Power

Source: arXiv - 2512.09673v1

Overview

Equivariant neural networks—models that respect symmetry transformations such as rotations or permutations—have become a go‑to tool for tasks ranging from 3‑D vision to graph learning. This paper zeroes in on a fundamental question: does forcing a network to be equivariant hurt its ability to represent complex functions? By dissecting two‑layer ReLU nets, the authors show that equivariance can indeed curtail expressive power, but that the loss can be recovered by scaling up the network—while still enjoying better generalization.

Key Contributions

• Theoretical proof of expressivity loss: Constructs a concrete example where a strictly equivariant 2‑layer ReLU network cannot represent a function that a non‑equivariant counterpart can.
• Compensation via model size: Demonstrates that increasing the number of hidden units (or channels) restores the missing expressive capacity.
• Complexity analysis: Shows that even after enlarging the network, the hypothesis space of the equivariant model has lower Rademacher complexity than an unrestricted network of comparable size, hinting at improved generalization.
• Layer‑wise equivariance vs. global equivariance: Provides a nuanced comparison, revealing that enforcing equivariance at each layer can be more restrictive than only at the output.
• Practical guidelines: Offers a rule‑of‑thumb for how many extra hidden units are needed to offset the expressivity penalty for common symmetry groups (e.g., cyclic, permutation).

Methodology

1. Model setting: The authors focus on the simplest yet expressive class—2‑layer fully‑connected ReLU networks. Each hidden unit is defined by a weight vector (the “channel”) and a bias; the output is a linear combination of ReLU activations.
2. Equivariance formalism: For a symmetry group (G) acting on the input space, a network (f) is equivariant if (f(g\cdot x)=g\cdot f(x)) for all (g\in G). The paper studies two enforcement strategies:
   • Global equivariance: the whole network satisfies the condition.
   • Layer‑wise equivariance: each linear layer is constrained to commute with the group action.
3. Expressivity analysis: By examining the arrangement of decision boundaries (the hyperplanes where ReLU units switch on/off) and the orientation of channel vectors, the authors construct a function that cannot be realized under the equivariance constraints unless the network is enlarged.
4. Compensation proof: They then prove that adding a factor of (|G|) (the size of the symmetry group) hidden units suffices to replicate any function that the unrestricted network can express.
5. Complexity bound: Using Rademacher complexity tools, they compare the capacity of the enlarged equivariant network to that of a standard network with the same number of parameters, showing the former is statistically “simpler.”

Results & Findings

Takeaway: Enforcing equivariance can double (or more) the hidden‑unit budget required for a given task, but the resulting model is statistically “tamer,” which often translates to better performance on limited data or noisy environments.

Practical Implications

• Model sizing for symmetry‑aware architectures: When designing equivariant CNNs, GNNs, or transformer variants that respect permutation/rotation invariance, allocate roughly (|G|) times more hidden channels than you would for a vanilla model.
• Resource‑efficient generalization: Even with the larger parameter count, the reduced complexity means you can often train with fewer epochs or smaller datasets while still achieving comparable accuracy.
• Hardware considerations: The extra channels are structured—they often share weights across symmetry orbits—so memory overhead can be mitigated with weight‑tying or grouped convolutions.
• Debugging expressivity bottlenecks: If an equivariant model stalls early in training, the paper’s analysis suggests checking whether the hidden‑unit count is sufficient relative to the symmetry group size.
• Transfer learning: Pre‑training a large equivariant backbone and fine‑tuning on a downstream task may yield better out‑of‑distribution robustness than a similarly sized non‑equivariant model.

Limitations & Future Work

• Scope limited to 2‑layer ReLU nets: While the insights likely extend to deeper architectures, formal proofs for multi‑layer or non‑linear activations (e.g., Swish, GELU) are missing.
• Assumes exact symmetry: Real‑world data often only approximately respects a group; the impact of soft equivariance constraints remains unexplored.
• Empirical validation on large‑scale benchmarks: The paper validates the theory on synthetic tasks; applying the findings to ImageNet‑scale vision models or massive graph datasets is an open direction.
• Automated sizing heuristics: Future work could integrate the derived scaling rule into architecture search tools that automatically balance expressivity and complexity for a given symmetry group.

Authors

• Yuzhu Chen
• Tian Qin
• Xinmei Tian
• Fengxiang He
• Dacheng Tao

Paper Information

• arXiv ID: 2512.09673v1
• Categories: cs.LG, cs.AI, cs.NE, stat.ML
• Published: December 10, 2025
• PDF: Download PDF