[Paper] Symmetry-Protected Lyapunov Neutral Modes in Equivariant Recurrent Networks

Source: arXiv - 2605.03338v1

Overview

This paper tackles a subtle but crucial problem for recurrent neural networks (RNNs) that need to remember continuous quantities—think of a robot tracking its heading, a physics simulator preserving phase, or a language model maintaining a latent “position” in a sequence. The authors show that when the network’s dynamics are exactly equivariant under a symmetry group (e.g., rotations, translations, or more exotic Lie groups), the system automatically inherits neutral (zero‑Lyapunov) directions that stay perfectly stable over arbitrarily long horizons. In other words, the symmetry itself guarantees a built‑in memory channel, without any delicate tuning.

Key Contributions

• Theoretical guarantee: Proves that any compact invariant set of an equivariant (C^1) vector field carries at least (\dim(G/H)) zero Lyapunov exponents tangent to the group orbit, where (G) is the symmetry group and (H) its stabilizer.
• Symmetry‑protected memory: Introduces the notion of symmetry‑protected Lyapunov neutral modes—directions that remain exactly neutral as long as equivariance holds.
• Controlled symmetry breaking: Demonstrates that breaking equivariance creates a pseudo‑gap in the Lyapunov spectrum, which directly predicts the finite memory lifetime of the network.
• Extensive empirical validation: Tests the theory on a wide variety of groups ((S^1), tori (T^q), (SO(n)), (U(m)), product groups) and on coupled RNN‑style systems, confirming orbit‑dimension scaling and tangent‑subspace alignment.
• Practical RNN design: Trains an exactly equivariant recurrent cell for a velocity‑input (S^1) path‑integration task, achieving near‑perfect step equivariance ((3.2\times10^{-8}) error) and superior horizon, speed, and phase‑generalization compared to GRU, LSTM, and orthogonal‑RNN baselines.

Methodology

1. Equivariant dynamical systems framework – The authors model an RNN as a continuous‑time autonomous vector field (f:\mathbb{R}^n\to\mathbb{R}^n) that satisfies (f(g\cdot x)=g\cdot f(x)) for all (g) in a Lie group (G).
2. Lyapunov analysis on group orbits – By examining the linearized dynamics (the Jacobian) along trajectories that lie on a compact invariant set, they prove that the tangent space to the group orbit is an invariant subspace with eigenvalue zero, yielding the neutral modes.
3. Symmetry breaking experiments – They introduce a controlled perturbation that slightly violates equivariance, then measure the resulting pseudo‑gap (small non‑zero Lyapunov exponent) and correlate it with the observed memory decay.
4. Numerical diagnostics – The paper uses several complementary metrics:
   • Normalized equivariance error (how far the learned dynamics deviate from exact symmetry)
   • Direct computation of group‑tangent Lyapunov exponents
   • Principal‑angle alignment between learned tangent subspaces and true group orbits
   • Autonomous‑flow zero‑input controls to isolate the neutral directions.
5. Task‑level validation – An equivariant recurrent cell is trained on a synthetic path‑integration problem (integrating angular velocity on a circle). The cell’s performance is benchmarked against standard RNN variants under identical training regimes.

Results & Findings

These results collectively validate the central claim: exact equivariance automatically protects certain directions from exponential divergence, giving RNNs a built‑in, mathematically guaranteed memory channel. When equivariance is imperfect, the size of the induced pseudo‑gap predicts how quickly that memory degrades.

Practical Implications

• Design of long‑term memory RNNs: Embedding the appropriate symmetry (e.g., rotation for heading, translation for position) directly into the architecture yields stable memory without ad‑hoc tricks like gating or orthogonal initialization.
• Robotics & control: Systems that must integrate sensor streams (odometry, inertial measurements) can benefit from equivariant recurrent cells that guarantee drift‑free integration over long horizons.
• Physics‑informed ML: Simulators that need to conserve quantities (angular momentum, phase) can encode the corresponding Lie group symmetry, ensuring that learned dynamics respect conservation laws at the level of Lyapunov spectra.
• Efficient training: The equivariant cell achieves better generalization with fewer parameters and less training time than standard GRU/LSTM baselines, potentially reducing compute costs for sequence models.
• Robustness to perturbations: The pseudo‑gap analysis offers a diagnostic tool: measuring deviation from exact equivariance predicts memory reliability, enabling runtime monitoring or adaptive correction.

Limitations & Future Work

• Exact equivariance requirement: Guarantees hold only when the network’s dynamics are perfectly equivariant. Numerical errors, discretization, or noisy data can introduce small violations that degrade memory.
• Finite‑dimensional focus: Theory is developed for finite‑dimensional autonomous vector fields; extending results to discrete‑time RNNs with stochastic inputs remains open.
• Scalability to large groups: While several classic Lie groups are covered, handling very high‑dimensional or non‑compact groups (e.g., affine transformations) may pose computational challenges.
• Real‑world benchmarks: Empirical validation is limited to synthetic path‑integration tasks. Applying equivariant recurrent cells to large‑scale problems (e.g., video prediction, language modeling) would test practical limits.
• Learning the symmetry: Future work could explore methods for discovering the appropriate symmetry from data, rather than hand‑specifying it, making the technique more broadly applicable.

Authors

• Hanson Hanxuan Mo

Paper Information

• arXiv ID: 2605.03338v1
• Categories: cs.NE, math.DS
• Published: May 5, 2026
• PDF: Download PDF