[Paper] Shallow Neural Networks Learn Low-Degree Spherical Polynomials with Learnable Channel Attention

Source: arXiv - 2512.20562v1

Overview

This paper shows that a modestly‑sized two‑layer neural network equipped with learnable channel attention can learn low‑degree spherical polynomials far more efficiently than standard over‑parameterized nets. By carefully structuring the training into a channel‑selection phase followed by ordinary gradient descent, the authors achieve a sample‑complexity that scales as (n = \Theta(d^{\ell_0}/\varepsilon)), which matches the minimax optimal rate for this regression problem.

Key Contributions

• Channel‑attention architecture: Introduces a lightweight attention mechanism that selects a subset of first‑layer channels, reducing the effective model size to the true polynomial degree (\ell_0).
• Two‑stage training recipe:
  1. Stage 1 – a single GD step jointly updates both layers to discover the correct channel set.
  2. Stage 2 – standard GD fine‑tunes the second‑layer weights using only the selected channels.
• Improved sample complexity: Proves that the required number of training samples is (n = \Theta(d^{\ell_0}/\varepsilon)), a dramatic improvement over the classic bound (\Theta\big(d^{\ell_0}\max{\varepsilon^{-2},\log d}\big)).
• Minimax‑optimal risk: Shows the trained network attains a non‑parametric regression risk of (\Theta(d^{\ell_0}/n)), which is provably optimal for kernels of rank (\Theta(d^{\ell_0})).
• Width requirement: Demonstrates that a finite hidden width of (m \ge \Theta!\big(n^{4}\log(2n/\delta)/d^{2\ell_0}\big)) suffices, avoiding the need for extreme over‑parameterization.

Methodology

1. Problem setting – The target function is a spherical polynomial of constant degree (\ell_0) defined on the unit sphere in (\mathbb{R}^d).
2. Network design – A two‑layer fully‑connected net with ReLU‑like activations, but the first layer contains (L \ge \ell_0) channels (i.e., groups of neurons) that can be turned on/off by learnable attention weights.
3. Stage 1 (Channel selection) – Perform a single gradient‑descent step on both layers. The update is crafted so that the attention weights amplify the channels that align with the true polynomial basis and suppress the rest. A probabilistic analysis shows the correct (\ell_0) channels are identified with high probability.
4. Stage 2 (Fine‑tuning) – Freeze the attention mask (keeping only the selected channels) and continue ordinary GD on the second‑layer coefficients. This reduces the problem to linear regression in the span of the identified basis functions.
5. Theoretical analysis – The authors combine tools from random matrix theory, concentration inequalities, and classical non‑parametric regression to bound the excess risk and to prove the lower bound on sample complexity.

Results & Findings

The key takeaway is that once the right channels are selected, the network behaves like an optimal kernel estimator, and the extra attention mechanism incurs negligible overhead.

Practical Implications

• Efficient learning of structured signals – In domains where data lives on a sphere (e.g., 3‑D point clouds, directional statistics, geodesic embeddings), low‑degree spherical harmonics are natural bases. This work suggests a simple NN can discover those bases automatically, saving data collection costs.
• Model compression – The attention mask effectively prunes the network to the minimal number of channels needed, offering a principled way to compress over‑parameterized models without sacrificing statistical efficiency.
• Fast training pipelines – Only a single GD step is required for channel discovery, which can be implemented as a cheap “warm‑up” phase before the usual training loop. This is attractive for large‑scale pipelines where epoch‑level hyper‑parameter sweeps are expensive.
• Guidance for architecture search – The results provide a theoretical justification for adding lightweight attention modules to shallow nets when the target function is believed to have low intrinsic dimensionality.
• Potential for transfer learning – The selected channels constitute a reusable feature extractor for any downstream task that shares the same spherical polynomial structure.

Limitations & Future Work

• Constant‑degree assumption – The analysis holds for (\ell_0 = \Theta(1)). Extending to higher‑degree or data‑dependent degrees remains open.
• Spherical domain restriction – Real‑world data often deviates from the perfect unit‑sphere assumption; robustness to noise and manifold curvature is not addressed.
• Two‑layer focus – While the theory is clean for shallow nets, it is unclear how the channel‑attention mechanism scales to deep architectures.
• Empirical validation – The paper is primarily theoretical; practical experiments on point‑cloud or graphics datasets would strengthen the claims.
• Alternative attention designs – Exploring more expressive attention (e.g., multi‑head, softmax‑based) could further improve sample efficiency or enable learning of richer function classes.

Overall, the work bridges a gap between classical approximation theory and modern deep learning by showing that a modest attention tweak can make shallow nets statistically optimal for a well‑studied class of functions.

Authors

• Yingzhen Yang

Paper Information

• arXiv ID: 2512.20562v1
• Categories: stat.ML, cs.LG, math.OC
• Published: December 23, 2025
• PDF: Download PDF