[Paper] Symmetry in language statistics shapes the geometry of model representations

Source: arXiv - 2602.15029v1

Overview

The paper uncovers why large language models (LLMs) and word‑embedding systems often organize concepts into surprisingly clean geometric shapes—think of months forming a circle or cities lining up by latitude and longitude. The authors demonstrate that a simple translation symmetry hidden in natural‑language co‑occurrence statistics is the mathematical engine that sculpts these structures, and they prove that the effect persists even when the raw statistics are heavily perturbed.

Key Contributions

• Identifies a translation symmetry in language co‑occurrence data (e.g., the probability of seeing “January” and “March” together depends only on the two‑month interval, not on the absolute months).
• Provides a rigorous proof that this symmetry forces high‑dimensional embeddings to arrange themselves on low‑dimensional manifolds (circles, lines, planes).
• Shows robustness: the same geometric patterns survive drastic manipulations of the training corpus, suggesting the symmetry is an emergent property rather than a fragile artifact.
• Links symmetry to a latent continuous variable, explaining why moderate‑size embeddings (far from the infinite‑dimensional limit) already exhibit the predicted geometry.
• Validates theory across three families of models: classic word‑embedding algorithms (e.g., GloVe, word2vec), modern sentence‑embedding encoders, and full‑scale LLMs (GPT‑style transformers).

Methodology

1. Statistical analysis of corpora – The authors compute pairwise co‑occurrence matrices for tokens (months, cities, years, etc.) and verify that the entries depend primarily on a relative attribute (time gap, geographic distance).
2. Theoretical framework – Using tools from statistical physics and random matrix theory, they prove that when a co‑occurrence matrix possesses translation symmetry, its top eigenvectors span a low‑dimensional subspace that any optimal embedding (e.g., minimizing a typical skip‑gram loss) will inherit.
3. Perturbation experiments – They deliberately delete all sentences containing specific token pairs (e.g., any sentence that mentions two months together) and retrain embeddings to test whether the geometry collapses.
4. Empirical validation – The team trains multiple models (word2vec, GloVe, Sentence‑BERT, and GPT‑2/3‑scale) on both original and perturbed corpora, then visualizes the learned representations with PCA/t‑SNE and probes them with linear classifiers to recover known latent variables (month angle, latitude, etc.).

Results & Findings

• Circular arrangement of months emerges in every model tested, with the angular position correlating strongly (R > 0.95) with the calendar order.
• One‑dimensional manifolds for years appear even when year‑year co‑occurrences are removed, indicating the symmetry is encoded indirectly via other tokens.
• Geographic decoding: linear probes recover latitude and longitude of city names with < 5 % mean absolute error in embeddings of dimension as low as 128.
• Perturbation resilience: after deleting all direct month‑month co‑occurrences, the circular pattern persists (only a slight increase in angular noise), confirming that the symmetry is enforced by a broader latent structure.
• Latent variable model: fitting a simple continuous latent variable (e.g., “time of year”) to the co‑occurrence matrix reproduces the observed eigenvalue spectrum, supporting the authors’ hypothesis.

Practical Implications

• Feature engineering: Developers can exploit the inherent geometry of embeddings for downstream tasks (e.g., clustering temporal entities, geospatial reasoning) without additional supervision.
• Model interpretability: The symmetry framework offers a principled way to diagnose why certain concepts are linearly separable, aiding debugging of embedding‑based pipelines.
• Data augmentation: Knowing that the geometry is robust to missing co‑occurrences suggests that aggressive data pruning or privacy‑preserving token removal may not degrade semantic structure as much as feared.
• Efficient representation learning: Since moderate‑dimensional embeddings already capture the low‑dimensional manifolds, practitioners can safely reduce model size for edge devices while retaining interpretable structure.
• Prompt design for LLMs: Understanding that LLM internal states respect these symmetries can guide prompt engineering—for instance, framing temporal queries in a way that aligns with the model’s circular month representation.

Limitations & Future Work

• The theory assumes global translation symmetry; many real‑world corpora exhibit only approximate or locally varying symmetry, which may limit applicability to niche domains (e.g., code, legal text).
• Experiments focus on English‑centric datasets; cross‑lingual or low‑resource languages might display different symmetry patterns.
• The latent‑variable explanation is demonstrated with a single continuous factor; extending the framework to capture multiple interacting latent dimensions (e.g., time × topic) remains an open challenge.
• Future work could explore how training objectives (contrastive vs. autoregressive) modulate the strength of the induced geometry, and whether fine‑tuning on downstream tasks preserves or distorts the symmetry.

Authors

• Dhruva Karkada
• Daniel J. Korchinski
• Andres Nava
• Matthieu Wyart
• Yasaman Bahri

Paper Information

• arXiv ID: 2602.15029v1
• Categories: cs.LG, cond-mat.dis-nn, cs.CL
• Published: February 16, 2026
• PDF: Download PDF