[Paper] nD-RoPE: A Generalized RoPE for n-Dimensional Position Embedding
Source: arXiv - 2606.12146v1
Overview
Rotary Position Embedding (RoPE) is widely adopted in Transformer models, yet its extension to high-dimensional domains lacks a unified theoretical formulation. Most existing approaches either apply rotations independently along each axis or empirically mix frequencies, which limits cross-dimensional interactions and yields direction-dependent representations. To address these limitations, we propose nD-RoPE, a decomposition-free generalization of RoPE to arbitrary dimensions. From a translation-invariant formulation in continuous Hilbert space, we derive a spectral condition for isotropy that requires treating positions and frequencies as coupled (n)-dimensional vectors. We instantiate this formulation with a multi-scale regular-simplex wave-vector design, which provides non-degenerate spatial coverage and a symmetric, directionally balanced second-order response. Experiments across images, videos, and point clouds demonstrate consistent performance gains and improved generalization in high-dimensional settings.
Key Contributions
This paper presents research in the following areas:
- cs.LG
- cs.AI
Methodology
Please refer to the full paper for detailed methodology.
Practical Implications
This research contributes to the advancement of cs.LG.
Authors
- Boyang Li
- Yulin Wu
- Sizhe Xu
- Nuoxian Huang
- Zhonghang Yuan
- Shangyi Guo
- Shu Yang
- Takahiro Yabe
Paper Information
- arXiv ID: 2606.12146v1
- Categories: cs.LG, cs.AI
- Published: June 10, 2026
- PDF: Download PDF