Learning Pseudorandom Numbers with Transformers
Source: Hacker News
Abstract
We study the ability of Transformer models to learn sequences generated by Permuted Congruential Generators (PCGs), a widely used family of pseudo‑random number generators (PRNGs). PCGs introduce substantial additional difficulty over linear congruential generators (LCGs) by applying a series of bit‑wise shifts, XORs, rotations and truncations to the hidden state. We show that Transformers can nevertheless successfully perform in‑context prediction on unseen sequences from diverse PCG variants, in tasks that are beyond published classical attacks.
In our experiments we scale moduli up to $2^{22}$ using up to 50 million model parameters and datasets with up to 5 billion tokens. Surprisingly, even when the output is truncated to a single bit, it can be reliably predicted by the model. When multiple distinct PRNGs are presented together during training, the model can jointly learn them, identifying structures from different permutations.
We demonstrate a scaling law with modulus $m$: the number of in‑context sequence elements required for near‑perfect prediction grows as $\sqrt{m}$. For larger moduli, optimization enters extended stagnation phases; learning moduli $m \ge 2^{20}$ requires incorporating training data from smaller moduli, demonstrating a critical necessity for curriculum learning.
Finally, we analyze embedding layers and uncover a novel clustering phenomenon: the top principal components spontaneously group the integer inputs into bitwise rotationally‑invariant clusters, revealing how representations can transfer from smaller to larger moduli.
Comments
- 10 + 13 pages
- 8 + 21 figures
Subjects
- Machine Learning (cs.LG)
- Disordered Systems and Neural Networks (cond-mat.dis-nn)
- Cryptography and Security (cs.CR)
Citation
Cite as: arXiv:2510.26792 (cs.LG)
or for this version: arXiv:2510.26792v2 (cs.LG)
DOI
https://doi.org/10.48550/arXiv.2510.26792 – arXiv‑issued DOI via DataCite
Submission history
- v1 – Thu, 30 Oct 2025 17:59:09 UTC (12,235 KB) – submitted by Tao Tao
- v2 – Mon, 16 Feb 2026 23:41:23 UTC (17,937 KB)