[Paper] Hierarchical temporal receptive windows and zero-shot timescale generalization in biologically constrained scale-invariant deep networks

Source: arXiv - 2601.02618v1

Overview

This paper shows how a biologically‑inspired neural architecture—built around scale‑invariant “time cells” found in the hippocampus—can automatically develop a hierarchy of temporal receptive windows (TRWs) and, remarkably, generalize to completely new timescales without additional training. By training such networks on a language‑classification task that mirrors the nested structure of language (letters → words → sentences), the authors demonstrate faster learning, far fewer parameters, and zero‑shot timescale generalization compared with conventional recurrent models.

Key Contributions

• Emergent TRW hierarchy: Even when each layer shares the same distribution of time constants, a feed‑forward model (SITHCon) spontaneously forms increasingly long temporal windows across depth, mirroring cortical TRW hierarchies.
• Scale‑invariant recurrent design (SITH‑RNN): Introduces a recurrent architecture that embeds hippocampal‑like time‑cell dynamics, providing a built‑in prior for “what happened when.”
• Parameter efficiency: Across a spectrum of RNN variants, SITH‑RNN learns the same task with orders‑of‑magnitude fewer trainable parameters.
• Zero‑shot timescale generalization: After training on a fixed set of sequence lengths, SITH‑RNN correctly processes sequences that are much longer or shorter than those seen during training, a capability that standard RNNs lack.
• Bridging neuroscience and AI: Offers concrete evidence that biologically plausible temporal coding schemes can improve practical machine‑learning systems, suggesting a new class of inductive biases for sequential modeling.

Methodology

1. Task design – The authors created a synthetic language classification problem: each input is a string of characters that forms a “word.” The network must map the word to its class label. This mimics the hierarchical nature of language (characters → words → meaning).
2. Network families
   • SITHCon (feed‑forward): Implements Scale‑Invariant Temporal History (SITH) kernels that encode past inputs with a set of exponentially spaced time constants, but without recurrence.
   • SITH‑RNN (recurrent): Extends SITHCon by adding a recurrent connection that updates a hidden state using the same scale‑invariant kernels, preserving biological plausibility (local, time‑cell‑like dynamics).
   • Baselines – Standard vanilla RNNs, LSTMs, GRUs, and a “generic” RNN with unrestricted parameters.
3. Training regime – All models were trained on the same dataset, using identical optimization settings (Adam, cross‑entropy loss). Model size was varied to keep total parameter counts comparable across families.
4. Evaluation –
   • Learning speed: Number of epochs to reach a target accuracy.
   • Parameter count: Total trainable weights.
   • Zero‑shot generalization: Test on sequences whose lengths fall outside the training distribution (e.g., 2× longer or 0.5× shorter).

The approach is deliberately simple enough for developers to reproduce: the core SITH kernels are just weighted sums of exponentially decaying traces, which can be implemented with a few lines of code in PyTorch or TensorFlow.

Results & Findings

• Hierarchical TRWs: In SITHCon, the first hidden layer responded primarily to recent characters, while deeper layers integrated information over progressively longer windows, despite each layer receiving the same set of time constants.
• Learning efficiency: The built‑in temporal prior allowed SITH‑RNN to converge dramatically faster, even with a tiny hidden state.
• Robustness to novel timescales: Because the SITH kernels are scale‑invariant (they cover a continuum of temporal scales), the network can interpolate to unseen sequence lengths without re‑training.

Overall, the experiments validate the hypothesis that a scale‑invariant temporal prior is a powerful inductive bias for sequential tasks.

Practical Implications

• Lightweight sequence models: Developers building on‑device NLP or time‑series classifiers can replace heavyweight LSTMs/Transformers with a SITH‑RNN‑style module, cutting memory and compute budgets dramatically.
• Robustness to variable-length inputs: Applications such as streaming sensor data, log analysis, or real‑time speech recognition often encounter unpredictable sequence lengths. A scale‑invariant recurrent core can handle these variations without needing padding tricks or curriculum training.
• Improved sample efficiency: In low‑data regimes (e.g., few‑shot language adaptation, medical time‑series), the built‑in temporal structure can accelerate convergence, reducing the amount of labeled data required.
• Neuro‑inspired AI libraries: The SITH kernel is a drop‑in layer that can be added to existing frameworks (PyTorch nn.Module, TensorFlow Layer). Open‑source implementations could become a new “temporal prior” primitive, similar to attention or convolution.
• Cross‑disciplinary tooling: For researchers building cognitive models of human memory, the same codebase can serve both scientific simulations and production systems, fostering tighter collaboration between neuroscience and AI engineering.

Limitations & Future Work

• Synthetic task: The language classification benchmark is deliberately simple; performance on real‑world NLP (e.g., sentiment analysis, translation) remains to be demonstrated.
• Fixed kernel shapes: The current SITH implementation uses a predetermined exponential basis. Allowing the network to learn the spacing or shape of the kernels could further improve adaptability.
• Scalability to very long contexts: While zero‑shot generalization works for moderate length changes, handling ultra‑long documents (thousands of tokens) may still require hierarchical stacking or memory‑augmented mechanisms.
• Biological fidelity vs. engineering trade‑offs: The model respects certain neuro‑biological constraints (locality, time‑cell dynamics) but abstracts away many cortical complexities (e.g., gating, neuromodulation). Future work could integrate additional brain‑inspired mechanisms such as predictive coding or attention.

Bottom line: By embedding a scale‑invariant temporal prior directly into the recurrent core, the authors provide a compelling blueprint for building faster, smaller, and more flexible sequence models—bridging insights from hippocampal time cells to practical AI systems.

Authors

• Aakash Sarkar
• Marc W. Howard

Paper Information

• arXiv ID: 2601.02618v1
• Categories: q-bio.NC, cs.AI, cs.CL, cs.LG, cs.NE
• Published: January 6, 2026
• PDF: Download PDF