[Paper] Modality-Dependent Memory Mechanisms in Cross-Modal Neuromorphic Computing

Source: arXiv - 2512.18575v1

Overview

This paper investigates how different memory modules behave inside spiking neural networks (SNNs) when they are tasked with processing visual and auditory neuromorphic data. By systematically swapping in Hopfield networks, Hierarchical Gated Recurrent Networks (HGRNs), and supervised contrastive learning (SCL) as the “memory” component, the authors reveal that the best‑performing mechanism depends heavily on the sensory modality—a finding that could reshape how we design low‑power, brain‑inspired AI systems.

Key Contributions

• First cross‑modal ablation study of memory‑augmented SNNs, covering both vision (N‑MNIST) and audition (SHD).
• Empirical comparison of three memory paradigms (Hopfield, HGRN, SCL) across five network architectures, exposing strong modality‑specific performance gaps.
• Demonstration that joint multi‑modal training with HGRN yields a single model that matches the accuracy of separate, modality‑specific networks.
• Engram similarity analysis showing minimal cross‑modal alignment (0.038), supporting the need for modality‑aware memory design.
• Quantification of energy efficiency, reporting a 603× reduction compared with conventional deep nets, highlighting the practical advantage of neuromorphic hardware.

Methodology

1. Datasets –

   • N‑MNIST: event‑based version of the classic MNIST digit dataset, representing visual spikes.
   • SHD: Spiking Heidelberg Digits, an auditory benchmark where spoken digits are encoded as spike trains.

2. Base SNN Architecture – A lightweight spiking backbone (Leaky‑Integrate‑and‑Fire neurons) that processes raw event streams.

3. Memory Modules –

   • Hopfield Network: classic associative memory with energy‑based retrieval.
   • Hierarchical Gated Recurrent Network (HGRN): a multi‑scale recurrent unit that gates information flow across time.
   • Supervised Contrastive Learning (SCL): a loss‑driven embedding that encourages intra‑class compactness and inter‑class separation.

4. Experimental Design –

   • Ablation: Each memory module is inserted into five SNN variants (different depth/width) and trained separately on each modality.
   • Joint Training: A single HGRN‑augmented SNN is trained on a combined visual+auditory dataset to test unified deployment.
   • Metrics: Classification accuracy, cross‑modal engram similarity, and energy consumption (measured on Intel Loihi‑compatible simulators).

5. Analysis Tools – Engram similarity is computed via cosine similarity of the learned memory weight vectors across modalities; energy is estimated from spike‑count‑based power models.

Results & Findings

• Hopfield excels on vision but collapses on audio, indicating strong specialization to spatial spike patterns.
• SCL offers the most balanced performance, sacrificing a few points on vision to gain a sizable boost on audio.
• Joint HGRN training reaches 94.41 % (visual) and 79.37 % (audio), achieving an 88.78 % average—essentially matching the separate‑model baseline while using a single set of weights.
• Engram similarity of 0.038 confirms that the learned memory representations for the two modalities are almost orthogonal, justifying the observed performance gaps.
• Energy: The best SNN configuration consumes roughly 0.16 % of the power required by an equivalent ANN, translating to a 603× efficiency gain.

Practical Implications

• Hardware‑aware model design – When targeting neuromorphic chips (e.g., Loihi, BrainChip), developers should pick memory modules that align with the dominant sensor modality of their application (vision‑heavy robotics vs. audio‑centric voice assistants).
• Unified deployments – The joint HGRN approach shows that a single SNN can serve multi‑sensor platforms without a proportional increase in memory footprint, simplifying firmware and reducing latency.
• Energy‑critical edge devices – The demonstrated 600× power savings make memory‑augmented SNNs attractive for battery‑operated wearables, drones, or IoT gateways that need continuous perception.
• Tooling impact – Frameworks like BindsNET, Norse, or SpykeTorch can incorporate these memory blocks as plug‑and‑play modules, enabling rapid prototyping of modality‑specific or multimodal pipelines.
• Safety‑critical systems – Knowing that a Hopfield‑based SNN may underperform on auditory cues can steer engineers to avoid it in applications where sound detection is vital (e.g., acoustic anomaly detection in factories).

Limitations & Future Work

• Dataset scope – Only two neuromorphic benchmarks were examined; broader modality coverage (e.g., tactile, radar) is needed to generalize the findings.
• Memory size scaling – The study kept memory capacity constant; exploring how scaling the number of stored patterns influences cross‑modal transfer remains open.
• Hardware validation – Energy estimates rely on simulator models; real‑world measurements on physical neuromorphic chips would solidify the claimed efficiency gains.
• Dynamic modality switching – Future research could investigate online adaptation where the same SNN switches memory strategies on the fly based on incoming sensor streams.

By exposing the modality‑dependent nature of memory mechanisms in spiking networks, this work equips developers with concrete guidance for building energy‑efficient, multimodal AI systems on the next generation of neuromorphic hardware.

Authors

• Effiong Blessing
• Chiung-Yi Tseng
• Somshubhra Roy
• Junaid Rehman
• Isaac Nkrumah

Paper Information

• arXiv ID: 2512.18575v1
• Categories: cs.LG, cs.AI, cs.NE
• Published: December 21, 2025
• PDF: Download PDF