[Paper] Neurosim: A Fast Simulator for Neuromorphic Robot Perception

Source: arXiv - 2602.15018v1

Overview

Neurosim is a high‑performance, GPU‑accelerated library that lets you simulate a wide range of robot sensors—event‑based cameras, RGB/Depth imagers, and IMUs—alongside the agile dynamics of multi‑rotor drones in richly changing environments. By delivering up to ~2700 FPS on a single desktop GPU and coupling with the ZeroMQ‑based Cortex messaging layer, Neurosim makes it possible to train and test neuromorphic perception and control pipelines in real‑time, bridging the gap between simulation and deployment.

Key Contributions

• Fast, unified sensor simulation for event‑based vision, conventional RGB/D cameras, depth, and inertial sensors, all running on the same GPU pipeline.
• Real‑time multi‑rotor dynamics engine that can handle complex, dynamic scenes (e.g., moving obstacles, wind gusts) without sacrificing frame rate.
• Cortex communication stack (ZeroMQ + native NumPy/PyTorch support) enabling low‑latency, high‑throughput data exchange between Python and C++ components.
• Demonstrated workflow for self‑supervised learning on time‑synchronized multi‑modal data and closed‑loop testing of neuromorphic controllers.
• Open‑source release (GitHub) with comprehensive examples, making the tool immediately usable by the robotics and ML community.

Methodology

Neurosim builds on two core ideas:

1. GPU‑centric sensor pipelines – Each sensor type is expressed as a series of CUDA kernels that generate pixel‑wise or event‑wise data directly on the GPU. By keeping the entire simulation on the device, data never needs to be copied back to the CPU until the final message dispatch, eliminating bottlenecks.

2. ZeroMQ‑backed message passing (Cortex) – Sensor outputs are wrapped as NumPy arrays or PyTorch tensors and streamed over a ZeroMQ socket. Cortex abstracts the serialization details, providing a simple publish/subscribe API that works identically in Python and C++. This design lets researchers plug in any ML model (e.g., a spiking neural network in PyTorch) without rewriting data‑handling code.

The authors evaluated the system by

• (i) training a self‑supervised depth‑estimation network on synchronized event‑camera and IMU streams, and
• (ii) deploying a neuromorphic controller that stabilizes a simulated quadrotor in a cluttered arena, all while maintaining real‑time frame rates.

Results & Findings

These numbers show that Neurosim can comfortably exceed the timing requirements of most real‑time robotic perception pipelines (typically 30–120 Hz), even when processing high‑rate event streams. The self‑supervised training experiment achieved comparable accuracy to a baseline trained on real hardware data, confirming that the simulated modalities are realistic enough for downstream learning.

Practical Implications

• Accelerated prototyping – Developers can iterate on perception algorithms (e.g., spiking CNNs, event‑based SLAM) without needing physical sensors or flight tests, cutting hardware costs and safety risks.
• Seamless ML integration – Cortex’s native tensor support means you can drop a PyTorch model directly into the simulation loop, enabling rapid experimentation with neuromorphic networks, reinforcement learning agents, or sensor‑fusion architectures.
• Scalable benchmarking – Because the whole stack runs on a GPU, you can benchmark multiple sensor configurations or control policies in parallel, useful for hyper‑parameter sweeps or automated architecture search.
• Real‑time hardware‑in‑the‑loop (HIL) – The low‑latency messaging makes it feasible to replace parts of the pipeline with actual hardware (e.g., a real IMU) while keeping the rest simulated, facilitating mixed‑reality testing.
• Open ecosystem – With the code publicly available, the community can contribute new sensor models (LiDAR, radar), extend dynamics to ground robots, or integrate with ROS 2 via a simple bridge.

Limitations & Future Work

• Physics fidelity – The dynamics engine focuses on speed; high‑precision aerodynamic effects (e.g., blade‑tip vortices) are abstracted, which may limit transfer to highly aggressive flight regimes.
• Sensor realism – While event‑camera noise models are included, some hardware quirks (e.g., temperature‑dependent drift, lens distortion) are not yet simulated.
• Scalability beyond a single GPU – Current implementation does not distribute simulation across multiple GPUs or nodes, which could become a bottleneck for massive multi‑robot scenarios.
• Integration with ROS – The authors note plans to provide a ROS 2 wrapper, which would broaden adoption in existing robotics stacks.

Future work aims to tighten the physics‑sensor coupling, add more diverse sensor families, and explore multi‑GPU orchestration for large‑scale swarm simulations.

Authors

• Richeek Das
• Pratik Chaudhari

Paper Information

• arXiv ID: 2602.15018v1
• Categories: cs.RO, cs.CV
• Published: February 16, 2026
• PDF: Download PDF