[Paper] SymSeqBench: a unified framework for the generation and analysis of rule-based symbolic sequences and datasets

Source: arXiv - 2512.24977v1

Overview

The paper presents SymSeqBench, a two‑part open‑source framework that makes it easy to generate, analyze, and benchmark rule‑based symbolic sequences. By grounding the tools in Formal Language Theory, the authors give AI researchers, cognitive scientists, and neuromorphic engineers a common playground for testing sequence‑learning models on tasks that mirror real‑world cognition (language, motor plans, decision chains, etc.).

Key Contributions

• SymSeq: a library for rigorously constructing symbolic sequences from user‑defined grammars and transformation rules, with built‑in analysis utilities (e.g., entropy, hierarchy depth).
• SeqBench: a curated benchmark suite of 30+ rule‑based sequence‑processing tasks (e.g., context‑free nesting, hierarchical counting, pattern completion) that reflect cognitively relevant challenges.
• Unified API: both tools share a modular Python interface, enabling seamless swapping of data generators, task definitions, and evaluation metrics.
• Formal‑theory bridge: each benchmark is explicitly linked to a class in the Chomsky hierarchy, allowing researchers to map model performance to computational complexity.
• Open‑source & extensible: released under an MIT license with documentation, Docker images, and example notebooks for rapid adoption.

Methodology

1. Grammar Specification – Users write a concise description of a formal grammar (regular, context‑free, context‑sensitive, etc.) using a JSON/YAML schema.
2. Sequence Generation – SymSeq parses the grammar, then samples strings according to user‑defined distributions (uniform, biased, Markovian).
3. Task Wrappers – SeqBench wraps each generated dataset in a standard torch.utils.data.Dataset (or TensorFlow tf.data.Dataset) that yields input‑output pairs for supervised or reinforcement‑learning setups.
4. Metrics & Analysis – The framework provides utilities to compute classic FLT metrics (e.g., pumping length, derivation tree depth) and modern ML metrics (accuracy, perplexity, sample efficiency).
5. Benchmark Execution – A command‑line interface runs a model across all tasks, aggregates results, and produces LaTeX/HTML reports for quick comparison.

The whole pipeline is deliberately language‑agnostic; the only requirement is that the downstream model can consume sequences of discrete symbols (e.g., token IDs, one‑hot vectors).

Results & Findings

• Baseline Models – The authors evaluated several architectures (LSTM, Transformer, Spiking Neural Network) on the full SeqBench suite. Predictably, models excelled on regular‑language tasks but showed steep performance drops on context‑free and context‑sensitive benchmarks.
• Complexity‑Performance Correlation – A clear monotonic relationship emerged between the Chomsky class of a task and the amount of data / training steps required for a given model to reach 80 % accuracy.
• Neuromorphic Advantage – A small‑scale spiking network with event‑driven learning matched LSTM performance on hierarchical counting tasks while using ~10× fewer operations, hinting at energy‑efficient sequence processing.
• Diagnostic Power – By isolating failure modes (e.g., inability to maintain nested stack depth), SeqBench helped pinpoint architectural bottlenecks that are invisible on standard language‑model benchmarks.

Practical Implications

• Model Debugging – Developers can use SymSeqBench as a “unit test suite” for any sequence model, quickly surfacing weaknesses in recursion handling, long‑range dependency tracking, or rule generalization.
• Curriculum Design – The graded difficulty across the Chomsky hierarchy enables systematic curriculum learning: start with regular patterns, then progressively introduce context‑free nesting, mirroring human language acquisition.
• Neuromorphic & Edge AI – The benchmark’s low‑overhead data format and support for spiking‑network evaluation make it a ready‑made testbed for energy‑constrained devices (e.g., wearables, robotics).
• Cross‑Disciplinary Research – Psycholinguists can generate controlled stimulus sets that are guaranteed to obey a formal grammar, while AI teams can evaluate whether their models exhibit comparable human‑like error patterns.
• Standardization – By anchoring tasks to formal language classes, the community gains a shared vocabulary for reporting “can the model handle context‑free structure?” rather than vague dataset names.

Limitations & Future Work

• Symbolic Focus – The current version only handles discrete symbol streams; extending to mixed continuous‑discrete modalities (e.g., audio waveforms with symbolic annotations) is left for later.
• Scalability – Generating extremely long context‑sensitive strings can become computationally expensive; the authors suggest integrating grammar‑compression techniques.
• Benchmark Diversity – While 30 tasks cover many classic FLT categories, real‑world corpora (e.g., code, music) are not yet represented; future releases aim to include domain‑specific extensions.
• Evaluation Metrics – The suite primarily reports accuracy and perplexity; richer diagnostics (e.g., probing of internal state representations) are planned.

Authors

• Barna Zajzon
• Younes Bouhadjar
• Maxime Fabre
• Felix Schmidt
• Noah Ostendorf
• Emre Neftci
• Abigail Morrison
• Renato Duarte

Paper Information

• arXiv ID: 2512.24977v1
• Categories: q-bio.NC, cs.AI, cs.LG, cs.NE
• Published: December 31, 2025
• PDF: Download PDF