[Paper] HalluCiteChecker: A Lightweight Toolkit for Hallucinated Citation Detection and Verification in the Era of AI Scientists

Source: arXiv - 2604.26835v1

Overview

The paper presents HalluCiteChecker, an open‑source toolkit that automatically spots and validates “hallucinated” citations—references that look real but don’t correspond to any actual publication. As AI‑powered writing assistants become commonplace, such bogus citations are on the rise, creating extra work for reviewers and threatening the credibility of scholarly communication. HalluCiteChecker aims to make detection fast, lightweight, and usable on a regular laptop without any internet connection.

Key Contributions

• Formalization of the problem – Defines hallucinated citation detection as a concrete NLP task with clear evaluation criteria.
• Lightweight, offline toolkit – Runs in seconds on a CPU‑only laptop; no need for large GPU clusters or external APIs.
• Easy integration – Distributed via PyPI and released under Apache 2.0, making it plug‑and‑play for CI pipelines, pre‑submission checks, or reviewer tooling.
• Open‑source implementation – Full code, documentation, and a demo video are publicly available, encouraging community extensions.
• Practical benchmark – Provides baseline performance numbers and error analyses that other researchers can use as a starting point.

Methodology

1. Citation Extraction – The system parses a manuscript (PDF, LaTeX, or plain text) to locate every in‑text citation and its corresponding bibliography entry.
2. Metadata Normalization – Extracted references are cleaned and transformed into a canonical form (authors, title, venue, year).
3. Candidate Retrieval – Using a local index of scholarly metadata (e.g., a snapshot of Crossref or DBLP), the toolkit performs fuzzy matching to find the most likely real paper.
4. Verification Scoring – A lightweight neural reranker (a few hundred thousand parameters) evaluates similarity between the extracted citation and candidate records, producing a confidence score.
5. Hallucination Flagging – If no candidate exceeds a configurable threshold, the citation is marked as hallucinated and reported to the user.

All steps rely on CPU‑friendly libraries (spaCy for parsing, Faiss for fast nearest‑neighbor search) and can be executed entirely offline.

Results & Findings

• Detection speed: Average of 1.8 seconds per 10‑page manuscript on a 2020‑era laptop (Intel i5, 8 GB RAM).
• Precision/Recall: Achieves ~92 % precision and ~78 % recall on a manually curated test set of 500 papers containing both genuine and fabricated citations.
• False‑positive analysis: Most errors stem from ambiguous abbreviations or missing metadata in the local index, not from the model itself.
• Scalability: Adding more metadata (e.g., arXiv snapshots) improves recall with only modest impact on runtime.

Practical Implications

• Reviewer assistance: Journals can integrate HalluCiteChecker into their submission portals to automatically flag suspicious references before peer review, cutting down manual verification time.
• Pre‑submission hygiene: Authors can run the tool locally to catch accidental hallucinations introduced by AI writing assistants, improving manuscript quality.
• CI/CD for research software: Teams that generate documentation or white‑papers from code can embed the checker in their CI pipelines to enforce citation integrity.
• Educational use: Graduate programs can teach students about responsible AI‑assisted writing by demonstrating how hallucinated citations are detected.
• Extensibility: Because the toolkit is modular, organizations can swap in proprietary metadata sources (e.g., internal technical reports) to tailor verification to their domain.

Limitations & Future Work

• Metadata coverage: The offline index may miss very recent or niche publications, limiting recall for cutting‑edge fields.
• Language support: Current implementation focuses on English‑language papers; extending to multilingual corpora will require additional tokenizers and metadata sources.
• Contextual reasoning: The system does not yet assess whether a citation is relevant to the surrounding claim—only whether the reference exists. Future work could combine hallucination detection with citation‑context verification.
• User feedback loop: Incorporating a mechanism for reviewers to confirm or correct flagged citations could improve the model over time, an avenue the authors plan to explore.

Authors

• Yusuke Sakai
• Hidetaka Kamigaito
• Taro Watanabe

Paper Information

• arXiv ID: 2604.26835v1
• Categories: cs.CL, cs.AI, cs.DL
• Published: April 29, 2026
• PDF: Download PDF