How I Built a Tool to Detect AI-Generated Fake References

Source: Dev.to

Introduction

LLMs have become part of everyday academic and technical writing.
But there is a problem the academic community has been flagging for a while
(Rolling Stone article): LLMs are very good at inventing citations.
They may look plausible and almost match real papers, yet they confidently cite work that does not exist at all. The academic community is calling these fabricated citations Ghost References.

Closing out my year with a journal‑editor shocker 🧵

“Checking new manuscripts today I reviewed a paper attributing two papers to me that I did not write. A daft thing for an author to do, of course. But intrigued I web‑searched one of the titles and that’s when it got really weird…” — Ben Williamson (@benpatrickwill.bsky.social) (2025‑12‑19)

As Professor Ben Williamson and Aaron Tay explained, the root problem is deep‑seated:

“The ghost reference problem is a chronic condition that has become acute.
The infection predates GenAI; the technology has simply lowered our immune response while accelerating transmission.”

The issue is compounded because LLMs with general web‑search capabilities can fail to reliably verify references—the web itself contains fake citations, creating a dangerous feedback loop. Even when wrong, these sources are widely assumed to be authentic, and the more they appear in published literature the more they are trusted. For instance, one of the Ghost References to Prof. Williamson’s work has already accumulated 43 citations in Google Scholar.

Addressing the Reviewer’s Burden

Peer reviewers are already stretched thin, and now—thanks to the proliferation of fake references—they have to manually copy‑paste every single reference into a search engine to verify its existence
(Jan Laksar’s LinkedIn post).

This is a tedious, low‑reward task that is often skipped in favor of focusing on the paper’s actual content. The resulting “verification gap” is exactly where ghost references can slip through.

When It Happened to Me

The abstract concern became a concrete problem when I discovered my own paper had been incorrectly cited in a published article
(Lidian Y. C. S.’s LinkedIn post).

Seeing the flawed metadata in a journal was a wake‑up call that led me to build CERCA, an open‑source tool designed to assist researchers, reviewers, and editors in quickly verifying the accuracy of references. It was created to improve trust, transparency, and reliability in academic writing.

What Is CERCA?

CERCA (Citation Extraction & Reference Checking Assistant) is a lightweight command‑line utility that:

1. Scans a PDF and extracts the reference list.
2. Queries trusted repositories (OpenAlex, Crossref, Zenodo).
3. Flags potentially invalid citations with confidence scores.
4. Shows mismatched metadata fields (author names, titles, years, etc.).

Demo

In seconds, CERCA:

• extracts references from a PDF,
• queries OpenAlex, Crossref, and Zenodo,
• flags suspicious citations,
• highlights which metadata fields don’t match.

Instead of copy‑pasting each reference manually, you receive a verification report that can be reviewed in minutes.

Development Insights

Building CERCA required solving a few interesting engineering challenges, especially around fuzzy matching and bibliographic parsing.

1. Parsing Messy Citations

Academic citations appear in dozens of styles (APA, MLA, IEEE, ACM, Vancouver, etc.). Creating a parser that reliably extracts references without false positives was the first hurdle. I leveraged Cermine, a Java library that handles heavy PDF parsing and metadata extraction.

2. Verification Logic

The second challenge was deciding when a citation is a typo versus a hallucination. I implemented fuzzy‑matching heuristics that compare extracted metadata against records returned by the trusted APIs. The tool can detect issues such as:

Example

Cerqueira, M.; Tavares, A.; Couto, C.; Maciel, R.; Santos, D.; Figueira, A.
"Assessing software practitioners' work engagement and job satisfaction."

CERCA detects:

⚠️ Author list mismatch (6 fabricated, 9 omitted)
⚠️ Title incomplete
⚠️ First‑author name inconsistency

Correct reference:

Cerqueira, L., Nunes, L., Guerra, R., … & Mendonça, M. (2025). Assessing Software Practitioners’ Work Engagement and Job Satisfaction in a Large Software Company—What We Have Learned. SN Computer Science, 6(3), 273.

3. Trusted Sources

CERCA queries OpenAlex, Crossref, and Zenodo—repositories that provide DOI‑level metadata and are less prone to the “fake citation” problem. The tool aggregates results, computes a confidence score, and presents a concise report.

Getting Started

# Clone the repository
git clone https://github.com/lidianycs/cerca.git
cd cerca

# Install dependencies (requires Python 3.9+)
pip install -r requirements.txt

# Run CERCA on a PDF
cerca check path/to/your-paper.pdf

The output is a markdown table summarizing each reference, its verification status, and any mismatched fields.

Why It Matters

• Reduces reviewer workload – no more manual copy‑pasting.
• Improves literature quality – ghost references are caught before publication.
• Increases trust – authors, editors, and readers can rely on accurate bibliographies.

Join the Effort

If you encounter ghost references in your own work, or you want to help improve CERCA:

• Star the repository on GitHub.
• Submit issues with examples of false positives/negatives.
• Contribute code or documentation.

Together we can curb the spread of fabricated citations and keep scholarly communication trustworthy.

🔍 Manual Fallback
If automatic search fails, you can right‑click to search for reference titles manually.

🔐 Privacy‑first design – PDFs are never uploaded and never leave your machine. All PDF parsing and reference extraction are performed locally.

Tech Stack

• Java + JavaFX – Cross‑platform desktop application
• Cermine – PDF parsing and metadata extraction
• OpenAlex, Crossref, Zenodo APIs – Reference verification
• JavaWuzzy – Handles citation variations and typos

I chose this stack to build a Java desktop app using JavaFX for cross‑platform compatibility (Windows, macOS, Linux).

Why Open Source?

Because the tool’s purpose is to be transparent. It addresses a collective problem, and by making CERCA open source I invite the community to audit the code, improve the parsers, and integrate more databases.

It is licensed under the GNU Affero General Public License (AGPL‑3.0).

Who Can Use CERCA?

• Researchers performing final manuscript checks
• Reviewers assessing reference consistency
• Editors supporting editorial quality control
• Meta‑researchers and reproducibility workflows

Join the Effort (Again)

Ghost references threaten scholarly trust. CERCA is a start, but it needs your expertise.

Try it now:

📥 Download CERCA (v1.1‑alpha) – Windows | macOS | Linux

Cerca does not solve the problem of ghost references completely and is still a work in progress. It is a small, practical step. If it helps a researcher catch even one incorrect reference, saves a reviewer time, or encourages more critical engagement with AI‑generated text, then it is already serving its purpose.

Help improve it:

• 🐛 Found an edge case?
• 💡 Have ideas?
• 🔧 Want to contribute?

👉🏾 Download the tool and explore the repository here.

This project is an invitation to the research and developer communities to experiment, evaluate, and build better tools together.

Share your results: Did CERCA catch a ghost reference in your work? I’d love to hear about it in the comments.