[Paper] Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination

Source: arXiv - 2604.24690v1

Overview

The paper investigates whether today’s large language models (LLMs) can perform the kind of deep, evidence‑driven reasoning that professional historians do. To test this, the authors built ProHist‑Bench, a benchmark rooted in the Chinese Imperial Examination (Keju) – a 1,300‑year‑long series of exams that demanded sophisticated knowledge of politics, society, and intellectual history. By evaluating 18 state‑of‑the‑art LLMs on 400 expert‑crafted questions, the study reveals a sizable gap between current model capabilities and the demands of serious historical research.

Key Contributions

• ProHist‑Bench dataset: 400 rigorously vetted historical questions spanning eight Chinese dynasties, each paired with 10,891 fine‑grained rubrics that capture multiple reasoning steps (evidence selection, argument construction, source criticism).
• Interdisciplinary benchmark design: Collaboration between NLP researchers and historians ensures the tasks reflect authentic historiographic skills rather than surface‑level fact recall.
• Comprehensive evaluation: 18 LLMs (including GPT‑4, Claude, LLaMA‑2, and open‑source alternatives) are tested under zero‑shot, few‑shot, and fine‑tuned settings, with performance broken down by question type (chronology, causality, source analysis, essay synthesis).
• Diagnostic analysis: The authors identify systematic failure modes—e.g., inability to cite primary sources, confusion between correlation and causation, and over‑reliance on memorized facts.
• Open‑source release: Full dataset, rubrics, and evaluation scripts are made publicly available to foster further research on domain‑specific reasoning in LLMs.

Methodology

1. Question Curation – Historians selected representative Keju prompts from eight dynasties, then rewrote them into modern English (and Chinese) while preserving the original analytical depth.
2. Rubric Construction – For each question, a multi‑level rubric was created:
   • Evidence Retrieval: Identify relevant primary/secondary sources.
   • Reasoning Chain: Outline causal links or interpretive arguments.
   • Answer Quality: Score factual accuracy, logical coherence, and historiographic nuance.
3. Model Evaluation – All LLMs were queried with the same prompt format. Responses were automatically parsed for citation patterns and then scored by the rubrics using a combination of rule‑based checks and human expert verification (10 % of samples).
4. Analysis Layers – Results were aggregated by model size, training data cut‑off, and prompting strategy (zero‑shot vs. few‑shot). Error analyses focused on where models deviated from the rubric expectations.

Results & Findings

• Overall performance: The best commercial model (GPT‑4) achieved an average rubric score of 58 %, still far below the 85 % threshold set by expert historians.
• Skill breakdown:
  • Factual recall (e.g., dates, names) – relatively high (≈80 % for top models).
  • Evidence selection – low (≈45 %); models often fabricated or omitted primary sources.
  • Causal reasoning – poorest area (≈30 %); models struggled to articulate multi‑step historical explanations.
• Prompting effect: Few‑shot examples improved factual recall marginally but did not close the gap in reasoning or source criticism.
• Fine‑tuning: Even models fine‑tuned on a subset of the benchmark showed modest gains (≈5 % absolute), indicating that the challenge lies more in reasoning architecture than data quantity.
• Error patterns: Common hallucinations involved conflating unrelated events, over‑generalizing from a single source, and treating modern interpretations as contemporaneous facts.

Practical Implications

• Tooling for historians – Current LLMs can serve as quick reference assistants for dates or basic summaries, but they cannot replace expert source analysis or argument construction.
• Educational tech – Platforms that generate practice exam questions or provide feedback on student essays should treat LLM‑generated historical content as a draft that requires human verification.
• Domain‑specific LLM development – The benchmark highlights the need for models that integrate structured historical knowledge bases (e.g., digitized archives) and explicit reasoning modules (chain‑of‑thought, retrieval‑augmented generation).
• Enterprise knowledge management – Companies dealing with legacy documents (e.g., legal, cultural heritage) can leverage LLMs for preliminary indexing, but must embed rigorous validation pipelines before using the output for decision‑making.

Limitations & Future Work

• Cultural scope – ProHist‑Bench focuses exclusively on Chinese Imperial Examination material; results may not generalize to Western or other historiographic traditions.
• Language bias – While both Chinese and English prompts are provided, the majority of evaluated models were primarily trained on English data, possibly disadvantaging them on nuanced Chinese source questions.
• Rubric subjectivity – Although expert‑crafted, the rubrics involve interpretive judgments that could vary across historiographic schools.
• Future directions suggested by the authors include expanding the benchmark to other regions, integrating multimodal sources (e.g., inscriptions, maps), and exploring hybrid architectures that combine LLMs with symbolic reasoning or retrieval from curated historical corpora.

Authors

• Lirong Gao
• Zeqing Wang
• Yuyan Cai
• Jiayi Deng
• Yanmei Gu
• Yiming Zhang
• Jia Zhou
• Yanfei Zhang
• Junbo Zhao

Paper Information

• arXiv ID: 2604.24690v1
• Categories: cs.CL
• Published: April 27, 2026
• PDF: Download PDF