[Paper] A Large-Language-Model Framework for Automated Humanitarian Situation Reporting

Source: arXiv - 2512.19475v1

Overview

Humanitarian agencies rely on fast, accurate situation reports to coordinate relief efforts, but creating these reports today is a manual, time‑consuming process. This paper introduces a fully automated pipeline that leverages large language models (LLMs) to ingest heterogeneous humanitarian documents (news articles, UN briefings, NGO updates) and output structured, evidence‑grounded reports complete with citations. The authors demonstrate the system across 13 real‑world crises, showing that AI can produce reports that are both reliable and ready for operational use.

Key Contributions

• End‑to‑end LLM framework that converts raw humanitarian texts into multi‑level, citation‑backed situation reports.
• Semantic clustering + automatic question generation to surface the most relevant, important, and urgent information gaps.
• Retrieval‑augmented generation (RAG) for answer extraction with explicit source citations, achieving >76 % precision/recall on citation linking.
• Multi‑level summarization (answer‑level, topic‑level, executive summary) that preserves interpretability and actionability.
• Internal evaluation metrics that mimic expert reasoning, yielding >0.80 F1 agreement between human and LLM assessments.
• Empirical validation on >1,100 documents from verified sources (e.g., ReliefWeb) covering natural disasters and conflicts.

Methodology

1. Document Ingestion & Clustering – Raw PDFs, HTML pages, and CSV feeds are vector‑encoded with a pre‑trained LLM and grouped by semantic similarity, ensuring that related pieces (e.g., damage assessments, logistics updates) are processed together.
2. Automatic Question Generation – For each cluster, the system prompts the LLM to generate questions that are (a) relevant to the crisis context, (b) important for decision‑makers, and (c) urgent. Human‑rated thresholds filter out low‑quality questions.
3. Retrieval‑Augmented Answer Extraction – Using the generated questions, a RAG pipeline retrieves the most pertinent passages from the original documents, then asks the LLM to synthesize concise answers while attaching inline citations (document ID + snippet).
4. Multi‑Level Summarization – Answers are first grouped by thematic tags, then summarized into topic‑level briefs, and finally distilled into an executive summary that highlights key impacts, needs, and recommended actions.
5. Evaluation Loop – The authors built automatic metrics (relevance, importance, urgency scores) that approximate expert scoring, and they also ran human‑in‑the‑loop validation to compute F1 agreement.

Results & Findings

• Question Quality: 84.7 % of generated questions were judged relevant, 84.0 % important, and 76.4 % urgent.
• Answer Extraction: 86.3 % relevance; citation precision = 78.1 %, recall = 76.5 %.
• Human vs. LLM Evaluation: Agreement surpassed an F1 of 0.80, indicating that the AI’s internal metrics align closely with expert judgments.
• Comparative Baselines: The proposed pipeline produced reports that were more structured (clear sections, traceable citations) and more actionable (higher urgency detection) than traditional keyword‑based summarizers or generic LLM outputs.

Practical Implications

• Rapid Situational Awareness: NGOs and UN agencies can generate daily or hourly briefings without waiting for analysts to manually sift through hundreds of documents.
• Transparent Decision‑Making: Inline citations let responders trace every claim back to its source, satisfying audit and accountability requirements.
• Scalable Crisis Monitoring: The modular pipeline can be deployed across multiple languages and data streams (social media, satellite reports) to maintain a continuous “situational dashboard.”
• Developer Integration: The framework exposes RESTful endpoints for each stage (clustering, Q‑gen, RAG, summarization), making it straightforward to embed into existing humanitarian information systems or incident‑response platforms.
• Cost Efficiency: By automating routine reporting, organizations can reallocate analyst time to higher‑level strategic tasks, potentially reducing operational costs by 30‑40 % in large‑scale emergencies.

Limitations & Future Work

• Source Quality Dependency: The system’s accuracy hinges on the reliability of input documents; noisy or biased sources can propagate errors.
• Language Coverage: Evaluation focused on English‑language reports; extending to low‑resource languages will require multilingual LLMs and better translation pipelines.
• Real‑Time Constraints: While the pipeline is automated, processing large document volumes still incurs latency; future work will explore streaming architectures and model distillation for faster inference.
• Human Oversight: Although the AI achieves high agreement with experts, a final human validation step is still recommended for high‑stakes decisions.

Bottom line: This research shows that with the right combination of LLM reasoning, retrieval, and evaluation, we can move from labor‑intensive report writing to near‑real‑time, evidence‑backed humanitarian intelligence—opening the door for smarter, faster disaster response.

Authors

• Ivan Decostanzi
• Yelena Mejova
• Kyriaki Kalimeri

Paper Information

• arXiv ID: 2512.19475v1
• Categories: cs.CL
• Published: December 22, 2025
• PDF: Download PDF