[Paper] Is Lying Only Sinful in Islam? Exploring Religious Bias in Multilingual Large Language Models Across Major Religions

Source: arXiv - 2512.03943v1

Overview

The paper Is Lying Only Sinful in Islam? Exploring Religious Bias in Multilingual Large Language Models Across Major Religions investigates how state‑of‑the‑art multilingual large language models (LLMs) handle religion‑related queries. By probing models in both English and Bengali, the authors uncover systematic bias—especially a tendency to favor Islamic perspectives—even when the question is neutral. Their findings raise red flags for any product that relies on LLMs for cross‑lingual content moderation, chatbots, or knowledge‑base generation in culturally diverse settings.

Key Contributions

• BRAND dataset – a new, publicly released “Bilingual Religious Accountable Norm Dataset” with > 2,400 question‑answer pairs covering Buddhism, Christianity, Hinduism, and Islam in English and Bengali.
• Three prompt styles (direct question, context‑rich, and counter‑factual) to test how phrasing influences model bias.
• Systematic evaluation of several popular multilingual LLMs (e.g., mBERT, XLM‑R, LLaMA‑2‑13B‑Chat) across languages, revealing a consistent performance gap: English > Bengali.
• Bias diagnosis – quantitative metrics (accuracy, F1, bias score) and qualitative analysis showing a pronounced tilt toward Islamic interpretations, even for religion‑neutral prompts.
• Cross‑disciplinary link – discussion of how these technical bias patterns intersect with Human‑Computer Interaction (HCI) concerns around religious sensitivity and user trust.

Methodology

1. Dataset Construction – The authors curated 2,400+ statements and questions from religious texts, scholarly articles, and crowd‑sourced inputs, then translated each entry into Bengali while preserving nuance.
2. Prompt Design – For every entry they generated three prompt variants:
   • Direct: “Is lying sinful in Islam?”
   • Contextual: “According to the Quran, is lying considered a sin?”
   • Counter‑factual: “If a Buddhist says lying is not a sin, is that correct?”
3. Model Selection – They evaluated a mix of open‑source and commercial multilingual LLMs (mBERT, XLM‑R, BLOOM‑560M, LLaMA‑2‑13B‑Chat, Gemini‑Pro).
4. Evaluation Metrics – Accuracy against the ground‑truth label (sinful vs. not sinful), macro‑averaged F1, and a custom religious bias score that measures deviation from a balanced answer distribution across the four faiths.
5. Statistical Analysis – Paired t‑tests and bootstrapped confidence intervals to confirm that observed differences are not due to random variation.

Results & Findings

• Language Gap – All models scored 8–15 % higher on English prompts than on Bengali equivalents.
• Islamic Bias – Across languages, the bias score consistently favored Islamic answers (e.g., the model would label a neutral statement as “Islamic” 62 % of the time vs. 18 % for Hinduism).
• Prompt Sensitivity – Counter‑factual prompts amplified bias, while contextual prompts reduced it slightly but never eliminated the tilt.
• Model‑Specific Trends – Larger, instruction‑tuned models (LLaMA‑2‑Chat, Gemini‑Pro) showed lower overall bias than smaller encoder‑only models, yet the Islamic preference persisted.
• Qualitative Cases – Examples where a model incorrectly asserted that “lying is only sinful in Islam” even when the question referenced Buddhism, highlighting potential for misinformation.

Practical Implications

• Content Moderation – Platforms that auto‑moderate user‑generated text in South Asian languages must treat LLM outputs with caution; a naïve deployment could unfairly flag or endorse content based on religious bias.
• Chatbots & Virtual Assistants – Voice assistants serving multilingual markets need bias‑aware post‑processing (e.g., rule‑based checks or calibrated response ensembles) to avoid alienating users of non‑Islamic faiths.
• Knowledge‑Base Generation – Automated summarization of religious documents for educational apps should incorporate bias detection pipelines to ensure balanced representation.
• Model Fine‑Tuning – The BRAND dataset offers a ready‑to‑use benchmark for fine‑tuning or RLHF (Reinforcement Learning from Human Feedback) loops aimed at reducing religious bias.
• Regulatory Compliance – In jurisdictions where religious discrimination is legally actionable, the documented bias could expose companies to compliance risk if not mitigated.

Limitations & Future Work

• Scope of Religions – The study focuses on the four major South Asian faiths; bias patterns may differ for other traditions (e.g., Sikhism, Jainism, indigenous beliefs).
• Language Coverage – Only English and Bengali were examined; extending to additional regional languages (Hindi, Tamil, Urdu) could reveal further disparities.
• Model Diversity – While a representative set of LLMs was tested, newer multimodal or retrieval‑augmented models were not included.
• Bias Metric – The custom bias score, though informative, is still a proxy; future work could adopt more nuanced fairness frameworks (e.g., counterfactual fairness).
• Mitigation Strategies – The paper stops at diagnosis; subsequent research should prototype debiasing techniques (data augmentation, adversarial training, post‑hoc calibration) and evaluate their effectiveness on BRAND.

Authors

• Kazi Abrab Hossain
• Jannatul Somiya Mahmud
• Maria Hossain Tuli
• Anik Mitra
• S. M. Taiabul Haque
• Farig Y. Sadeque

Paper Information

• arXiv ID: 2512.03943v1
• Categories: cs.CL, cs.HC
• Published: December 3, 2025
• PDF: Download PDF