[Paper] Bangla Hate Speech Classification with Fine-tuned Transformer Models

Source: arXiv - 2512.02845v1

Overview

The paper tackles hate‑speech detection for Bangla—a language spoken by over 230 million people but still under‑served by NLP tools. By fine‑tuning several transformer models on the BLP 2025 shared‑task data, the authors demonstrate that language‑specific pre‑training (BanglaBERT) dramatically outperforms generic multilingual models, offering a practical path toward safer online communities in low‑resource settings.

Key Contributions

• Comprehensive baseline suite – Re‑implemented classic baselines (majority, random, SVM) plus Logistic Regression, Random Forest, and Decision Tree for a solid reference point.
• Transformer benchmarking – Evaluated four transformer families (DistilBERT, BanglaBERT, m‑BERT, XLM‑RoBERTa) on two hate‑speech subtasks (binary detection and fine‑grained classification).
• Empirical proof of language‑specific pre‑training – Showed BanglaBERT, despite its smaller footprint, consistently beats larger multilingual models on Bangla hate‑speech tasks.
• Open‑source reproducibility – Provided code and model checkpoints, enabling other researchers and engineers to replicate and extend the work.

Methodology

1. Data – Used the BLP 2025 shared‑task dataset, which contains Bangla social‑media posts annotated for hate speech (Subtask 1A: binary, Subtask 1B: multi‑class).
2. Baseline models – Trained traditional classifiers on TF‑IDF features derived from the raw text.
3. Transformer fine‑tuning – Loaded pre‑trained checkpoints (DistilBERT, m‑BERT, XLM‑RoBERTa, BanglaBERT) and added a single linear classification head.
4. Training regime – Applied standard practices: AdamW optimizer, learning‑rate warm‑up, early stopping on validation loss, and class‑weight balancing to mitigate label skew.
5. Evaluation – Reported macro‑averaged F1 scores (the official shared‑task metric) for both subtasks, comparing each model against the baselines.

Results & Findings

• All transformer models beat the classic baselines, confirming the power of contextual embeddings for Bangla.
• BanglaBERT achieved the highest macro‑F1 on both tasks, despite having fewer parameters than the multilingual counterparts.
• DistilBERT lagged behind, likely due to its reduced capacity and lack of Bangla‑specific pre‑training data.

Practical Implications

• Moderation pipelines – Social‑media platforms can integrate BanglaBERT as a drop‑in component to flag hateful content in real time, reducing reliance on manual review.
• Resource‑efficient deployment – BanglaBERT’s smaller size means lower GPU/CPU requirements, making it feasible for edge devices or low‑budget cloud setups.
• Transferable workflow – The same fine‑tuning recipe can be applied to other low‑resource languages, encouraging broader adoption of language‑specific models.
• Open‑source tooling – With the authors’ code publicly available, developers can quickly prototype custom moderation bots, sentiment analyzers, or community‑health dashboards for Bangla‑speaking audiences.

Limitations & Future Work

• Dataset scope – The shared‑task corpus is limited in size and domain (mostly public posts), which may not capture the full diversity of Bangla hate speech across platforms.
• Class imbalance – Some hate‑speech categories are under‑represented, potentially inflating macro‑F1 scores; more balanced data would yield a clearer picture.
• Model robustness – The study did not explore adversarial attacks or code‑switching (Bangla‑English mixing), common in real‑world posts.
• Future directions – Expanding the dataset, incorporating multilingual code‑switch handling, and experimenting with lightweight distillation of BanglaBERT for on‑device inference are promising next steps.

Authors

• Yalda Keivan Jafari
• Krishno Dey

Paper Information

• arXiv ID: 2512.02845v1
• Categories: cs.CL
• Published: December 2, 2025
• PDF: Download PDF