[Paper] They Said Memes Were Harmless-We Found the Ones That Hurt: Decoding Jokes, Symbols, and Cultural References

Source: arXiv - 2602.03822v1

Overview

The paper tackles a surprisingly tricky problem: detecting harmful content hidden in memes. Because memes blend images, text, and cultural symbols, existing AI models often miss the hateful intent, mistake satire for abuse, or can’t explain why they flagged something. The authors propose CROSS‑ALIGN+, a three‑stage system that injects world knowledge, sharpens decision boundaries, and produces human‑readable explanations, pushing meme‑based abuse detection well beyond the current state of the art.

Key Contributions

• Cultural‑aware multimodal encoding: Enriches image‑text representations with structured knowledge from ConceptNet, Wikidata, and Hatebase to capture implicit symbols and references.
• Boundary‑refining LoRA adapters: Lightweight parameter‑efficient adapters that fine‑tune large vision‑language models (LVLMs) to better separate satire from genuine hate.
• Cascaded explanation generator: A post‑hoc module that produces step‑by‑step rationales (what visual cue, what textual cue, what cultural link) for each prediction, dramatically improving interpretability.
• Comprehensive evaluation: Benchmarked on five public meme‑abuse datasets and eight LVLMs, showing up to 17 % relative F1 gain over the strongest baselines.
• Open‑source toolkit: The authors release code, pretrained adapters, and a small knowledge‑lookup API to facilitate reproducibility and downstream adoption.

Methodology

CROSS‑ALIGN+ works in three sequential stages:

1. Cultural Knowledge Injection (Stage I)

   • The raw meme (image + overlaid text) is first processed by a standard LVLM encoder (e.g., CLIP‑ViT).
   • Detected entities (objects, OCR text, facial expressions) are linked to concepts in ConceptNet (common‑sense relations), Wikidata (entity facts), and Hatebase (known hate symbols).
   • These external embeddings are concatenated with the LVLM’s hidden states, giving the model a “cultural lens” to interpret symbols like “Pepe the Frog” or “OK hand” that may carry hateful connotations in specific sub‑communities.

2. Decision‑Boundary Sharpening (Stage II)

   • Instead of fine‑tuning the massive LVLM from scratch, the authors attach Low‑Rank Adaptation (LoRA) adapters to the final classification head.
   • LoRA learns a small set of task‑specific weight updates (≈0.5 % of the original parameters) that push the decision surface away from ambiguous regions where satire and hate overlap.
   • This parameter‑efficient approach keeps training fast and preserves the LVLM’s general visual‑language knowledge.

3. Cascaded Explanation Generation (Stage III)

   • After a meme is classified, a lightweight transformer decoder takes the enriched multimodal representation and produces a three‑part rationale:
     1. Visual cue (e.g., “the image shows a hand making the ‘OK’ sign”).
     2. Textual cue (e.g., “the caption reads ‘All good’”).
     3. Cultural link (e.g., “the ‘OK’ sign has been co‑opted by extremist groups per Hatebase”).
   • The explanations are trained with a mixture of supervised rationales (from a small human‑annotated subset) and self‑generated pseudo‑labels, encouraging the model to be transparent without sacrificing accuracy.

Results & Findings

• Consistent gains across all benchmarks, confirming that external knowledge and LoRA adapters complement each other.
• Interpretability test: Human judges rated the generated explanations as “clearly helpful” in 84 % of cases, versus 32 % for vanilla LVLM outputs.
• Efficiency: Adding Stage I and Stage III adds only ~0.2 B extra parameters; inference latency grows < 15 ms per meme on a single A100 GPU.

Practical Implications

• Content moderation pipelines: Platforms can plug the lightweight LoRA adapters into their existing LVLMs, instantly boosting detection of culturally nuanced hate without a full model retrain.
• Policy‑aware AI: The explicit rationales make it easier for compliance teams to audit decisions, satisfy regulatory demands (e.g., EU Digital Services Act), and reduce false‑positive bans on satire.
• Developer tooling: The released API for knowledge lookup (ConceptNet/Wikidata/Hatebase) can be reused for other multimodal tasks such as brand safety, misinformation flagging, or contextual advertising.
• Cross‑cultural deployment: Because the knowledge bases are multilingual, the framework can be adapted to non‑English meme ecosystems with minimal extra data collection.

Limitations & Future Work

• Knowledge‑base coverage: The system inherits biases and gaps from ConceptNet, Wikidata, and Hatebase; obscure or emerging symbols may still slip through.
• Static knowledge linking: Entity linking is performed offline per meme, which can be a bottleneck for high‑throughput streams; future work could explore end‑to‑end differentiable retrieval.
• Explainability depth: While the cascaded explanations are human‑readable, they are not formally verified; integrating causal attribution methods could make the rationales more robust.
• Generalization to video memes: The current design handles static images; extending the pipeline to short video loops (e.g., TikTok) is an open challenge.

Overall, CROSS‑ALIGN+ demonstrates that marrying structured cultural knowledge with efficient model adaptation yields both higher detection performance and the transparency that real‑world moderation systems desperately need.

Authors

• Sahil Tripathi
• Gautam Siddharth Kashyap
• Mehwish Nasim
• Jian Yang
• Jiechao Gao
• Usman Naseem

Paper Information

• arXiv ID: 2602.03822v1
• Categories: cs.CL
• Published: February 3, 2026
• PDF: Download PDF