[Paper] Exploring Protein Language Model Architecture-Induced Biases for Antibody Comprehension

Source: arXiv - 2512.09894v1

Overview

A new study probes how the internal design of protein language models (PLMs) shapes their ability to “read” antibody sequences. By comparing three cutting‑edge PLMs—AntiBERTa, BioBERT, and ESM‑2—with a generic GPT‑2 baseline, the authors reveal that architectural nuances lead to distinct biases in recognizing antibody‑specific signals such as V‑gene usage, somatic hypermutation, and isotype class. The work bridges deep‑learning model engineering with practical antibody‑design tasks, offering developers concrete guidance on choosing or tailoring PLMs for immunology‑focused applications.

Key Contributions

• Systematic benchmark of three state‑of‑the‑art PLMs and a general‑purpose language model on antibody target‑specificity prediction.
• Quantitative analysis of biological biases (V‑gene, somatic hypermutation, isotype) induced by each model’s architecture.
• Attention‑attribution study showing that antibody‑specialized models naturally attend to complementarity‑determining regions (CDRs), while generic models need explicit CDR‑focused training to achieve similar focus.
• Practical recommendations for model selection and fine‑tuning strategies in computational antibody design pipelines.

Methodology

1. Dataset – Curated a large collection of paired heavy‑chain antibody sequences with known antigen targets, annotated with V‑gene families, mutation counts, and isotype labels.
2. Models –
   • AntiBERTa: a transformer pre‑trained on antibody repertoires.
   • BioBERT: a biomedical BERT model fine‑tuned on protein data.
   • ESM‑2: a large‑scale protein transformer from Meta AI.
   • GPT‑2: vanilla decoder‑only model used as a baseline.
3. Task – Multi‑class classification of antibody target specificity (e.g., viral vs. bacterial antigens).
4. Training – Each model was fine‑tuned on the same training split with identical hyper‑parameters to isolate architectural effects.
5. Bias Evaluation – After training, the authors probed the hidden representations for correlation with V‑gene usage, somatic hypermutation patterns, and isotype information using linear probes and mutual‑information metrics.
6. Attention Attribution – Gradient‑based attention rollout was applied to visualize which residues the models relied on; special focus was placed on the six CDR loops (CDR1‑3 of heavy and light chains).

Results & Findings

• All PLMs outperform the generic GPT‑2, confirming that protein‑specific pre‑training matters.
• AntiBERTa exhibits the highest intrinsic focus on CDRs, translating into superior target‑specificity predictions.
• BioBERT and ESM‑2 capture mutation and isotype signals well but require additional supervision to attend to CDRs.
• Attention visualizations show that, without explicit CDR‑aware fine‑tuning, generic models spread attention across framework regions, diluting functional relevance.

Practical Implications

• Model selection: For projects that need precise epitope mapping or CDR‑level engineering (e.g., affinity maturation), AntiBERTa is the plug‑and‑play choice.
• Fine‑tuning recipes: When using a general protein model (ESM‑2, BioBERT), prepend a small CDR‑masking or region‑highlighting step during fine‑tuning to steer attention toward the functional loops.
• Feature extraction pipelines: The identified biases can be leveraged as lightweight “biological fingerprints” (e.g., V‑gene embeddings) for downstream tasks such as repertoire clustering or isotype prediction without training a full model.
• Tooling: The attention‑attribution code released with the paper can be integrated into existing ML‑ops frameworks (e.g., Hugging Face Transformers) to audit model decisions on antibody data, improving interpretability for regulatory submissions.

Limitations & Future Work

• The benchmark focuses on heavy‑chain sequences only; light‑chain contributions and paired‑chain dynamics remain unexamined.
• All experiments use publicly available repertoires, which may not capture rare or engineered antibody formats (e.g., bispecifics).
• The authors note that scaling up model size (beyond the current 1B‑parameter range) could alter bias patterns, a hypothesis worth testing.
• Future research directions include multi‑modal models that jointly ingest sequence and structural data, and exploring contrastive pre‑training objectives tailored to antibody‑specific functional motifs.

Authors

• Mengren
• Liu
• Yixiang Zhang
• Yiming
• Zhang

Paper Information

• arXiv ID: 2512.09894v1
• Categories: cs.LG
• Published: December 10, 2025
• PDF: Download PDF