[Paper] Predicting Contextual Informativeness for Vocabulary Learning using Deep Learning

Source: arXiv - 2602.18326v1

Overview

This paper presents a deep‑learning pipeline that automatically picks the most informative sentences (contexts) to teach high‑school students new vocabulary words. By comparing three increasingly sophisticated models, the authors show how modern language embeddings—when fine‑tuned with teacher feedback—can generate a cheap, large‑scale supply of “near‑perfect” teaching examples.

Key Contributions

• Three‑tiered modeling comparison:
  1. Unsupervised similarity using MPNet contextual embeddings.
  2. Supervised fine‑tuning of Qwen‑3 embeddings with a nonlinear regression head.
  3. Hybrid model that adds handcrafted linguistic features to the supervised Qwen‑3 system.
• Retention Competency Curve (RCC): a new visual metric that simultaneously shows (a) how many “good” contexts are discarded and (b) the ratio of good‑to‑bad contexts retained, giving a single, intuitive performance lens.
• Empirical breakthrough: The hybrid model (iii) achieves a good‑to‑bad ratio of 440 while only discarding 30 % of the truly useful contexts (i.e., it keeps 70 % of the good ones).
• Practical pipeline: Demonstrates that a modern embedding model, guided by modest human supervision, can produce a low‑cost, high‑quality corpus of teaching examples for a wide range of target words.

Methodology

1. Data collection – A corpus of sentences containing target vocabulary items was curated, and each sentence was manually labeled by language teachers as good (highly informative for learning) or bad (low utility).
2. Embedding generation –
   • Unsupervised: MPNet was used to generate uniform contextual embeddings for every sentence.
   • Supervised: Qwen‑3, a large language model, was fine‑tuned on the labeled data. Its embeddings were then passed through a small nonlinear regression head that predicts a “informativeness score.”
3. Feature augmentation – For model (iii), the authors added handcrafted features such as sentence length, lexical diversity, presence of synonyms/antonyms, and syntactic simplicity. These were concatenated with the Qwen‑3 embeddings before the regression head.
4. Training & evaluation – The models were trained to minimize mean‑squared error between predicted scores and the binary teacher labels. Performance was assessed using the Retention Competency Curve, which plots the proportion of discarded good contexts against the resulting good‑to‑bad ratio.

Results & Findings

• The RCC shows that model (iii) dominates the other two across the entire trade‑off spectrum.
• Adding linguistic heuristics to the neural embeddings yields a ~2× boost in the good‑to‑bad ratio over pure fine‑tuning, confirming that domain‑specific cues still matter.
• The system can generate thousands of high‑quality contexts per word at a fraction of the cost of manual curation.

Practical Implications

• Curriculum designers can plug the model into existing authoring tools to auto‑suggest example sentences, dramatically reducing the time teachers spend hunting for suitable contexts.
• EdTech platforms (e.g., language‑learning apps, adaptive tutoring systems) can use the pipeline to personalize vocabulary exposure: the model can rank candidate sentences on‑the‑fly based on a learner’s proficiency level.
• Content creators (e.g., textbook publishers) can quickly assemble large, diverse example banks for new word lists, ensuring each entry is pedagogically sound.
• Because the approach relies on a modest amount of labeled data, schools with limited resources can fine‑tune the system for their own curricula or regional dialects.

Limitations & Future Work

• Label sparsity: The training set still depends on expert annotations; scaling to thousands of words may require semi‑supervised or active‑learning strategies.
• Domain bias: The corpus used for experiments is primarily academic English; performance on informal or domain‑specific texts (e.g., social media, technical manuals) remains untested.
• Interpretability: While the handcrafted features improve performance, the model’s decision process is still largely a black box; future work could explore explainable AI techniques to surface why a context is deemed “good.”
• Multilingual extension: The study focuses on English; extending the pipeline to other languages would involve handling different morphological and syntactic cues.

Authors

• Tao Wu
• Adam Kapelner

Paper Information

• arXiv ID: 2602.18326v1
• Categories: cs.CL
• Published: February 20, 2026
• PDF: Download PDF