Did My LoRA Learn Tenacious Style—or Just Memorize Augmented Patterns?

Source: Dev.to

In Week 11 of Tenacious‑Bench we trained a LoRA adapter on Tenacious‑style B2B sales emails using Supervised Fine‑Tuning (SFT).
We observed a real performance lift: **Δ A = +0.263 (p  “The adapter improved predictive behavior on measured data; generalization vs. memorization requires additional diagnostics.”

That is stronger science and better engineering.

6) Minimal diagnostics to separate style learning from memorization

A) Grouped hold‑out by original family

• Do not split augmentation siblings across train/hold‑out.
• Keep all variants of one original together in one split.

Interpretation:

• Stable performance on grouped hold‑out → stronger evidence of true style learning.
• Large drop → evidence of augmentation‑family memorization.

B) Gradient‑norm breakdown by LoRA module

• Log gradient norms for LoRA parameters and aggregate by:
  • q/k/v/o
  • gate/up/down

This doesn’t “prove style” alone, but it makes your mechanism claim concrete: where did training pressure concentrate?

7) Practical conclusion for FDE fine‑tuning work

The issue generalizes to any narrow, augmented SFT project (sales writing, summarization, code style, domain formatting):

• Loss convergence is necessary.
• Benchmark gain is valuable.
• Neither alone proves intended behavior learning.

If you want to claim “learned policy,” add grouped hold‑out and module‑level gradient diagnostics as standard evidence.

Final takeaway

Your LoRA adapter likely learned a useful steering update.
However, with heavy augmentation concentration, the safest conclusion is:

“We improved next‑token policy on this distribution; we are validating whether that policy generalizes beyond augmentation families.”

This framing is honest, technically grounded, and production‑defensible.