Cross-Modal Knowledge Distillation for circular manufacturing supply chains for low-power autonomous deployments

Source: Dev.to

Introduction: The Learning Journey That Sparked This Exploration

It all started when I was experimenting with deploying computer vision models on edge devices for a smart recycling facility. I had developed a sophisticated multi‑modal AI system that could identify materials, assess quality, and predict degradation patterns using visual, thermal, and spectral data. The model performed exceptionally well in the lab—achieving 98.7 % accuracy on material classification. But when I deployed it to the actual sorting robots in the facility, I hit a wall: the computational requirements were too high for the low‑power ARM processors running on solar‑charged batteries.

During my investigation of model compression techniques, I discovered that the thermal imaging data—computationally expensive to process—contained patterns that could be approximated from visual data alone, once the model had learned the underlying relationships. This insight led me to cross‑modal knowledge distillation, where a lightweight “student” model trained on a single modality (visual) mimics the behavior of a complex “teacher” ensemble that processes multiple modalities.

The key finding was that knowledge transfer was not just about model compression; it enabled AI systems to operate autonomously in resource‑constrained environments while retaining the intelligence needed for complex decision‑making in circular supply chains.

Technical Background: The Convergence of Multiple Disciplines

The Circular Manufacturing Challenge

Circular manufacturing shifts from linear “take‑make‑dispose” models to closed‑loop systems where materials are continuously recovered, reprocessed, and reused. Autonomous AI in this context must handle:

• Material Identification – Recognizing materials across various states of degradation.
• Quality Assessment – Determining if materials can be reused, repaired, or need recycling.
• Process Optimization – Making real‑time decisions about sorting, routing, and processing.
• Predictive Maintenance – Anticipating equipment failures in remote locations.

Traditionally, each task required a separate AI model processing a different data modality, creating computational bottlenecks for low‑power deployments.

Cross‑Modal Knowledge Distillation Fundamentals

Traditional knowledge distillation compresses a large model into a smaller one while preserving performance. Cross‑modal distillation adds a new dimension: transferring knowledge across different data types.

Three fundamental approaches:

1. Feature‑based distillation – Matching intermediate representations between modalities.
2. Attention‑based distillation – Transferring attention patterns that highlight important regions.
3. Relational distillation – Preserving relationships between different samples or features.

For circular manufacturing applications, a hybrid approach that combines all three methods yields the best results, especially when linking visual appearance to material properties.

Implementation Details: Building the Cross‑Modal Framework

Architecture Overview

A teacher‑student framework with modality‑specific encoders and a shared distillation module proved most effective. Below is the core teacher architecture:

import torch
import torch.nn as nn
import torch.nn.functional as F

class MultiModalTeacher(nn.Module):
    """Teacher model processing multiple modalities"""
    def __init__(self, visual_dim=512, thermal_dim=256, spectral_dim=128):
        super().__init__()

        # Modality‑specific encoders
        self.visual_encoder = self._build_visual_encoder(visual_dim)
        self.thermal_encoder = self._build_thermal_encoder(thermal_dim)
        self.spectral_encoder = self._build_spectral_encoder(spectral_dim)

        # Cross‑modal fusion
        self.fusion_layer = nn.Sequential(
            nn.Linear(visual_dim + thermal_dim + spectral_dim, 512),
            nn.ReLU(),
            nn.Dropout(0.3)
        )

        # Task‑specific heads
        self.material_classifier = nn.Linear(512, 50)   # 50 material types
        self.quality_regressor = nn.Linear(512, 1)      # Quality score
        self.degradation_predictor = nn.Linear(512, 10) # Degradation states

    def forward(self, visual_input, thermal_input, spectral_input):
        visual_features = self.visual_encoder(visual_input)
        thermal_features = self.thermal_encoder(thermal_input)
        spectral_features = self.spectral_encoder(spectral_input)

        # Concatenate and fuse
        fused = torch.cat([visual_features, thermal_features, spectral_features], dim=1)
        fused = self.fusion_layer(fused)

        return {
            'material': self.material_classifier(fused),
            'quality': self.quality_regressor(fused),
            'degradation': self.degradation_predictor(fused),
            'features': {
                'visual': visual_features,
                'thermal': thermal_features,
                'spectral': spectral_features,
                'fused': fused
            }
        }

The Lightweight Student Model

The student processes only visual data but learns to approximate the teacher’s multi‑modal understanding:

class VisualOnlyStudent(nn.Module):
    """Student model using only visual input"""
    def __init__(self, visual_dim=256, hidden_dim=128):
        super().__init__()

        # Efficient visual encoder (MobileNet‑like)
        self.visual_encoder = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size=3, stride=2, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU6(),
            nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU6(),
            nn.AdaptiveAvgPool2d((1, 1)),
            nn.Flatten(),
            nn.Linear(64, visual_dim)
        )

        # Compact task heads
        self.material_classifier = nn.Linear(visual_dim, 50)
        self.quality_regressor = nn.Linear(visual_dim, 1)

    def forward(self, visual_input):
        visual_features = self.visual_encoder(visual_input)

        return {
            'material': self.material_classifier(visual_features),
            'quality': self.quality_regressor(visual_features),
            'features': visual_features
        }

Cross‑Modal Distillation Loss

A composite loss function combines soft‑target matching, feature alignment, and relational constraints:

class CrossModalDistillationLoss(nn.Module):
    def __init__(self, temperature=3.0, alpha=0.7, beta=0.2, gamma=0.1):
        super().__init__()
        self.temperature = temperature
        self.alpha = alpha
        self.beta = beta
        self.gamma = gamma
        self.kl_div = nn.KLDivLoss(reduction='batchmean')
        self.mse = nn.MSELoss()
        self.cosine = nn.CosineEmbeddingLoss()

    def forward(self, teacher_outputs, student_outputs):
        # 1. Soft target (logits) distillation
        t_logits = teacher_outputs['material'] / self.temperature
        s_logits = student_outputs['material'] / self.temperature
        loss_soft = self.kl_div(F.log_softmax(s_logits, dim=1),
                                F.softmax(t_logits, dim=1)) * (self.temperature ** 2)

        # 2. Feature‑based distillation (visual features)
        loss_feat = self.mse(student_outputs['features'],
                             teacher_outputs['features']['visual'])

        # 3. Relational distillation (pairwise similarity)
        t_feat = teacher_outputs['features']['visual']
        s_feat = student_outputs['features']
        t_sim = F.normalize(t_feat, dim=1) @ F.normalize(t_feat, dim=1).t()
        s_sim = F.normalize(s_feat, dim=1) @ F.normalize(s_feat, dim=1).t()
        loss_rel = self.mse(s_sim, t_sim)

        # Composite loss
        total_loss = self.alpha * loss_soft + self.beta * loss_feat + self.gamma * loss_rel
        return total_loss

The loss combines three terms:

• Soft target loss (loss_soft) aligns the student’s class predictions with the teacher’s softened logits.
• Feature loss (loss_feat) forces the student’s visual embeddings to match the teacher’s visual embeddings.
• Relational loss (loss_rel) preserves pairwise relationships among samples, encouraging the student to capture the teacher’s internal geometry.