Is That Mole Dangerous? Build a Real-Time Skin Lesion Classifier with WebGPU and EfficientNetV2 🚀

Source: Dev.to

Introduction

Healthcare is moving to the edge. Imagine being able to screen a suspicious skin lesion directly in your browser with the privacy of local execution and the speed of a native app. Thanks to the WebGPU API and TensorFlow.js, we can now run heavy‑duty computer vision models like EfficientNetV2 with unprecedented performance.

In this tutorial we’ll dive deep into building a high‑performance Edge AI application for skin lesion classification. We will leverage WebGPU for hardware‑accelerated inference, ensuring that sensitive health data never leaves the user’s device. If you’ve been looking to master computer vision in the browser or want to see how the next generation of web graphics APIs can be used for deep learning, you’re in the right place! 💻🥑

Data Flow Diagram

graph TD
    A[User Camera Stream] --> B[React Canvas Wrapper]
    B --> C{WebGPU Supported?}
    C -- Yes --> D[TF.js WebGPU Backend]
    C -- No --> E[TF.js WebGL/CPU Fallback]
    D --> F[EfficientNetV2 Inference]
    F --> G[Probability Distribution]
    G --> H[Medical Priority Assessment]
    H --> I[UI Alert/Recommendation]

Prerequisites

• React 18+ for the frontend structure.
• TensorFlow.js (@tensorflow/tfjs) with the WebGPU extension.
• A fine‑tuned EfficientNetV2 model (converted to model.json format).
• A browser that supports WebGPU (Chrome 113+ or Edge).

WebGPU Initialization

WebGPU is the successor to WebGL, offering much lower overhead and better access to GPU compute capabilities. In TensorFlow.js, initializing it is straightforward but requires an asynchronous check.

import * as tf from '@tensorflow/tfjs';
import '@tensorflow/tfjs-backend-webgpu';

async function initializeAI() {
  try {
    // Attempt to set the backend to WebGPU
    await tf.setBackend('webgpu');
    await tf.ready();
    console.log("🚀 Running on WebGPU: The future is here!");
  } catch (e) {
    console.warn("WebGPU not available, falling back to WebGL.");
    await tf.setBackend('webgl');
  }
}

Model Loading

EfficientNetV2 is perfect for this task because it offers state‑of‑the‑art accuracy while being significantly faster and smaller than its predecessors. We’ll load a model fine‑tuned on the ISIC (International Skin Imaging Collaboration) dataset.

const MODEL_URL = '/models/efficientnet_v2_skin/model.json';

const useSkinClassifier = () => {
  const [model, setModel] = React.useState(null);

  React.useEffect(() => {
    const loadModel = async () => {
      const loadedModel = await tf.loadGraphModel(MODEL_URL);
      // Warm up the model to avoid first‑inference lag
      const dummyInput = tf.zeros([1, 224, 224, 3]);
      loadedModel.predict(dummyInput);
      setModel(loadedModel);
    };
    loadModel();
  }, []);

  return model;
};

Prediction Logic

The core logic involves capturing the video frame, resizing it to 224×224 (the expected input for EfficientNetV2), and normalizing the pixel values.

const predict = async (videoElement, model) => {
  if (!model || !videoElement) return;

  const result = tf.tidy(() => {
    // 1. Convert video frame to tensor
    const img = tf.browser.fromPixels(videoElement);

    // 2. Preprocess: Resize and Normalize to [-1, 1] or [0, 1]
    const resized = tf.image.resizeBilinear(img, [224, 224]);
    const offset = tf.scalar(127.5);
    const normalized = resized.sub(offset).div(offset).expandDims(0);

    // 3. Inference
    return model.predict(normalized);
  });

  const probabilities = await result.data();
  const topResult = getTopClass(probabilities);

  // Clean up tensors
  tf.dispose(result);

  return topResult;
};

Production Considerations

Building a prototype is easy; building a production‑grade medical screening tool is hard. You need to handle:

• Lighting variations
• Motion blur
• Out‑of‑distribution (OOD) data (e.g., when a user points the camera at a dog instead of a skin lesion)

Pro Tip: For production environments, use model quantization to reduce bundle size and Web Workers to keep the UI thread buttery smooth.

If you are looking for advanced architectural patterns for deploying AI in high‑stakes environments, check out the technical deep‑dives at the WellAlly Blog. They have resources on optimizing TensorFlow models for enterprise‑scale React applications and handling complex state for real‑time vision pipelines.

Class Mapping & Priority Assessment

Our system isn’t just giving a label; it’s assessing “Medical Priority.” We map classes like Melanoma to high priority and Nevus to low priority.

const CLASSES = {
  0: { name: 'Actinic keratoses', priority: 'Medium' },
  1: { name: 'Basal cell carcinoma', priority: 'High' },
  2: { name: 'Benign keratosis', priority: 'Low' },
  3: { name: 'Dermatofibroma', priority: 'Low' },
  4: { name: 'Melanoma', priority: 'Urgent' },
  5: { name: 'Melanocytic nevi', priority: 'Low' },
  6: { name: 'Vascular lesions', priority: 'Medium' }
};

const getTopClass = (probs) => {
  const maxIdx = probs.indexOf(Math.max(...probs));
  return {
    ...CLASSES[maxIdx],
    confidence: probs[maxIdx]
  };
};

Conclusion

We’ve successfully built a localized, hardware‑accelerated skin lesion classifier. By using EfficientNetV2 and WebGPU, we achieve near‑native performance without the user ever needing to download an “App.”

Note: AI‑based screening tools are meant to assist, not replace, professional medical diagnosis. Always include a disclaimer in your UI! 🩺

What’s Next?

• Try implementing quantization (Int8 or Float16) to see how it affects WebGPU performance.
• Check out WellAlly’s advanced guides for more insights on scaling these types of applications.

Have you experimented with WebGPU yet? Drop a comment below and let us know your thoughts! 👇