From Pixels to Physical Therapy: Building a Real-Time Pose Correction System with MediaPipe and React

Source: Dev.to

The Architecture: From Frame to Feedback

To achieve “real‑time” performance (30+ FPS) in a browser, we need a pipeline that minimizes latency. By using WebAssembly (WASM), MediaPipe processes video frames at near‑native speeds.

graph TD
    A[Webcam Stream] --> B[React Camera Component]
    B --> C[MediaPipe Pose Engine]
    C --> D{Keypoint Detection}
    D --> E[Angle Calculation Logic]
    E --> F[Visual Overlay / Canvas]
    E --> G[Speech Synthesis Feedback]
    G --> H[User: "Straighten your back!"]

Prerequisites

• MediaPipe Pose – high‑fidelity body tracking.
• React – UI layer.
• WebAssembly – run the ML models efficiently in‑browser.

Step 1: Initializing the Pose Engine

Set up the MediaPipe Pose listener, which detects 33 3D landmarks on the human body.

import { Pose } from "@mediapipe/pose";
import * as cam from "@mediapipe/camera_utils";

const pose = new Pose({
  locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/pose/${file}`,
});

pose.setOptions({
  modelComplexity: 1,
  smoothLandmarks: true,
  minDetectionConfidence: 0.5,
  minTrackingConfidence: 0.5,
});

Step 2: The Math – Calculating Joint Angles

For physical therapy we need joint angles. For a squat, calculate the angle between hip, knee, and ankle.

/**
 * Calculates the angle between three points (A, B, C)
 * B is the vertex (e.g., the knee)
 */
function calculateAngle(a, b, c) {
  const radians =
    Math.atan2(c.y - b.y, c.x - b.x) - Math.atan2(a.y - b.y, a.x - b.x);
  let angle = Math.abs((radians * 180.0) / Math.PI);
  if (angle > 180.0) angle = 360 - angle;
  return angle;
}

// Example usage in the MediaPipe results callback:
const kneeAngle = calculateAngle(
  results.poseLandmarks[24], // Hip
  results.poseLandmarks[26], // Knee
  results.poseLandmarks[28]  // Ankle
);

Step 3: Real‑Time Feedback Logic

Turn the angle values into actionable voice feedback.

const provideFeedback = (angle) => {
  const synth = window.speechSynthesis;

  if (angle > 160) {
    // Standing up too early
    return "Lower your hips more.";
  } else if (angle < 90) {
    // Too deep for some patients
    const utter = new SpeechSynthesisUtterance(
      "Great depth, keep your back straight!"
    );
    synth.speak(utter);
  }
};

Scaling to Production

A weekend prototype is a great start, but production‑grade remote physiotherapy platforms need:

• Data persistence and analytics.
• HIPAA‑compliant storage and transmission.
• Advanced filtering for noisy environments.
• Multi‑person tracking or integration with HL7/FHIR standards.

For in‑depth guidance, explore the articles at wellally.tech/blog.

Conclusion: Your Browser Is a Kinesiologist

By combining MediaPipe’s skeleton tracking with simple trigonometric functions, we can build a system that “sees” and “corrects” human movement directly in the browser. This demonstrates the power of modern Vision AI—bringing specialized expertise to anyone with a webcam.

What’s next?

• Add a rep counter using a state machine.
• Integrate a Chart.js dashboard to track knee mobility over time.
• Share your progress with the community!