Generative AI improves a wireless vision system that sees through obstructions

Source: MIT News - AI

MIT researchers have spent more than a decade studying techniques that enable robots to find and manipulate hidden objects by “seeing” through obstacles. Their methods utilize surface‑penetrating wireless signals that reflect off concealed items.

Now, the researchers are leveraging generative artificial‑intelligence models to overcome a longstanding bottleneck that limited the precision of prior approaches. The result is a new method that produces more accurate shape reconstructions, which could improve a robot’s ability to reliably grasp and manipulate objects that are blocked from view.

How the Technique Works

1. Partial reconstruction – Reflected wireless signals are used to build an incomplete 3‑D model of a hidden object.
2. Generative AI completion – A specially trained generative AI model fills in the missing parts of the shape.

The team also introduced an expanded system that uses generative AI to reconstruct an entire room, including all the furniture. This system:

• Sends wireless signals from a single stationary radar.
• Captures reflections off humans moving in the space.

“What we’ve done now is develop generative AI models that help us understand wireless reflections. This opens up a lot of interesting new applications, but technically it is also a qualitative leap in capabilities, from being able to fill in gaps we were not able to see before to being able to interpret reflections and reconstruct entire scenes,” says Fadel Adib, associate professor in the Department of Electrical Engineering and Computer Science, director of the Signal Kinetics group in the MIT Media Lab, and senior author of two papers on these techniques. “We are using AI to finally unlock wireless vision.”

Potential Applications

• Warehouse robots – Verify packed items before shipping, reducing waste from product returns.
• Smart‑home robots – Understand a person’s location in a room, improving safety and efficiency of human‑robot interaction.

Publications

• First paper – Link to arXiv PDF

  • Lead author: Laura Dodds (research assistant)
  • Co‑authors: Maisy Lam, Waleed Akbar, Yibo Cheng

• Second paper – Link to arXiv PDF

  • Lead author: Kaichen Zhou (former postdoc)
  • Co‑authors: Laura Dodds, Sayed Saad Afzal

Both papers will be presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Surmounting Specularity

The Adib Group previously demonstrated the use of millimeter‑wave (mmWave) signals to create accurate reconstructions of 3‑D objects hidden from view (e.g., a lost wallet buried under a pile). These waves—the same type used in Wi‑Fi—can pass through common obstructions such as drywall, plastic, and cardboard, and reflect off hidden objects.

The Specularity Problem

• Specular reflection: mmWaves tend to bounce off a surface in a single direction. Large portions of an object’s surface therefore reflect signals away from the sensor, rendering those areas invisible.
• As a result, prior reconstructions captured only the top surface of an object, missing the bottom and sides.

“When we want to reconstruct an object, we are only able to see the top surface and we can’t see any of the bottom or sides,” explains Laura Dodds.

Physics‑based interpretation of reflected signals limited reconstruction accuracy. The new papers overcome this limitation by using a generative AI model to fill in the missing parts.

Training the Generative Model

• Data scarcity – No existing mmWave datasets are large enough to train a high‑performing generative model.
• Synthetic adaptation – The team adapted images from large computer‑vision datasets to mimic mmWave reflection properties (specularity and noise).

“We were simulating the property of specularity and the noise we get from these reflections so we can apply existing datasets to our domain. It would have taken years for us to collect enough new data to do this,” says Maisy Lam.

The physics of mmWave reflections were embedded directly into the adapted data, creating a synthetic dataset that taught the generative AI to produce plausible shape reconstructions.

Wave‑Former System

1. Proposal stage – Generates a set of potential object surfaces based on mmWave reflections.
2. Completion stage – Feeds each proposal to the generative AI model, which completes the shape.
3. Refinement stage – Iteratively refines the surfaces until a full reconstruction is achieved.

Performance

• Reconstructed ~70 everyday objects (cans, boxes, utensils, fruit) hidden behind/under cardboard, wood, drywall, plastic, and fabric.
• Boosted accuracy by nearly 20 % over state‑of‑the‑art baselines.

Seeing “Ghosts”

The same approach was extended to reconstruct entire indoor scenes by leveraging mmWave reflections off humans moving in a room.

• Multipath reflections – Human motion creates secondary reflections: a wave bounces off a person, then off a wall or object, and finally returns to the sensor.
• These secondary reflections are called “ghost signals.”

Typically discarded as noise, ghost signals actually contain valuable information about room layout. By analyzing how these reflections change over time, the system can obtain a coarse understanding of the environment and then refine it into a detailed reconstruction.

“By analyzing how these reflections change over time, we can start to get a coarse understanding of the environme…,” Dodds explains. (sentence truncated in source)

RISE: Refining Scene Reconstruction with mmWave Radar and Generative AI

“The signals we get from mmWave radar are incredibly coarse—just a few points that give us a vague sense of what’s around us. But trying to directly interpret these signals is going to be limited in accuracy and resolution.” — Dodds

How It Works

• Researchers used a training method similar to that employed for generative AI models.
• The AI learns to interpret the coarse scene reconstructions and to understand the behavior of multipath mmWave reflections.
• The model fills in the gaps, refining the initial reconstruction until the scene is fully completed.

Performance

• The system, named RISE, was tested on more than 100 human trajectories captured by a single mmWave radar.
• On average, RISE produced reconstructions about twice as precise as those generated by existing techniques.

Future Directions

• Improve the granularity and detail of the reconstructions.
• Build large foundation models for wireless signals, analogous to GPT, Claude, and Gemini for language and vision, unlocking new applications.

Acknowledgments

This work is supported, in part, by:

• National Science Foundation (NSF)
• MIT Media Lab
• Amazon