Generating SEM Images from Segmentation Masks

Source: Dev.to

Acknowledgements

We would like to thank our mentors, Asaf Nisani and Yoav Lebendiker, for their guidance throughout the project.

Dataset Preparation

Our project aimed to generate highly specific SEM wafer images based on user input. To support this approach, we needed a high‑quality paired dataset of SEM images and their corresponding segmentation masks. Since such data does not naturally exist, we generated it ourselves.

We converted the MIIC (Microscopic Images of Integrated Circuits) dataset—originally designed for anomaly detection and image inpainting—into segmentation maps using SAM‑2. The final dataset consisted of paired samples of:

• SEM images
• Corresponding segmentation masks

The images were taken from the MIIC dataset for anomaly detection and image inpainting.

Model Training

Pix2Pix

Pix2Pix is well‑suited for scenarios where paired training data is available. It learns a direct mapping from segmentation masks to SEM images using a generator‑discriminator framework.

• Training data: 3,000 SEM images and 3,000 segmentation masks (split into training and test sets)
• Training details: Standard Pix2Pix configuration

Evaluation Metrics

These results indicate that Pix2Pix generated sharp, realistic SEM images with strong structural similarity to the ground truth.

CycleGAN

CycleGAN is designed for unpaired image translation. Although our dataset was paired, we included CycleGAN in the comparison due to its flexibility and widespread use in scientific imaging tasks.

• Training epochs: 200
• Optimizations applied:
  • Reduced identity loss to better preserve grayscale SEM texture
  • Increased discriminator augmentation to mitigate overfitting

Evaluation Metrics

While the model improved during training, its outputs remained noticeably weaker than those produced by Pix2Pix, which benefited from the supervised setup.

Limitations of the Segmentation‑Based Workflow

Although segmentation masks provide an intuitive way for users to describe structure, the masks in our dataset were not randomly colored. Their colors were automatically assigned by SAM‑2, which applies consistent and specific color patterns for background and different structures. A new user drawing their own segmentation mask cannot know these exact color rules. Consequently, user‑generated masks may differ from the distribution the model was trained on, leading to less accurate and less reliable outputs even when the structural layout is correct.

Classification Evaluation of Real vs. Generated Images

To further assess realism, we trained a separate classification model to distinguish between real SEM images and generated ones. The idea was simple: if the synthetic images are truly realistic, the classifier should struggle to tell them apart and achieve accuracy close to 0.5.

• Training data: Balanced set of real SEM images and images generated by Pix2Pix and CycleGAN.
• Results:
  • 0.96 accuracy on Pix2Pix‑generated images
  • 0.99 accuracy on CycleGAN‑generated images

These results suggest that, despite strong visual quality and favorable similarity metrics, the generated images still contain subtle artifacts or patterns that allow a neural network to reliably distinguish them from real SEM data.

Future Work

• Explainability: Apply techniques to understand which features the classifier relies on when separating real and generated images, revealing hidden biases, texture inconsistencies, or frequency‑domain artifacts.
• Model Refinement: Further train generative models, explore architectural adjustments, or tune loss functions to reduce detectable differences.
• Residual Noise Injection: Train a model that learns the residual noise patterns specific to real SEM images and injects this signal into generated images as a post‑processing step.
• Downstream Task Validation: Train a real‑world SEM task model on generated images and evaluate it on real SEM test data. Successful performance transfer would validate synthetic SEM generation as an effective tool for data augmentation and model training.