SEM Generation

Dates

November 2023 - Present

Role

Builder / Researcher

diffusers

NumPy

Optuna

OpenCV

scikit-image

Matplotlib

seaborn

PyTorch

CUDA

Project

Predictive and Generative Modeling of Mixed-Dimensional Aerogels with Programmable Properties

Implementation

After trying a variety of methods, we settled on a conditioned diffusion model, based on a U-Net architecture. The "prompt" embedding included composition labels, as well as predicted structural parameters for each recipe. After generating images at 128x128 pixel resolution, an 8x upscale (fine-tuned model based on ESRGAN) was used to produce high-resolution images. This enabled the same image processing, labeling, and ealuation tools to be applied to the generated images as their experimental, real, counterparts. Success, or accuracy, of the generation model was evaluated by comparing the extracted structural parameters of real and generated images.

Impact

This pipeline enabled us to perform virtual optimization of our samples by composition, avoiding expensive and timely physical experiments, increasing our throughput Additionally, the stochastic nature of diffusion models enabled the generation of diverse microstructure, allowing for the observation of a distribution of characteristics for any given input label, increasing the fidelity, robustness, and accuracy of simulations.

@U-Net CNNs Paper

@HF Space (inactive)

SEM Game - Spot the Generated Images!

Click to see the answers

This is a game we have played when presenting in academic settings. It highlights the quality and feasibility of our model and approach. Xs denote generated (fake) images.

Example Generated Microstructures as seen via SEM.

By using a different random seed we are able to generate a diverse representation, or population, of microstructures for any given sample, all using the same prompt.