Building Generative Machine Learning Models to Analyze Microscope Images of Material Systems

Student Name: Himani Anilkumar Mishra Home Institution: Cornell University NNCI Site: ShyNE @ Northwestern University Principal Investigators: Professor Vinayak Dravid and Dr. Roberto dos Reis REU Mentors: Alfred Yan Abstract: Generative machine learning models offer a promising approach to developing qualitative understanding of microscopic images and the techniques used to obtain them. Current models, such as OpenAI’s CLIP and DALL-E, have successfully learned to classify and generate images from their textual descriptions. In materials science, these models can be trained on microscopic images of materials to gain insights into their properties and suggest synthesis methods for specific applications. This project aims to provide rich, qualitative descriptions of scanning electron microscope (SEM) images and the parameters used to acquire them, as well as to auto-generate images from textual descriptions. The model’s insights into materials systems can guide the specification of synthesis parameters (e.g. temperature, time, etc.) to achieve targeted functionalities. The chosen material class to test this project is Metal-Organic Frameworks (MOFs), known for their porous, tunable structures with applications ranging from carbon capture to catalysis. Given their wide applications, it is worthwhile to determine a system to analyze them. To acquire training data, images of various MOFs (UiO-66, NU-1000, MIL-101 Cr, and MIL-101 Fe) are obtained using Hitachi S-3400 SEM at different operating voltages, magnifications and other SEM settings. To enhance the model quality despite limited data, data augmentation techniques such as blurring and stretching are employed. The model will be trained to predict the material type, size, shape, morphology, the SEM settings used to capture the image, and potential applications of the material given an image input.