3rd Place Undergraduate Research from the KSU College of Computing and Software Engineering
Abstract
The growing concern in data privacy has led to new paradigms in Machine Learning primarily focused around keep data safe and secure. In our research project, we studied Federated Learning, specifically utilizing knowledge distillation and an autoencoder in an attempt to create a sustainable model that could be used in a field such as Heathcare. We propose a Federated Model using the Flower framework, trained on the MedMNIST2D dataset (Organ(A/C/S)MNIST), using Knowledge Distillation as a method of sharing the global model, and a Variational Autoencoder to deal with the problem of Data Heterogeneity that can arise on a distributed network. Our results on a cumulative model are tentative but hope to prove that the idea can be utilized in networks with varying sizes of edge device, usage, and types.
Our Aproach.
Federated Learning
Using Flower, a Federated learning framework in Python, we create a primary server with a collection of clients to simulate what a real-world environment might be like. In FL, the local client models will train on data they collect, which are then transmitted to the global model, where they are then combined and redistributed. We aim to use this methodology due to the rising concerns in Data Privacy and amount of sensitive information being collected on the general public, with the hope to keep data as secure as possible, by never transmitting the data itself.
Knowledge Distillation
The basic idea behind Knowledge Distillation is to train a smaller 'student' model from a larger, more complex, 'teacher' model. The primary goal is to Distill the Knowledge learned from the teacher model to the student while also reducing the size and computational cost. The concept is mainly used in environments with resource-constrained devices as a way to make them more energy efficient and better for edge device users.
Variational Autoencoder
The Autoencoder was created as a measure against Data Heterogeneity. Using this, images do not necessarily have to be the same size if they're collected from different types of machines. There is also the benefit of being able to generate new images if one user does not have a decent amount of data to train on, however this can increase the bias of the model, so usage needs to be monitored.