University of Toronto
In this project, I aimed addressed the growing concerns surrounding data protection in generative models, a crucial class of machine learning models extensively employed in artificial intelligence applications. These models, designed to learn and mimic patterns in data, have shown the inadvertent memorization of sensitive information, prompting the need for privacy measures. Differentially private algorithms have been proposed, but the inherent trade-off between privacy and data quality remains a significant challenge in the field.
To tackle this problem, I conducted a comparative analysis of differentially private generative models, focusing on Conditional Generative Adversarial Network (CGAN), Conditional Variational Autoencoder (CVAE), and the Diffusion Model. The study aimed to evaluate the impact of privacy budgets on data quality using various metrics, striving to determine the optimal model striking a balance between quality and privacy preservation.
The results revealed significant differences between outputs of the Generative Adversarial Network (GAN) and Variational Autoencoder (VAE). While GAN produced sharp images with visual clarity, VAE generated images with a higher degree of continuity and realism, albeit with some distortion. The study also explored Fréchet Inception Distance (FID) scores as a metric for gauging the realism of generated images. Notably, VAE consistently exhibited lower FID scores than GAN, positioning it as the preferred choice for realism.
The robustness against membership inference attacks was evaluated, and unexpected accuracy results raised concerns about potential TensorFlow API issues or overfitting. Despite this, the research demonstrated that non-differentially private generative models, specifically the Diffusion model, outperformed their counterparts in terms of data quality.
In conclusion, the project shed light on the delicate balance between privacy and accuracy in generative models. The results supported previous research findings, emphasizing the need for further investigation into optimal model selection for practical applications, considering training speed, resource requirements, and addressing open issues such as the limitations of existing metrics in capturing human judgment about data quality.