Self-supervised spontaneous latent-based facial expression sequence generation

Yap, Chuin Hong ORCID: https://orcid.org/0000-0003-2251-9308, Yap, Moi Hoon ORCID: https://orcid.org/0000-0001-7681-4287, Davison, Adrian K and Cunningham, Ryan ORCID: https://orcid.org/0000-0001-6883-6515 (2023) Self-supervised spontaneous latent-based facial expression sequence generation. IEEE Open Journal of Signal Processing, 4. pp. 304-312. ISSN 2644-1322

Preview

Published Version Available under License Creative Commons Attribution Non-commercial No Derivatives. Download (17MB) | Preview

Abstract

In this paper, we investigate the spontaneity issue in facial expression sequence generation. Current leading methods in the field are commonly reliant on manually adjusted conditional variables to direct the model to generate a specific class of expression. We propose a neural network-based method which uses Gaussian noise to model spontaneity in the generation process, removing the need for manual control of conditional generation variables. Our model takes two sequential images as input, with additive noise, and produces the next image in the sequence. We trained two types of models: single-expression, and mixed-expression. With single-expression, unique facial movements of certain emotion class can be generated; with mixed expressions, fully spontaneous expression sequence generation can be achieved. We compared our method to current leading generation methods on a variety of publicly available datasets. Initial qualitative results show our method produces visually more realistic expressions and facial action unit (AU) trajectories; initial quantitative results using image quality metrics (SSIM and NIQE) show the quality of our generated images is higher. Our approach and results are novel in the field of facial expression generation, with potential wider applications to other sequence generation tasks.