Pulse estimation from face videos using transformer

Abstract

Remote photoplethysmography is used for non-contact heart rate (HR) estimation by measuring the skin color variations from face videos. These variations are subtle and easily get affected by noise. Earlier deep learning based rPPG estimators have used intrinsic rPPG feature representation for mitigating noise. They utilize information from the entire facial region for denoising. Unfortunately, they fail to provide correct HR estimation as different facial regions contain different noise characteristics induced by various facial expressions. Moreover, the convolutional neural network based ar chitectures are affected by local noise characteristics and the sequential architectures fail to preserve long temporal dependencies resulting in incorrect HR estimation. To address these issues, we propose PERFORMER, that is, Pulse Estimation from Face Videos using Transformer. It benefits from the multi-head attention mechanism that enables the feature subspace learning for extracting the multiple correlations, among the color variations, that correspond to the periodic pulse. Moreover, the ability of the Transformer to preserve global context suppresses the effect of local noise charac teristics. Furthermore, we have proposed novel signal embedding that isolates rPPG information present in a facial region, thereby preventing contamination from noise in other facial regions. The embeddings enable our architecture to enhance the rPPG fea ture representation and suppress noise. We have thoroughly investigated the efficacy of synthetic temporal signals and data augmentations for introducing diverse training set for improving the generalization of our architecture. Experiments on extensively utilized datasets, COHFACE and UBFC-rPPG, demonstrate that our architecture outperforms previous well-known architectures.