Hierarchical Cross-Attention Transformer for Non-contact Multimodal Pain Classification using Remote Physiological Signals and Visual Features

Abstract

Automatic pain recognition is critical for diagnosis, treatment planning, and patient recovery. Physiological signals offer objective, involuntary markers of pain but require physical sensors, which can cause discomfort or pose risks in patients such as neonates or the elderly. To mitigate these drawbacks, non-contact alternatives, including facial expressions and remotely captured physiological responses, have been increasingly explored. However, existing methods primarily focus on cardiac signals while overlooking respiratory information, an autonomic indicator of pain. Moreover, fusion of visual and remote physiological features remains underexplored, with most approaches relying on shallow mechanisms or contact-based supervision, limiting scalability. To address these limitations, we propose a non-contact framework that jointly leverages remotely extracted pulse and respiratory signals along with facial features for pain recognition. The pulse is obtained via the Eulerian method, and the respiratory signal through Lagrangian motion analysis. Our model employs a hierarchical Transformer-based architecture with modality-specific Transformers for intra-modal encoding, symmetric cross-attention for inter-modal interaction, and a fusion Transformer for aggregating complementary information. The framework operates without requiring contact-based physiological ground truth during training. Experimental results show that our framework outperforms existing methods in both binary and multiclass classification tasks. Code can be accessed at: https://github.com/AnupKumarGupta/HCAT-Pain. © 2026 IEEE.