Raw Waveform Modeling for Speech Emotion Recognition Across Diverse Datasets

Abstract

Human speech is a fundamental mode of communication has been extensively studied in speech processing. Speech emotion recognition (SER), a subfield of artificial intelligence and signal processing, aims to automatically identify and interpret emotional states from spoken language. While prior research has predominantly focused on specific datasets and limited handcrafted features. This work introduces a dual modeling approach that integrates raw waveform analysis with mel-frequency cepstral coefficients (MFCC) layered convolutional neural network (CNN) architectures, enabling the capture of both temporal and perceptual features for robust SER. To address real-world challenges, datasets were augmented by adding noise, and experiments were conducted across four benchmark datasets to evaluate the model’s generalizability in noisy conditions. The proposed model achieved accuracies of 94% on Surrey audio-visual expressed emotion database (SAVEE), 91.25% on Ryerson audio-visual database of emotional speech and song (RAVDESS), 84.75% on Berlin database of emotional speech (EMODB), and 79% on interactive emotional dyadic motion capture (IEMOCAP) using raw speech in the first experiment. Performance was even better in the second experiment of augmented data with additive white Gaussian noise (AWGN) with 92.5% on EMODB, 92.75% on RAVDESS and 90.25% on IEMOCAP, with SAVEE performing at 85.5%. Experimental results demonstrate that the proposed hybrid approach outperforms conventional models, highlighting its potential for reliable emotion classification in both general and healthcare contexts. © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2026.