A feature-enhanced shift graph convolutional network and its application in skeleton-based action recognition

Abstract

Human action recognition is the process of automatically identifying and classifying human actions in a video sequence. It involves analyzing the motion and appearance of humans in the video and recognizing the action they are performing. Human actions can be represented using various data modalities, such as RGB videos, skeleton graphs, depth sequences and heat maps. In recent years, skeleton-based action recognition has drawn a lot of attention in the area of computer vision. To extract skeletal data, pose estimation algorithms are used on action videos to track the key joints involved in the action. These joints are connected with edges representing the bones involved in the action, thus forming a graph structure of the human action. Skeleton data is lightweight as compared to video data, and is more robust against changes in appearance, lighting conditions, background clutter and camera viewpoints. Among existing methods, Graph Convolutional Networks (GCNs) have achieved exceptional results as they are highly efficient in feature extraction from non-euclidean or irregular data. However, most existing GCN-based methods are computationally expensive and have inflexible receptive fields, due to which their expressiveness is limited. As a result, focus has shifted towards building lightweight architectures which require fewer parameters. One such method is Shift-GCN [1], which uses shift graph operations that are both lightweight and increase the flexibility of receptive fields in both spatial and temporal dimensions. However, although this method captures non-local and distant spatial relationships in a lightweight and more efficient manner, it does not perform well on fine-grained actions that have subtle differences and require capturing graph connection information.