Design of CNN for multi-class classification of videos and images

Abstract

Classification of images and action in a video are a very challenging task requiring high computation resources. Image classification in itself is a complex task where multiple convolution and pooling layers extract features from the images and these features go through a set of fully connected layers that classify the features extracted into different classes, this task becomes even more complex and resource heavy with the addition of another dimension i.e. time for video classification problems. Models like VGG, ResNet and NASNetLarge are one of the go-to models for image classification as they provide great results with the complexity of the model that they bring. Using the ResNet and NASNetLarge we proposed a new model for image classification that takes the best from ResNet and NASNetLarge structure, the proposed model is essentially a ResNet50 model with better feature extraction capability from NASNetLarge. For the task of Video classification we went through a number of models and their combination with each other, SlowFast network is a model that performs 3D convolution on the video frames with different frame rates and combines them to get the class of the input. SlowFast uses ResNet50 model as it’s base so we simply used the Modified architecture from image classification here. Another model that we reviewed was VideoCapsuleNet and U-Net, combined these models as well to create a better video classification model. Finally, a SlowFast model implemented in PyTorch with modification from U-Net.