Human behavior analysis using smartphone sensor data

Abstract

Over the past decade, smartphones have become an integral part of our daily life. With the advancements in technology, smartphones have become feature-rich, affordable and very sophisticated containing several built-in sensors such as orientation sensor, accelerometer sensor and gyroscope sensor. Equipping smartphones with intelligence has been a topic of interest to researchers working in diverse fields such as biometrics, healthcare, and financial services.Readings from dedicated body-worn inertial sensors have been shown to carry information useful for various tasks including human gait recognition, sports activity recognition, fall detection and health monitoring. In general, the major disadvantage of such approaches is the cost of employing dedicated sensors for data acquisition. In addition, these approaches are less user-friendly as they are likely to cause inconvenience to the users. The possibility of overcoming these drawbacks by utilizing the built-in sensors in smartphones motivated us to perform the analysis of smartphone sensor data to extract information regarding human behavioral characteristics. The analysis of human behavior has been proven to be effective in various applications including biometric-based user authentication, smart spaces, human-machine interactions, physical activity recognition and surveillance. The key advantage is that the human behavior can be captured unobtrusively without requiring a conscious effort on the part of the user. Therefore, the human behavior-based intelligence for smartphones is quite promising.The prime objective of this thesis is to enhance the capabilities of smartphone-based biometric recognition and smartphone-based health monitoring systems through the analysis of human behavioral information acquired from the smartphone’s built-in sensors. Specifically, we analyze the behavioral information with the objective to develop efficient approaches for biometric authentication, gender recognition and physical activity recognition. Nowadays, it is quite common to see people performing banking transactions and storing sensitive information on smartphones. Therefore, it is extremely important that these devices are able to perform user authentication. In this thesis, we propose an approach for user authentication in smartphones using behavioral biometrics. This approach analyses the behavioral data, which is collected while the user performs different gestures during his/her interaction with the device. In addition to the touch pointlocations, the proposed approach utilizes the information from the built-in accelerometer sensor and orientation sensor in the smartphone. The modified Hausdorff distance (MHD) is employed for matching features of the gestures performed by the user. The biometric authentication performance can be improved by supplementing traditional biometric information with soft biometrics like gender, age, height, weight, and ethnicity. Such soft biometric attributes can also be exploited in various applications including surveillance, human-machine interactions, and smart spaces. In this thesis, we perform gender recognition while a user interacts with the device as well as when a user walks with a smartphone in the trouser pocket. Our first approach performs gender recognition by capturing the behavioral information while the user interacts with the smartphone’s touchscreen. The behavioral data comprising readings from the accelerometer sensor, gyroscope sensor and orientation sensor are acquired during the user’s interaction with the device. Two-dimensional attribute maps are then formed using the set of attributes. GIST descriptors computed on these images provide thediscriminatory information for gender recognition. Our second approach for gender recognition utilizes gait information collected using built-in sensors in the smartphone. Specifically, readings from the accelerometer sensor and gyroscope sensor are captured while the user walks with the smartphone in the trouser pocket. We propose a histogram of gradient-based approach to extract features useful for gender recognition. Smartphone-based activity recognition has attracted a lot of attention as it provides information about daily physical activities performed by an individual and consequently, helps improve the health monitoring applications. Excessive sitting and lack of adequate levels of physical activity are associated with health problems such as obesity, diabetes, cardiovascular disease, poor metabolic health and depression, leading to the increased risk of mortality. In this thesis, we present an approach that utilizes readings from thebuilt-in sensors in the smartphone to recognize various physical activities performed by the user. Accelerometer and gyroscope sensor signals are analyzed to identify the activity performed by the user. We propose a descriptor-based approach to compute the discriminatory characteristics for activity recognition. In summary, the results presented in this thesis clearly suggest that the data acquired from the built-in sensors of a smartphone carries information useful for analysis of theuser’s behavior. Our experimental results show that the approaches proposed in this thesis achieve state-of-the-art performances for gesture-based biometric authentication, gesture-based gender recognition, gait-based gender recognition, and physical activity recognition.