Enhanced Gated Recurrent Neural Networks for Prediction on Time Series Data with Missing Values

Philip P Weerakody

Deep learning networks provide some of the best performance against benchmark time series datasets but suffer from the impacts of missing values if their architectures are not improved for handling missing values. The importance of this issue is compounded by the massive quantity of real-world data containing missing values, which is commonly observed in domains such as medical diagnostics, climate and environmental monitoring, and financial forecasting. Moreover, the explosion of IoT sensors and big data collection, sourced from numerous devices with different sampling frequencies, has generated large amounts of irregular multivariate data in which missing observations frequently occur at short time intervals for one or more feature variables. Yet, despite the significance of this issue, comparatively low levels of research attempt to develop deep learning models that are proven to be robust under conditions of high levels of missing observations. Most existing research concentrates on imputation methods that aim to create a complete dataset before feeding it into a model. These methods often overlook the rich information embedded in missing value patterns and tend to rely on complex ensembles of several machine learning (ML) architectures. In contrast, there is less exploration of inference-based models or model layers that deal directly with observed and missing data. The research in this thesis attempts to address the gap in developing robust deep-learning models for handling high rates of missing values in time series data. Novel inference-based approaches and algorithms are proposed with enhanced gated Recurrent Neural Networks (RNNs) based on GRU and LSTM units to maintain or improve the model performance in the presence of high rates of missing values. These approaches can utilise temporal patterns in both observed and unobserved data, minimising the need for imputation. LSTM and GRU were selected as they are both leading models for dealing with sequential data and tasks that require memory of past events. Also, the internal structures of Gated RNNs have proven to be highly adaptable, allowing for customisation to handle specific types of data while maintaining their accuracy and, in some cases, improving their efficiency. Moreover, this research departs from the common approach of managing missing values using high-level network architectures by concentrating on developments at the cell level of gated recurrent network structures. Specifically, this thesis presents three novel approaches based on enhanced gated RNN units, which learn important parts of an input sequence using several different techniques to learn relevant patterns. The first approach uses an internal recurrent cell gate operation applying cyclic waveforms that determine a periodic pattern based on the characteristics of the input features. In the second approach, we develop a skipping mechanism that uses a Reinforcement Learning technique to learn dynamic skip values based on policy gradient optimisation. The third approach proposes a recurrent cell that inputs spike-encoded time series values and uses spike activations to aid in identifying important observations within a sequence. The models are tested against several real-world time series datasets from various application domains to evaluate comparative classification or regression performance against conventional gated RNN models and advanced, recurrent cell adaptations. Comparisons are also made against conventional non-neural network machine learning algorithms, which can process sequential data. The findings of this research show the proposed enhanced gated RNN models can operate as filters for identifying informative observations or segments within an input set, resulting in superior performance over the LSTM and GRU models as well as state-of-the-art gated RNN variants developed to handle irregular time series data. Performance evaluation of each model takes into account dataset characteristics such as varying sequence lengths and fixed or dynamic period temporal patterns. The results confirm the ability of inference-based gated RNN models with adapted cell structures to provide advantages over alternative gated RNN models when dealing with high rates of missing values.

Enhanced Gated Recurrent Neural Networks for Prediction on Time Series Data with Missing Values

Files and links (1)

Abstract

Details

Metrics