Journal article
Redundancy and attention in convolutional LSTM for gesture recognition
IEEE Transactions on Neural Networks and Learning Systems, Vol.31(4), pp.1323-1335
2020
Abstract
Convolutional long short-term memory (ConvLSTM) networks have been widely used for action/gesture recognition, and different attention mechanisms have also been embedded into ConvLSTM networks. This paper explores the redundancy of spatial convolutions and the effects of the attention mechanism in ConvLSTM, based on our previous gesture recognition architectures that combine the 3-D convolutional neural network (CNN) and ConvLSTM. Depthwise separable, group, and shuffle convolutions are used to replace the convolutional structures in ConvLSTM for the redundancy analysis. In addition, four ConvLSTM variants are derived for attention analysis: 1) by removing the convolutional structures of the three gates in ConvLSTM; 2) by applying the attention mechanism on the ConvLSTM input; and 3) by reconstructing the input and 4) output gates with the modified channelwise attention mechanism. Evaluation results demonstrate that the spatial convolutions in the three gates scarcely contribute to the spatiotemporal feature fusion and that the attention mechanisms embedded into the input and output gates cannot improve the feature fusion. In other words, ConvLSTM mainly contributes to the temporal fusion along with the recurrent steps to learn long-term spatiotemporal features when taking spatial or spatiotemporal features as input. A new LSTM variant is derived on this basis in which the convolutional structures are embedded only into the input-to-state transition of LSTM. The code of the LSTM variants is publicly available. 1 1
Details
- Title
- Redundancy and attention in convolutional LSTM for gesture recognition
- Authors/Creators
- G. Zhu (Author/Creator)L. Zhang (Author/Creator)L. Yang (Author/Creator)L. Mei (Author/Creator)S.A.A. Shah (Author/Creator)M. Bennamoun (Author/Creator)P. Shen (Author/Creator)
- Publication Details
- IEEE Transactions on Neural Networks and Learning Systems, Vol.31(4), pp.1323-1335
- Publisher
- IEEE
- Identifiers
- 991005545068807891
- Copyright
- © 2020 IEEE
- Murdoch Affiliation
- School of Engineering and Energy
- Language
- English
- Resource Type
- Journal article
UN Sustainable Development Goals (SDGs)
This output has contributed to the advancement of the following goals:
Source: InCites
Metrics
83 Record Views
InCites Highlights
These are selected metrics from InCites Benchmarking & Analytics tool, related to this output
- Collaboration types
- Domestic collaboration
- International collaboration
- Citation topics
- 4 Electrical Engineering, Electronics & Computer Science
- 4.116 Robotics
- 4.116.1097 Gesture Recognition
- Web Of Science research areas
- Computer Science, Artificial Intelligence
- Computer Science, Hardware & Architecture
- Computer Science, Theory & Methods
- Engineering, Electrical & Electronic
- ESI research areas
- Computer Science