TY - GEN
T1 - Gates Are Not What You Need in RNNs
AU - Zakovskis, Ronalds
AU - Draguns, Andis
AU - Gaile, Eliza
AU - Ozolins, Emils
AU - Freivalds, Karlis
N1 - Publisher Copyright:
© 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.
PY - 2023
Y1 - 2023
N2 - Recurrent neural networks have flourished in many areas. Consequently, we can see new RNN cells being developed continuously, usually by creating or using gates in a new, original way. But what if we told you that gates in RNNs are redundant? In this paper, we propose a new recurrent cell called Residual Recurrent Unit (RRU) which beats traditional cells and does not employ a single gate. It is based on the residual shortcut connection, linear transformations, ReLU, and normalization. To evaluate our cell’s effectiveness, we compare its performance against the widely-used GRU and LSTM cells and the recently proposed Mogrifier LSTM on several tasks including, polyphonic music modeling, language modeling, and sentiment analysis. Our experiments show that RRU outperforms the traditional gated units on most of these tasks. Also, it has better robustness to parameter selection, allowing immediate application in new tasks without much tuning. We have implemented the RRU in TensorFlow, and the code is made available at https://github.com/LUMII-Syslab/RRU.
AB - Recurrent neural networks have flourished in many areas. Consequently, we can see new RNN cells being developed continuously, usually by creating or using gates in a new, original way. But what if we told you that gates in RNNs are redundant? In this paper, we propose a new recurrent cell called Residual Recurrent Unit (RRU) which beats traditional cells and does not employ a single gate. It is based on the residual shortcut connection, linear transformations, ReLU, and normalization. To evaluate our cell’s effectiveness, we compare its performance against the widely-used GRU and LSTM cells and the recently proposed Mogrifier LSTM on several tasks including, polyphonic music modeling, language modeling, and sentiment analysis. Our experiments show that RRU outperforms the traditional gated units on most of these tasks. Also, it has better robustness to parameter selection, allowing immediate application in new tasks without much tuning. We have implemented the RRU in TensorFlow, and the code is made available at https://github.com/LUMII-Syslab/RRU.
KW - Deep learning
KW - Gates
KW - Recurrent neural networks
KW - Residual neural networks
KW - Robustness
UR - https://www.scopus.com/pages/publications/85172411991
U2 - 10.1007/978-3-031-42505-9_27
DO - 10.1007/978-3-031-42505-9_27
M3 - Conference paper
AN - SCOPUS:85172411991
SN - 9783031425042
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 304
EP - 324
BT - Artificial Intelligence and Soft Computing - 22nd International Conference, ICAISC 2023, Proceedings
A2 - Rutkowski, Leszek
A2 - Scherer, Rafał
A2 - Korytkowski, Marcin
A2 - Pedrycz, Witold
A2 - Tadeusiewicz, Ryszard
A2 - Zurada, Jacek M.
PB - Springer Science and Business Media Deutschland GmbH
T2 - 22nd International Conference on Artificial Intelligence and Soft Computing, ICAISC 2023
Y2 - 18 June 2023 through 22 June 2023
ER -