Elastic exponential linear units for convolutional neural networks SCIE SCOPUS

Cited 15 time in WEB OF SCIENCE Cited 18 time in Scopus
Title
Elastic exponential linear units for convolutional neural networks
Author(s)
김대호; 김진아; 김재일
KIOST Author(s)
Kim, Jinah(김진아)
Publication Year
2020-09
Abstract
Activation functions play an important role in determining the depth and nonlinearity of deep learning models. Since Rectified Linear Unit (ReLU) was introduced, many modifications in which noise is intentionally injected have been proposed to avoid overfitting risks. Furthermore, Exponential Linear Unit (ELU) and its variants with trainable parameters have been proposed to reduce the bias shift effects which are often observed from ReLU-type activation functions. In this paper, we propose a novel activation function, called Elastic Exponential Linear Unit (EELU), which combines the advantages of both types of the activation functions in a generalized form. EELU not only has an elastic slope in the positive part, but also preserves the negative signal by a small nonzero gradient. We also present a new strategy to insert neuronal noise following Gaussian distribution in the activation function to improve generalization. In our experiments, we demonstrate how the EELU can represent a wider variety of features with random noise than other activation functions by visualizing the latent features of convolutional neural networks. We evaluate the effectiveness of the EELU through extensive experiments with image classification using CIFAR-10/CIFAR-100, ImageNet, and Tiny ImageNet. Experimental results show that the EELU achieved generalization performance and improved the classification accuracy over the conventional and re
ISSN
0925-2312
URI
https://sciwatch.kiost.ac.kr/handle/2020.kiost/38589
DOI
10.1016/j.neucom.2020.03.051
Bibliographic Citation
NEUROCOMPUTING, v.406, pp.253 - 266, 2020
Publisher
ELSEVIER
Type
Article
Language
English
Document Type
Article
Publisher
ELSEVIER
Files in This Item:
There are no files associated with this item.

qrcode

Items in ScienceWatch@KIOST are protected by copyright, with all rights reserved, unless otherwise indicated.

Browse