Elastic exponential linear units for convolutional neural networks

Abstract

Activation functions play an important role in determining the depth and nonlinearity of deep learning models. Since Rectified Linear Unit (ReLU) was introduced, many modifications in which noise is intentionally injected have been proposed to avoid overfitting risks. Furthermore, Exponential Linear Unit (ELU) and its variants with trainable parameters have been proposed to reduce the bias shift effects which are often observed from ReLU-type activation functions. In this paper, we propose a novel activation function, called Elastic Exponential Linear Unit (EELU), which combines the advantages of both types of the activation functions in a generalized form. EELU not only has an elastic slope in the positive part, but also preserves the negative signal by a small nonzero gradient. We also present a new strategy to insert neuronal noise following Gaussian distribution in the activation function to improve generalization. In our experiments, we demonstrate how the EELU can represent a wider variety of features with random noise than other activation functions by visualizing the latent features of convolutional neural networks. We evaluate the effectiveness of the EELU through extensive experiments with image classification using CIFAR-10/CIFAR-100, ImageNet, and Tiny ImageNet. Experimental results show that the EELU achieved generalization performance and improved the classification accuracy over the conventional and re