Dynamic Activation Functions In Deep Neural Networks
Abstract
Activation functions are considered as main component in artificial neural networks. The current paper considers learning activation functions with combination of activation functions. We propose two approaches to use activation functions and construction of adaptive activation parameters to input data. Namely, to show effectiveness, we investigate linear form and non-linear form to combine activation functions, then introduce adaptive activation function. Numerical experiments show the proposed activation techniques overcome by performances and accuracy than standard rectified unit family functions.
References
Guo Y., Liu Y., Oerlemans A., Lao S., Wu S., Lew M.S. Deep learning for visual understanding: a review,
Neurocomputing, Vol. 187, 27–48 (2016).
Krizhevsky A., Sutskever I., Hinton G.E. Imagenet classification with deep convolutional neural
networks, Advances In Neural Information Processing Systems, Vol. 25, 1106–1114 (2012).
Li X., Cai C., Zhang R., Ju L., He J. Deep cascaded convolutional models for cattle pose estimation,
Computers and Electronics in Agriculture, Vol. 164, 45-67 (2019).
Gu G., Liu J., Li Z., Huo W., Zhao Y. Joint learning based deep supervised hashing for large-scale image
retrieval, Neurocomputing, Vol. 385, 348-357 (2020).
Yang J., Zhang D., Frangi A., Yang J. Two-dimensional PCA a new approach to appearance-based face
representation and recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence,
Vol. 26, Issue 1, 131-137 (2004).
Taigman Y., Yang M., Ranzato M., Wolf L. Deepface: Closing the gap to human-level performance in
face verification, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
Columbus, U.S.A, 1701-1708 (2014).
Li C., Chen Z., Wu Q. M., Liu C. Deep saliency detection via channel-wise hierarchical feature
responses, Neurocomputing, Vol. 322, 80-92 (2018).
Tuo Q., Zhao H., Hu Q. Hierarchical feature selection with subtree based graph regularization,
Knowledge-Based Systems, Vol. 163, 996-1008 (2019).
Wu G., Lu W., Gao G., Zhao C., Liu J. Regional deep learning model for visual tracking,
Neurocomputing, Vol. 175, 310–323 (2016).
An S., Boussaid F., Bennamoun M., Sohel F. Exploiting layerwise convexity of rectifier networks with
sign constrained weights, Neural Networks, Vol. 105, 419-430 (2018).
Apicella A., Isgro F., Prevete R. A simple and efficient architecture for trainable activation functions,
Neurocomputing, Vol. 370, 1-15 (2019).
He K., Zhang X., Ren S., Sun J. Delving deep into rectifiers: surpassing human-level performance on
ImageNet classification, The IEEE International Conference on Computer Vision, 1026-1034 (2015).
Li Y., Fan C., Li Y., Wu Q., Ming Y. Improving deep neural network with Multiple Parametric
Exponential Linear Units, Neurocomputing, Vol. 301, 11-24 (2018).
Glorot X., Bordes A., Bengio Y. Deep sparse rectifier neural networks. Proceedings of the 14th
International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA. Vol. 15,
-323 (2011).
Marakhimov A. R., Khudaybergenov K. K. A fuzzy MLP approach for identification of nonlinear
systems, Contemporary problems in mathematics and physics, CMFD, Vol. 65, no. 1, Peoples'
Friendship University of Russia, M., 44–53 (2019).
Marakhimov A.R., Khudaybergenov K.K. Convergence analysis of feedforward neural networks with
backpropagation, Bulletin of National University of Uzbekistan: Mathematics and Natural Sciences:
Vol. 2, Issue 2, 77-93 (2019), Available at: https://uzjournals.edu.uz/mns_nuu/vol2/iss2/1
Yusupbekov N. R., Marakhimov A. R., Igamberdiev H. Z., Umarov Sh. X. An Adaptive Fuzzy-Logic
Traffic Control System in Conditions of Saturated Transport Stream, The Scientific World Journal
Vol. 2016, 23-36 (2016).
Yusupbekov N.R., Marakhimov A.R., Igamberdiev H.Z., Umarov Sh.X. Application of soft-computing
technologies to the traffic control system design problems. 12th International Conference on
Application of Fuzzy Systems and Soft Computing, ICAFS 2016, 29-30 August, Vienna, Austria
(2016).
Marakhimov A.R., Siddikov I.H., Nasridinov A., Byun J.Y. Structural Synthesis of Information
Computer Networks of Automated Control Systems Based on Genetic Algorithms, Computer Science
and its Applications, Vol. 330, 1055-1063 (2015).
Nasridinov A., Marakhimov A. Park Y.H. A design of wireless sensor networks based on fuzzy
modeling for comfortable human life, Asia Live Sciences, The Asian International Journal of Life
Sciences, July 2015, Philippines, 265-277.
Yusupbekov N.R., Marakhimov A.R. Synthesis of the intelligent traffic control systems in conditions
saturated transport stream, International Journal of International Journal of Chemical Technology,
Vol 1|No 6 (2024): International Journal of Informatics and Data Science Research
Control and Management Jointly with The Journal of Korea Multimedia Society. Special Issue, South
Korea, Seoul, 12-18 (2015).
Jagtap D.A., Kawaguchi K., Karniadakis G.E. Adaptive activation functions accelerate convergence in
deep and physics-informed neural networks, Journal of Computational Physics, Vol. 404, 45-67
(2020).
Konstantinidis D., Argyriou V., Stathaki T., Grammalidis N. A modular CNN-based building detector
for remote sensing images, Computer Networks, Vol. 168, 93-121 (2020).
Jiang W., Wu L., Liu S., Liu M. CNN-based two-stage cell segmentation improves plant cell tracking,
Pattern Recognition Letters, Vol. 128, 311-317 (2019).
Xu Z., Zhao J., Yu Y., Zeng H. Improved 1D-CNNs for behavior recognition using wearable sensor
network, Computer Communications, Vol. 15, Issue 11, 165-171 (2020).
Amin S. U., Alsulaiman M., Muhammad G., Mekhtiche M. A., Hossain M. S. Deep Learning for EEG
motor imagery classification based on multi-layer CNNs feature fusion, Future Generation
Computer Systems, Vol. 101, 542-554 (2019).
LeCun Y., Bottou L., Bengio Y., Haffner P. Gradient-based learning applied to document recognition,
IEEE 86, Vol. 11, 2278–2324 (1998).
Krizhevsky A., Learning multiple layers of features from tiny images. Technical report, University
of Toronto, (2009).
J. Deng, W. Dong, R. Socher, L. Li, K. Li, F. Li, Imagenet: a large-scale hierarchical image database,
Conference: 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition,
Miami, Florida, USA, 20-25 June 2009.
MNST Dataset, available at: http://yann.lecun.com/exdb/mnist/
CIFAR Dataset, available at: https://www.cs.toronto.edu/~kriz/cifar.html
ImageNet Dataset, available at: http://image-net.org/download
