Dynamic Activation Functions in Deep Neural Networks
Abstract
Activation functions are considered as main component in artificial neural networks. The current paper considers learning activation functions with combination of activation functions. We propose two approaches to use activation functions and construction of adaptive activation parameters to input data. Namely, to show effectiveness, we investigate linear form and non-linear form to combine activation functions, then introduce adaptive activation function. Numerical experiments show the proposed activation techniques overcome by performances and accuracy than standard rectified unit family functions.
References
Guo Y., Liu Y., Oerlemans A., Lao S., Wu S., Lew M.S. Deep learning for visual understanding: a
review, Neurocomputing, Vol. 187, 27–48 (2016).
Krizhevsky A., Sutskever I., Hinton G.E. Imagenet classification with deep convolutional neural
networks, Advances In Neural Information Processing Systems, Vol. 25, 1106–1114 (2012).
Li X., Cai C., Zhang R., Ju L., He J. Deep cascaded convolutional models for cattle pose estimation,
Computers and Electronics in Agriculture, Vol. 164, 45-67 (2019).
Gu G., Liu J., Li Z., Huo W., Zhao Y. Joint learning based deep supervised hashing for large-scale
image retrieval, Neurocomputing, Vol. 385, 348-357 (2020).
Yang J., Zhang D., Frangi A., Yang J. Two-dimensional PCA a new approach to appearance-based
face representation and recognition, IEEE Transactions on Pattern Analysis and Machine
Intelligence, Vol. 26, Issue 1, 131-137 (2004).
Taigman Y., Yang M., Ranzato M., Wolf L. Deepface: Closing the gap to human-level
performance in face verification, Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, Columbus, U.S.A, 1701-1708 (2014).
Li C., Chen Z., Wu Q. M., Liu C. Deep saliency detection via channel-wise hierarchical feature
responses, Neurocomputing, Vol. 322, 80-92 (2018).
Tuo Q., Zhao H., Hu Q. Hierarchical feature selection with subtree based graph regularization,
Knowledge-Based Systems, Vol. 163, 996-1008 (2019).
Wu G., Lu W., Gao G., Zhao C., Liu J. Regional deep learning model for visual tracking,
Neurocomputing, Vol. 175, 310–323 (2016).
An S., Boussaid F., Bennamoun M., Sohel F. Exploiting layerwise convexity of rectifier networks
with sign constrained weights, Neural Networks, Vol. 105, 419-430 (2018).
Apicella A., Isgro F., Prevete R. A simple and efficient architecture for trainable activation
functions, Neurocomputing, Vol. 370, 1-15 (2019).
He K., Zhang X., Ren S., Sun J. Delving deep into rectifiers: surpassing human-level performance
on ImageNet classification, The IEEE International Conference on Computer Vision, 1026-
Vol 1|No 6 (2024): International Journal of Informatics and Data Science Research
(2015).
Li Y., Fan C., Li Y., Wu Q., Ming Y. Improving deep neural network with Multiple Parametric
Exponential Linear Units, Neurocomputing, Vol. 301, 11-24 (2018).
Glorot X., Bordes A., Bengio Y. Deep sparse rectifier neural networks. Proceedings of the 14th
International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA. Vol.
, 315-323 (2011).
Marakhimov A. R., Khudaybergenov K. K. A fuzzy MLP approach for identification of nonlinear
systems, Contemporary problems in mathematics and physics, CMFD, Vol. 65, no. 1, Peoples'
Friendship University of Russia, M., 44–53 (2019).
Marakhimov A.R., Khudaybergenov K.K. Convergence analysis of feedforward neural
networks with backpropagation, Bulletin of National University of Uzbekistan: Mathematics
and Natural Sciences: Vol. 2, Issue 2, 77-93 (2019), Available at:
https://uzjournals.edu.uz/mns_nuu/vol2/iss2/1
Yusupbekov N. R., Marakhimov A. R., Igamberdiev H. Z., Umarov Sh. X. An Adaptive FuzzyLogic Traffic Control System in Conditions of Saturated Transport Stream, The Scientific
World Journal Vol. 2016, 23-36 (2016).
Yusupbekov N.R., Marakhimov A.R., Igamberdiev H.Z., Umarov Sh.X. Application of softcomputing technologies to the traffic control system design problems. 12th International
Conference on Application of Fuzzy Systems and Soft Computing, ICAFS 2016, 29-30 August,
Vienna, Austria (2016).
Marakhimov A.R., Siddikov I.H., Nasridinov A., Byun J.Y. Structural Synthesis of Information
Computer Networks of Automated Control Systems Based on Genetic Algorithms, Computer
Science and its Applications, Vol. 330, 1055-1063 (2015).
Nasridinov A., Marakhimov A. Park Y.H. A design of wireless sensor networks based on fuzzy
modeling for comfortable human life, Asia Live Sciences, The Asian International Journal of
Life Sciences, July 2015, Philippines, 265-277.
Yusupbekov N.R., Marakhimov A.R. Synthesis of the intelligent traffic control systems in
conditions saturated transport stream, International Journal of International Journal of
Chemical Technology, Control and Management Jointly with The Journal of Korea Multimedia
Society. Special Issue, South Korea, Seoul, 12-18 (2015).
Jagtap D.A., Kawaguchi K., Karniadakis G.E. Adaptive activation functions accelerate
convergence in deep and physics-informed neural networks, Journal of Computational
Physics, Vol. 404, 45-67 (2020).
Konstantinidis D., Argyriou V., Stathaki T., Grammalidis N. A modular CNN-based building
detector for remote sensing images, Computer Networks, Vol. 168, 93-121 (2020).
Vol 1|No 6 (2024): International Journal of Informatics and Data Science Research
Jiang W., Wu L., Liu S., Liu M. CNN-based two-stage cell segmentation improves plant cell
tracking, Pattern Recognition Letters, Vol. 128, 311-317 (2019).
Xu Z., Zhao J., Yu Y., Zeng H. Improved 1D-CNNs for behavior recognition using wearable
sensor network, Computer Communications, Vol. 15, Issue 11, 165-171 (2020).
Amin S. U., Alsulaiman M., Muhammad G., Mekhtiche M. A., Hossain M. S. Deep Learning for
EEG motor imagery classification based on multi-layer CNNs feature fusion, Future Generation
Computer Systems, Vol. 101, 542-554 (2019).
LeCun Y., Bottou L., Bengio Y., Haffner P. Gradient-based learning applied to document
recognition, IEEE 86, Vol. 11, 2278–2324 (1998).
Krizhevsky A., Learning multiple layers of features from tiny images. Technical report,
University of Toronto, (2009).
J. Deng, W. Dong, R. Socher, L. Li, K. Li, F. Li, Imagenet: a large-scale hierarchical image
database, Conference: 2009 IEEE Computer Society Conference on Computer Vision and
Pattern Recognition, Miami, Florida, USA, 20-25 June 2009.
MNST Dataset, available at: http://yann.lecun.com/exdb/mnist/
CIFAR Dataset, available at: https://www.cs.toronto.edu/~kriz/cifar.html
ImageNet Dataset, available at: http://image-net.org/download
