International Journal of Intelligent Systems and Applications (IJISA)
IJISA Vol. 12, No. 5, 8 Oct. 2020
Cover page and Table of Contents: PDF (size: 1105KB)
MCS-MCMC for Optimising Architectures and Weights of Higher Order Neural Networks
Full Text (PDF, 1105KB), PP.52-72
Views: 0 Downloads: 0
Author(s)
Noor Aida Husaini
1,*
Rozaida Ghazali
1
Nureize Arbaiy
1
Ayodele Lasisi
1
Index Terms
Modified Cuckoo Search, Markov chain Monté Carlo, MCS-MCMC, Higher Order Neural Network, weight optimisation, Backpropagation
Abstract
The standard method to train the Higher Order Neural Networks (HONN) is the well-known Backpropagation (BP) algorithm. Yet, the current BP algorithm has several limitations including easily stuck into local minima, particularly when dealing with highly non-linear problems and utilise computationally intensive training algorithms. The current BP algorithm is also relying heavily on the initial weight values and other parameters picked. Therefore, in an attempt to overcome the BP drawbacks, we investigate a method called Modified Cuckoo Search-Markov chain Monté Carlo for optimising the weights in HONN and boost the learning process. This method, which lies in the Swarm Intelligence area, is notably successful in optimisation task. We compared the performance with several HONN-based network models and standard Multilayer Perceptron on four (4) time series datasets: Temperature, Ozone, Gold Close Price and Bitcoin Closing Price from various repositories. Simulation results indicate that this swarm-based algorithm outperformed or at least at par with the network models with current BP algorithm in terms of lower error rate.
Cite This Paper
Noor Aida Husaini, Rozaida Ghazali, Nureize Arbaiy, Ayodele Lasisi, "MCS-MCMC for Optimising Architectures and Weights of Higher Order Neural Networks", International Journal of Intelligent Systems and Applications(IJISA), Vol.12, No.5, pp.52-72, 2020. DOI:10.5815/ijisa.2020.05.05
Reference
[1]Husaini, N.A., et al. Jordan pi-sigma neural network for temperature prediction. in International Conference on Ubiquitous Computing and Multimedia Applications. 2011. Springer.
[2]Holland, J., Adaptation in natural and artificial systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence. 1992, MIT Press: Ann Arbor, USA.
[3]Goldberg, D., Genetic algorithms in search, optimization and machine learning. 1989, Boston, USA: Addison Wesley.
[4]De Jong, K., Analysis of the behavior of a class of genetic adaptive systems. 1975, University of Michigan: Ann Arbor, MI.
[5]Fogel, L., A. Owens, and W. MJ, Artificial intelligence through simulated evolution. 1966, Chichester, UK: John Wiley.
[6]Storn, R. Differential evolution design of an IIR-filter. in IEEE International Conference on Evolutionary Computation. 1996. Nagoya.
[7]dos Santos Coelho, L. and D.L. de Andrade Bernert, An improved harmony search algorithm for synchronization of discrete-time chaotic systems. Chaos, Solitons & Fractals, 2009. 41(5): p. 2526-2532.
[8]Karaboga, D., B. Akay, and C. Ozturk. Artificial bee colony (abc) optimization algorithm for training feed-forward neural networks. in Proceedings of the 4th international conference on Modeling Decisions for Artificial Intelligence, ser. MDAI ’07. 2007. Springer-Verlag.
[9]Aljarah, I., H. Faris, and S. Mirjalili, Optimizing connection weights in neural networks using the whale optimization algorithm. Soft Computing, 2018. 22(1): p. 1-15.
[10]Mason, K., J. Duggan, and E. Howley. Neural network topology and weight optimization through neuro differential evolution. in Proceedings of the Genetic and Evolutionary Computation Conference Companion. 2017.
[11]Salimans, T. and D.P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. in Advances in neural information processing systems. 2016.
[12]Yang, X.S. and S. Deb. Cuckoo search via Lévy flights. in Proceedings of the World Congress on Nature & Biologically Inspired Computing (NaBIC '09. 2009. India: IEEE Publications.
[13]Gandomi, A., X.-S. Yang, and A. Alavi, Cuckoo search algorithm: a metaheuristic approach to solve structural optimization problems. Engineering with Computers, 2012. 29(1): p. 17-35.
[14]Sickel, J., K. Lee, and J. Heo, Differential evolution and its applications to power plant control, I. International Conference on Intelligent Systems Applications to Power Systems, Editor. 2007. p. 1-6.
[15]Zhang, J., Y. Zhang, and R. Gao. Genetic algorithms for optimal design of vehicle suspensions. in IEEE International Conference on Engineering of Intelligent Systems. 2006.
[16]Kaveh, A. and T. Bakhshpoori, Optimum design of steel frames using cuckoo search algorithm with Levy flights,Structural Design of Tall and Special Buildings. 2011.
[17]Valian, E., S. Mohanna, and S. Tavakoli, Improved cuckoo search algorithm for feed forward neural network training. International Journal of Artificial Intelligence & Applications, 2011. 2(3): p. 36-43.
[18]Walton, S., et al., Modified cuckoo search: A new gradient free optimisation algorithm. Chaos, Solitons & Fractals, 2011. 44(9): p. 710-718.
[19]Husaini, N.A., R. Ghazali, and I.T.R. Yanto. Enhancing modified cuckoo search algorithm by using MCMC random walk. in 2016 2nd International Conference on Science in Information Technology (ICSITech). 2016. IEEE.
[20]Saremi, S., S. Mirjalili, and A. Lewis, Grasshopper optimisation algorithm: theory and application. Advances in Engineering Software, 2017. 105: p. 30-47.
[21]Sun, J., C.-H. Lai, and X.-J. Wu, Particle swarm optimisation: classical and quantum perspectives. 2016: Crc Press.
[22]Dems, K. and Z. Mroz, Variational Approach by Means of Adjoint Systems to Structural Optimization and Sensitivity Analysis--II. International Journal of Solids and Structures, 1984. 20: p. 527-552.
[23]Chenasis, D., Existence of a Solution in a Domain Identification Problem. Journal of Mathematical Analysis and Applications, 1975. 52: p. 189-219.
[24]Fujii, N. Second Variation and Its Application in a Domain Optimization Problem. in Proceedings of the 4th IFAC Symposium on Control of Distributed-Parameter Systems. 1986. Pergamon, Oxford, England.
[25]Fujii, N. Existence of an Optimal Domain in a Domain Optimization Problem. in Proceedings of the 13th IFIP Conference on System Modelling and Optimization. 1988. Springer-Verlag, Berlin, Germany,.
[26]Fujii, N., Lower-Semicontinuity in Domain Optimization Problems. Journal of Optimization Theory and Applications, 1988. 59: p. 407-422.
[27]Zheng, L. An Improved Firefly Algorithm Hybrid with Fireworks. in Computational Intelligence and Intelligent Systems: 10th International Symposium, ISICA 2018, Jiujiang, China, October 13–14, 2018, Revised Selected Papers. 2019. Springer.
[28]Gülcü, Ş., et al., A parallel cooperative hybrid method based on ant colony optimization and 3-Opt algorithm for solving traveling salesman problem. Soft Computing, 2018. 22(5): p. 1669-1685.
[29]Arabasadi, Z., et al., Computer aided decision making for heart disease detection using hybrid neural network-Genetic algorithm. Computer Methods and Programs in Biomedicine, 2017. 141: p. 19-26.
[30]Lahmiri, S., A variational mode decompoisition approach for analysis and forecasting of economic and financial time series. Expert Systems with Applications, 2016. 55: p. 268-273.
[31]Antipov, G., M. Baccouche, and J.-L. Dugelay. Face aging with conditional generative adversarial networks. in 2017 IEEE international conference on image processing (ICIP). 2017. IEEE.
[32]Springenberg, J.T., et al. Bayesian optimization with robust Bayesian neural networks. in Advances in neural information processing systems. 2016.
[33]Wang, G.-G., et al., A new hybrid method based on krill herd and cuckoo search for global optimisation tasks. International Journal of Bio-Inspired Computation, 2016. 8(5): p. 286-299.
[34]Jin, C. and S.-W. Jin, Parameter optimization of software reliability growth model with S-shaped testing-effort function using improved swarm intelligent optimization. Applied Soft Computing, 2016. 40: p. 283-291.
[35]Sim, K. and W. Sun, Ant colony optimization for routing and loadbalancing: survey and new directions. IEEE T Syst Man Cy A, 2003. 33(5): p. 560-572.
[36]Marinakis, Y., M. Marinaki, and G. Dounias, Particle swarm optimization for pap- smear diagnosis. Expert Syst Appl, 2008. 35(4): p. 1645-1656.
[37]Mirjalili, S. and A. Lewis, The whale optimization algorithm. Advances in engineering software, 2016. 95: p. 51-67.
[38]Lynn, N. and P.N. Suganthan, Heterogeneous comprehensive learning particle swarm optimization with enhanced exploration and exploitation. Swarm and Evolutionary Computation, 2015. 24: p. 11-24.
[39]Malaysian Meteorological Department. Weather Forecast. 2010 [cited 2011 February, 18]; Available from: http://www.met.gov.my.