International Journal of Intelligent Systems and Applications (IJISA)
IJISA Vol. 16, No. 4, 8 Aug. 2024
Cover page and Table of Contents: PDF (size: 1661KB)
An Exploratory Study on Simulated Annealing for Feature Selection in Learning-to-rank
PDF (1661KB), PP.86-103
Views: 0 Downloads: 0
Author(s)
Index Terms
Learning-to-rank, Feature Selection, Meta-heuristics, Simulated Annealing, Local Beam Search
Abstract
Learning-to-rank is an applied domain of supervised machine learning. As feature selection has been found to be effective for improving the accuracy of learning models in general, it is intriguing to investigate this process for learning-to-rank domain. In this study, we investigate the use of a popular meta-heuristic approach called simulated annealing for this task. Under the general framework of simulated annealing, we explore various neighborhood selection strategies and temperature cooling schemes. We further introduce a new hyper-parameter called the progress parameter that can effectively be used to traverse the search space. Our algorithms are evaluated on five publicly benchmark datasets of learning-to-rank. For a better validation, we also compare the simulated annealing-based feature selection algorithm with another effective meta-heuristic algorithm, namely local beam search. Extensive experimental results show the efficacy of our proposed models.
Cite This Paper
Mohd. Sayemul Haque, Md. Fahim, Muhammad Ibrahim, "An Exploratory Study on Simulated Annealing for Feature Selection in Learning-to-rank", International Journal of Intelligent Systems and Applications(IJISA), Vol.16, No.4, pp.86-103, 2024. DOI:10.5815/ijisa.2024.04.06
Reference
[1]Ricardo Baeza-Yates, Berthier Ribeiro-Neto, et al. Modern information retrieval, volume 463. ACM press New York, 1999.
[2]Michael I Jordan and Tom M Mitchell. Machine learning: Trends, perspectives, and prospects. Science, 349(6245):255–260, 2015.
[3]Liu, T. Y. (2009). Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval, 3(3), 225-331.
[4]Isabelle Guyon and Andre´ Elisseeff. An introduction to variable and feature selection. Journal of machine learning research, 3(Mar):1157–1182, 2003.
[5]Mohammed A Ambusaidi, Xiangjian He, Priyadarsi Nanda, and Zhiyuan Tan. Building an intrusion detection system using a filter-based feature selection algorithm. IEEE transactions on computers, 65(10):2986–2998, 2016.
[6]Noelia Sa´nchez-Marono, Amparo Alonso-Betanzos, and Mar´ıa Tombilla-Sanroma´n. Filter methods for feature selection–a comparative study. In International Conference on Intelligent Data Engineering and Automated Learn- ing, pages 178–187. Springer, 2007.
[7]Ron Kohavi and Dan Sommerfield. Feature subset selection using the wrapper method: Overfitting and dynamic search space topology. In KDD, pages 192–197, 1995.
[8]Sebastia´n Maldonado and Richard Weber. A wrapper method for feature selection using support vector machines. Information Sciences, 179(13):2208–2217, 2009.
[9]Girish Chandrashekar and Ferat Sahin. A survey on feature selection methods. Computers & Electrical Engineering, 40(1):16–28, 2014.
[10]Huan Liu and Lei Yu. Toward integrating feature selection algorithms for classification and clustering. IEEE Transactions on knowledge and data engineering, 17(4):491–502, 2005.
[11]Manik Sharma and Prableen Kaur. A comprehensive analysis of nature-inspired meta-heuristic techniques for feature selection problem. Archives of Computational Methods in Engineering, pages 1–25, 2020.
[12]Jitendra Rajpurohit, Tarun Kumar Sharma, Ajith Abraham, and A Vaishali. Glossary of metaheuristic algorithms. Int. J. Comput. Inf. Syst. Ind. Manag. Appl, 9:181–205, 2017.
[13]Xin-She Yang. Harmony search as a metaheuristic algorithm. In Music-inspired harmony search algorithm, pages 1–14. Springer, 2009.
[14]Shih-Wei Lin, Zne-Jung Lee, Shih-Chieh Chen, and Tsung-Yuan Tseng. Parameter determination of support vector machine and feature selection using simulated annealing approach. Applied soft computing, 8(4):1505–1512, 2008.
[15]Jihoon Yang and Vasant Honavar. Feature subset selection using a genetic algorithm. In Feature extraction, con- struction and selection, pages 117–136. Springer, 1998.
[16]Lucija Brezocˇnik, Iztok Fister, and Vili Podgorelec. Swarm intelligence algorithms for feature selection: a review. Applied Sciences, 8(9):1521, 2018.
[17]Paolo Dario, Giulio Sandini, and Patrick Aebischer. Robots and Biological Systems: Towards a New Bionics?: Proceedings of the NATO Advanced Workshop on Robots and Biological Systems, held at II Ciocco, Toscana, Italy, June 26–30, 1989, volume 102. Springer Science & Business Media, 2012.
[18]Wojciech Siedlecki and Jack Sklansky. On automatic feature selection. In Handbook of pattern recognition and computer vision, pages 63–87. World Scientific, 1993.
[19]Majdi M Mafarja and Seyedali Mirjalili. Hybrid whale optimization algorithm with simulated annealing for feature selection. Neurocomputing, 260:302–312, 2017.
[20]Mustafa Wasif Allvi, Mahamudul Hasan, Lazim Rayan, Mohammad Shahabuddin, Md Mosaddek Khan, and Muhammad Ibrahim. Feature selection for learning-to-rank using simulated annealing. International Journal of Advanced Computer Science and Applications, 11(3):699–705, 2020.
[21]Ibrahim, Osman Ali Sadek and Younis, Eman MG , Hybrid online--offline learning to rank using simulated annealing strategy based on dependent click model, Knowledge and Information Systems, 64(10), pp. 2833 – 2847, 2022, Springer.
[22]Xiubo Geng, Tie-Yan Liu, Tao Qin, and Hang Li. Feature selection for ranking. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 407–414, 2007.
[23]Andrea Gigli, Claudio Lucchese, Franco Maria Nardini, and Raffaele Perego. Fast feature selection for learning to rank. In Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval, pages 167–170, 2016.
[24]Sanmay Das. Filters, wrappers and a boosting-based hybrid for feature selection. In Icml, volume 1, pages 74–81. Citeseer, 2001.
[25]Daniel Xavier Sousa, Se´rgio Canuto, Marcos Andre´ Gonc¸alves, Thierson Couto Rosa, and Wellington Santos Mar- tins. Risk-sensitive learning to rank with evolutionary multi-objective feature selection. ACM Transactions on Information Systems (TOIS), 37(2):1–34, 2019.
[26]Feng Pan, Tim Converse, David Ahn, Franco Salvetti, and Gianluca Donato. Greedy and randomized feature selec- tion for web search ranking. In 2011 IEEE 11th International Conference on Computer and Information Technology, pages 436–442. IEEE, 2011.
[27]Han-Jiang Lai, Yan Pan, Yong Tang, and Rong Yu. Fsmrank: Feature selection algorithm for learning to rank. IEEE transactions on neural networks and learning systems, 24(6):940–952, 2013.
[28]Ashwini Rahangdale and Shital Raut. Deep neural network regularization for feature selection in learning-to-rank. IEEE Access, 7:53988–54006, 2019.
[29]Peter JM Van Laarhoven and Emile HL Aarts. Simulated annealing. In Simulated annealing: Theory and applica- tions, pages 7–15. Springer, 1987.
[30]Rene´ VV Vidal. Applied simulated annealing, volume 396. Springer, 1993.
[31]Dimitris Bertsimas, John Tsitsiklis, et al. Simulated annealing. Statistical science, 8(1):10–15, 1993.
[32]Scott Kirkpatrick, C Daniel Gelatt, and Mario P Vecchi. Optimization by simulated annealing. science, 220(4598):671–680, 1983.
[33]Stuart Russell and Peter Norvig. Artificial intelligence: a modern approach. 2002.
[34]Max Kuhn and Kjell Johnson. Feature engineering and selection: A practical approach for predictive models. CRC Press, 2019.
[35]Noelia Sa´nchez-Marono, Amparo Alonso-Betanzos, and Mar´ıa Tombilla-Sanroma´n. Filter methods for feature selection–a comparative study. In International Conference on Intelligent Data Engineering and Automated Learn- ing, pages 178–187. Springer, 2007.
[36]ChangYing Wang, Min Lin, YiWen Zhong, and Hui Zhang. Solving travelling salesman problem using multiagent simulated annealing algorithm with instance-based sampling. International Journal of Computing Science and Mathematics, 6(4):336–353, 2015.
[37]David T Connolly. An improved annealing scheme for the qap. European Journal of Operational Research, 46(1):93–100, 1990.
[38]Harry Cohn and Mark Fielding. Simulated annealing: searching for an optimal temperature schedule. SIAM Journal on Optimization, 9(3):779–802, 1999.
[39]Harold Szu and Ralph Hartley. Fast simulated annealing. Physics letters A, 122(3-4):157–162, 1987.
[40]David Abramson. Constructing school timetables using simulated annealing: sequential and parallel algorithms. Management science, 37(1):98–113, 1991.
[41]Manoranjan Dash and Huan Liu. Feature selection for classification. Intelligent data analysis, 1(1-4):131–156, 1997.
[42]David W Aha and Richard L Bankert. A comparative evaluation of sequential feature selection algorithms. In Learning from data, pages 199–206. Springer, 1996.
[43]Markus Freitag and Yaser Al-Onaizan. Beam search strategies for neural machine translation. arXiv preprint arXiv:1702.01806, 2017.
[44]Peng Si Ow and Thomas E Morton. Filtered beam search in scheduling. The International Journal Of Production Research, 26(1):35–62, 1988.
[45]Tao Qin and Tie-Yan Liu. Introducing letor 4.0 datasets. arXiv preprint arXiv:1306.2597, 2013.
[46]William Hersh, Chris Buckley, TJ Leone, and David Hickam. Ohsumed: An interactive retrieval evaluation and new large test collection for research. In SIGIR’94, pages 192–201. Springer, 1994.
[47]Tao Qin, Tie-Yan Liu, Jun Xu, and Hang Li. Letor: A benchmark collection for research on learning to rank for information retrieval. Information Retrieval, 13(4):346–374, 2010.
[48]https://www.microsoft.com/en-us/research/project/mslr/
[49]Christopher JC Burges. From ranknet to lambdarank to lambdamart: An overview. Learning, 11(23-581):81, 2010.