Sadia Haq Tamanna; Muhammad Muhtasim; Aroni Saha Prapty; Amrin Nahar; Md. Tanvir Ahmed Tagim; Fahmida Rahman Moumi; Shadia Afrin

Evaluation of Machine Learning Algorithms for Malware Detection: A Comprehensive Review

PDF (523KB), PP.51-67

Views: 0 Downloads: 0

Author(s)

Sadia Haq Tamanna ¹ Muhammad Muhtasim ^1,* Aroni Saha Prapty ¹ Amrin Nahar ¹ Md. Tanvir Ahmed Tagim ¹ Fahmida Rahman Moumi ¹ Shadia Afrin ¹

1. Department of Computer Science and Engineering, Bangladesh Army University of Engineering & Technology, Qadirabad Cantonment, Natore-6431, Bangladesh

* Corresponding author.

DOI: https://doi.org/10.5815/ijwmt.2025.02.05

Received: 6 Jun. 2024 / Revised: 26 Jul. 2024 / Accepted: 2 Feb. 2025 / Published: 8 Apr. 2025

Index Terms

Malware Detection, Static Analysis, Dynamic Analysis, Android Security, Malware Classification, Random Forest

Abstract

Malware outperforms conventional signature-based techniques by posing a dynamic and varied threat to digital environments. In cybersecurity, machine learning has become a potent device, providing flexible and data-driven models for malware identification. The significance of choosing the optimal method for this purpose is emphasized in this review paper. Assembling various datasets comprising benign and malicious samples is the first step in the research process. Important data pretreatment procedures like feature extraction and dimensionality reduction are also included. Machine learning techniques, ranging from decision trees to deep learning models, are evaluated based on metrics like as accuracy, precision, recall, F1-score, and ROC-AUC, which determine how well they distinguish dangerous software from benign applications. A thorough examination of numerous studies shows that the Random Forest algorithm is the most effective in identifying malware. Because Random Forest can handle complex and dynamic malware so well, it performs very well in batch and real-time scenarios. It also performs exceptionally well in static and dynamic analysis circumstances. This study emphasizes how important machine learning is, and how Random Forest is the basis for creating robust malware detection. Its effectiveness, scalability, and adaptability make it a crucial tool for businesses and individuals looking to protect sensitive data and digital assets. In conclusion, by highlighting the value of machine learning and establishing Random Forest as the best-in-class method for malware detection, this review paper advances the subject of cybersecurity. Ethical and privacy concerns reinforce the necessity for responsible implementation and continuous research to tackle the changing malware landscape.

Cite This Paper

Sadia Haq Tamanna, Muhammad Muhtasim, Aroni Saha Prapty, Amrin Nahar, Md. Tanvir Ahmed Tagim, Fahmida Rahman Moumi, Shadia Afrin, "Evaluation of Machine Learning Algorithms for Malware Detection: A Comprehensive Review", International Journal of Wireless and Microwave Technologies(IJWMT), Vol.15, No.2, pp. 51-67, 2025. DOI:10.5815/ijwmt.2025.02.05

Reference

[1]I. Almomani, A. Alkhayer, and W. El-Shafai, “An Automated Vision-Based Deep Learning Model for Efficient Detection of Android Malware Attacks,” IEEE Access, vol. 10, pp. 2700–2720, 2022, doi: 10.1109/ACCESS.2022.3140341.
[2]P. Agrawal and B. Trivedi, “Machine Learning Classifiers for Android Malware Detection,” in Advances in Intelligent Systems and Computing, Springer, 2021, pp. 311–322. doi: 10.1007/978-981-15-5616-6_22.
[3]R. Damaševičius, A. Venčkauskas, J. Toldinas, and Š. Grigaliūnas, “Ensemble‐based classification using neural networks and machine learning models for windows pe malware detection,” Electronics (Switzerland), vol. 10, no. 4, pp. 1–26, Feb. 2021, doi: 10.3390/electronics10040485.
[4]I. Shhadat, B. Bataineh, A. Hayajneh, and Z. A. Al-Sharif, “The Use of Machine Learning Techniques to Advance the Detection and Classification of Unknown Malware,” in Procedia Computer Science, Elsevier B.V., 2020, pp. 917–922. doi: 10.1016/j.procs.2020.03.110.
[5]B. M. Khammas, “Ransomware Detection using Random Forest Technique,” ICT Express, vol. 6, no. 4, pp. 325– 331, Dec. 2020, doi: 10.1016/j.icte.2020.11.001.
[6]M. Kedziora, P. Gawin, M. Szczepanik, and I. Jozwiak, “Malware Detection Using Machine Learning Algorithms and Reverse Engineering of Android Java Code,” International Journal of Network Security & Its Applications, vol. 11, no. 01, pp. 01–14, Jan. 2019, doi: 10.5121/ijnsa.2019.11101.
[7]S. Il Bae, G. Bin Lee, and E. G. Im, “Ransomware detection using machine learning algorithms,” in Concurrency and Computation: Practice and Experience, John Wiley and Sons Ltd, Sep. 2020. doi: 10.1002/cpe.5422.
[8]H. Han, S. Lim, K. Suh, S. Park, S. J. Cho, and M. Park, “Enhanced android malware detection: An SVM-based machine learning approach,” in Proceedings - 2020 IEEE International Conference on Big Data and Smart Computing, BigComp 2020, Institute of Electrical and Electronics Engineers Inc., Feb. 2020, pp. 75–81. doi: 10.1109/BigComp48618.2020.00-96.
[9]L. Da Costa and V. Moia, “A Lightweight and Multi-Stage Approach for Android Malware Detection Using Non- Invasive Machine Learning Techniques,” IEEE Access, vol. 11, pp. 73127–73144, 2023, doi: 10.1109/ACCESS.2023.3296606.
[10]M. E. Khoda, T. Imam, J. Kamruzzaman, I. Gondal, and A. Rahman, “Robust Malware Defense in Industrial IoT Applications Using Machine Learning with Selective Adversarial Samples,” IEEE Trans Ind Appl, vol. 56, no. 4, pp. 4415–4424, Jul. 2020, doi: 10.1109/TIA.2019.2958530.
[11]M. Aljabri et al., “Detecting Malicious URLs Using Machine Learning Techniques: Review and Research Directions,” IEEE Access, vol. 10, pp. 121395–121417, 2022, doi: 10.1109/ACCESS.2022.3222307.
[12]J. Senanayake, H. Kalutarage, and M. O. Al-Kadri, “Android mobile malware detection using machine learning: A systematic review,” Electronics (Switzerland), vol. 10, no. 13. MDPI AG, Jul. 01, 2021. doi: 10.3390/electronics10131606.
[13]L. Gong et al., “Systematically Landing Machine Learning onto Market-Scale Mobile Malware Detection,” IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 7, pp. 1615–1628, Jul. 2021, doi: 10.1109/TPDS.2020.3046092.
[14]M. Al-Kasassbeh, S. Mohammed, M. Alauthman, and A. Almomani, “Feature selection using a machine learning to classify a malware,” in Handbook of Computer Networks and Cyber Security: Principles and Paradigms, Springer International Publishing, 2019, pp. 889–904. doi: 10.1007/978-3-030-22277-2_36.
[15]D. Gupta and R. Rani, “Improving malware detection using big data and ensemble learning,” Computers and Electrical Engineering, vol. 86, Sep. 2020, doi: 10.1016/j.compeleceng.2020.106729.
[16]F. Xiao, Z. Lin, Y. Sun, and Y. Ma, “Malware Detection Based on Deep Learning of Behavior Graphs,” Math Probl Eng, vol. 2019, 2019, doi: 10.1155/2019/8195395.
[17]S. Sharma, C. Rama Krishna, S. K. Sahay, and M. Scholar, “Detection of Advanced Malware by Machine Learning Techniques.”
[18]I. Baptista, S. Shiaeles, and N. Kolokotronis, “A Novel Malware Detection System Based On Machine Learning and Binary Visualization.”
[19]K. Lee, S. Y. Lee, and K. Yim, “Machine Learning Based File Entropy Analysis for Ransomware Detection in Backup Systems,” IEEE Access, vol. 7, pp. 110205–110215, 2019, doi: 10.1109/ACCESS.2019.2931136.
[20]T. Kim, B. Kang, M. Rho, S. Sezer, and E. G. Im, “A multimodal deep learning method for android malware detection using various features,” IEEE Transactions on Information Forensics and Security, vol. 14, no. 3, pp. 773–788, Mar. 2019, doi: 10.1109/TIFS.2018.2866319.
[21]R. Kalakoti, S. Nomm, and H. Bahsi, “In-Depth Feature Selection for the Statistical Machine Learning-Based Botnet Detection in IoT Networks,” IEEE Access, vol. 10, pp. 94518–94535, 2022, doi: 10.1109/ACCESS.2022.3204001.
[22]N. A. Azeez, O. E. Odufuwa, S. Misra, J. Oluranti, and R. Damaševičius, “Windows PE malware detection using ensemble learning,” Informatics, vol. 8, no. 1, Mar. 2021, doi: 10.3390/informatics8010010.
[23]V. Kouliaridis and G. Kambourakis, “A comprehensive survey on machine learning techniques for android malware detection,” Information (Switzerland), vol. 12, no. 5, 2021, doi: 10.3390/info12050185.
[24]L. Cai, Y. Li, and Z. Xiong, “JOWMDroid: Android malware detection based on feature weighting with joint optimization of weight-mapping and classifier parameters,” Comput Secur, vol. 100, Jan. 2021, doi: 10.1016/j.cose.2020.102086.
[25]F. Khan, C. Ncube, L. K. Ramasamy, S. Kadry, and Y. Nam, “A Digital DNA Sequencing Engine for Ransomware Detection Using Machine Learning,” IEEE Access, vol. 8, pp. 119710–119719, 2020, doi: 10.1109/ACCESS.2020.3003785.
[26]Y. Li, K. Xiong, T. Chin, and C. Hu, “A Machine Learning Framework for Domain Generation Algorithm-Based Malware Detection,” IEEE Access, vol. 7, pp. 32765–32782, 2019, doi: 10.1109/ACCESS.2019.2891588.
[27]R. Kumar, X. Zhang, W. Wang, R. U. Khan, J. Kumar, and A. Sharif, “A Multimodal Malware Detection Technique for Android IoT Devices Using Various Features,” IEEE Access, vol. 7, pp. 64411–64430, 2019, doi: 10.1109/ACCESS.2019.2916886.
[28]K. Shaukat, S. Luo, S. Chen, and D. Liu, “Cyber Threat Detection Using Machine Learning Techniques: A Performance Evaluation Perspective,” in 1st Annual International Conference on Cyber Warfare and Security, ICCWS 2020 - Proceedings, Institute of Electrical and Electronics Engineers Inc., Oct. 2020. doi: 10.1109/ICCWS48432.2020.9292388.
[29]R. Vinayakumar, M. Alazab, K. P. Soman, P. Poornachandran, and S. Venkatraman, “Robust Intelligent Malware Detection Using Deep Learning,” IEEE Access, vol. 7, pp. 46717–46738, 2019, doi: 10.1109/ACCESS.2019.2906934.
[30]A. K. Ajay and J. C.D., “Automated multi-level malware detection system based on reconstructed semantic view of executables using machine learning techniques at VMM,” Future Generation Computer Systems, vol. 79, pp. 431– 446, Feb. 2018, doi: 10.1016/j.future.2017.06.002.
[31]B. Urooj, M. A. Shah, C. Maple, M. K. Abbasi, and S. Riasat, “Malware Detection: A Framework for Reverse Engineered Android Applications Through Machine Learning Algorithms,” IEEE Access, vol. 10, pp. 89031–89050, 2022, doi: 10.1109/ACCESS.2022.3149053.
[32]K. Bakour and H. M. Ünver, “VisDroid: Android malware classification based on local and global image features, bag of visual words and machine learning techniques,” Neural Comput Appl, vol. 33, no. 8, pp. 3133–3153, Apr. 2021, doi: 10.1007/s00521-020-05195-w.
[33]J. Abawajy, A. Darem, and A. A. Alhashmi, “Feature subset selection for malware detection in smart iot platforms,” Sensors (Switzerland), vol. 21, no. 4, pp. 1–19, Feb. 2021, doi: 10.3390/s21041374.
[34]M. Ali, S. Shiaeles, G. Bendiab, and B. Ghita, “Malgra: Machine learning and N-GRAM malware feature extraction and detection system,” Electronics (Switzerland), vol. 9, no. 11, pp. 1–20, Nov. 2020, doi: 10.3390/electronics9111777.
[35]D. W. Fernando, N. Komninos, and T. Chen, “A Study on the Evolution of Ransomware Detection Using Machine Learning and Deep Learning Techniques,” Internet of Things, vol. 1, no. 2. MDPI, pp. 551–604, Dec. 01, 2020. doi: 10.3390/iot1020030.
[36]H. Yang, S. Li, X. Wu, H. Lu, and W. Han, “A Novel Solutions for Malicious Code Detection and Family Clustering Based on Machine Learning,” IEEE Access, vol. 7, pp. 148853–148860, 2019, doi: 10.1109/ACCESS.2019.2946482.
[37]H. El Merabet and A. Hajraoui, “A Survey of Malware Detection Techniques based on Machine Learning,” 2019. [Online]. Available: www.ijacsa.thesai.org
[38]R. Feng et al., “MobiDroid: A performance-sensitive malware detection system on mobile platform,” in Proceedings of the IEEE International Conference on Engineering of Complex Computer Systems, ICECCS, Institute of Electrical and Electronics Engineers Inc., Nov. 2019, pp. 61–70. doi: 10.1109/ICECCS.2019.00014.
[39]W. Yuan, Y. Jiang, H. Li, and M. Cai, “A Lightweight On-Device Detection Method for Android Malware,” IEEE Trans Syst Man Cybern Syst, vol. 51, no. 9, pp. 5600–5611, Sep. 2021, doi: 10.1109/TSMC.2019.2958382.
[40]H. Rathore, S. Agarwal, S. K. Sahay, and M. Sewak, “Malware Detection using Machine Learning and Deep Learning,” Apr. 2019, doi: 10.1007/978-3-030-04780-1_28.
[41]Y. Gao, H. Hasegawa, Y. Yamaguchi, and H. Shimada, “Malware Detection by Control-Flow Graph Level Representation Learning With Graph Isomorphism Network,” IEEE Access, vol. 10, pp. 111830–111841, 2022, doi: 10.1109/ACCESS.2022.3215267.
[42]W. K. Wong, F. H. Juwono, and C. Apriono, “Vision-Based Malware Detection: A Transfer Learning Approach Using Optimal ECOC-SVM Configuration,” IEEE Access, vol. 9, pp. 159262–159270, 2021, doi: 10.1109/ACCESS.2021.3131713.
[43]X. Wang and C. Li, “Android malware detection through machine learning on kernel task structures,” Neurocomputing, vol. 435, pp. 126–150, May 2021, doi: 10.1016/j.neucom.2020.12.088.
[44]H. M. Ünver and K. Bakour, “Android malware detection based on image-based features and machine learning techniques,” SN Appl Sci, vol. 2, no. 7, Jul. 2020, doi: 10.1007/s42452-020-3132-2.
[45]J. Singh and J. Singh, “Detection of malicious software by analyzing the behavioral artifacts using machine learning algorithms,” Inf Softw Technol, vol. 121, May 2020, doi: 10.1016/j.infsof.2020.106273.
[46]R. Damaševičius, A. Venčkauskas, J. Toldinas, and Š. Grigaliūnas, “Ensemble‐based classification using neural networks and machine learning models for windows pe malware detection,” Electronics (Switzerland), vol. 10, no. 4, pp. 1–26, Feb. 2021, doi: 10.3390/electronics10040485.
[47]A. Alotaibi, “Identifying Malicious Software Using Deep Residual Long-Short Term Memory,” IEEE Access, vol. 7, pp. 163128–163137, 2019, doi: 10.1109/ACCESS.2019.2951751.
[48]K. Khariwal, J. Singh, and A. Arora, “IPDroid: Android malware detection using intents and permissions,” in Proceedings of the World Conference on Smart Trends in Systems, Security and Sustainability, WS4 2020, Institute of Electrical and Electronics Engineers Inc., Jul. 2020, pp. 197–202. doi: 10.1109/WorldS450073.2020.9210414.
[49]E. Odat and Q. M. Yaseen, “A Novel Machine Learning Approach for Android Malware Detection Based on the Co- Existence of Features,” IEEE Access, vol. 11, pp. 15471–15484, 2023, doi: 10.1109/ACCESS.2023.3244656.
[50]A. Azmoodeh, A. Dehghantanha, and K. K. R. Choo, “Robust Malware Detection for Internet of (Battlefield) Things Devices Using Deep Eigenspace Learning,” IEEE Transactions on Sustainable Computing, vol. 4, no. 1, pp. 88–95, Jan. 2019, doi: 10.1109/TSUSC.2018.2809665.

International Journal of Wireless and Microwave Technologies (IJWMT)