Khadoudja Ghanem; Ziad Kherbache; Omar Ourdighi

Enhancing Adversarial Examples for Evading Malware Detection Systems: A Memetic Algorithm Approach

PDF (1234KB), PP.1-16

Views: 0 Downloads: 0

Author(s)

Khadoudja Ghanem ^1,* Ziad Kherbache ¹ Omar Ourdighi ¹

1. University Constantine 2, Abdelhamid Mehri, Faculty of New Technologies of Information and Communication, Department of Computer Science, 25000, Algéria

* Corresponding author.

DOI: https://doi.org/10.5815/ijcnis.2025.01.01

Received: 26 Aug. 2023 / Revised: 25 Oct. 2023 / Accepted: 27 Dec. 2023 / Published: 8 Feb. 2025

Index Terms

Malware Detection, Adversarial Examples, Memetic Algorithms, Genetic Algorithm, Malconv Model, Machine Learning

Abstract

Malware detection using Machine Learning techniques has gained popularity due to their high accuracy. However, ML models are susceptible to Adversarial Examples, specifically crafted samples intended to deceive the detectors. This paper presents a novel method for generating evasive AEs by augmenting existing malware with a new section at the end of the PE file, populated with binary data using memetic algorithms. Our method hybridizes global search and local search techniques to achieve optimized results. The Malconv Model, a well-known state-of-the-art deep learning model designed explicitly for detecting malicious PE files, was used to assess the evasion rates. Out of 100 tested samples, 98 successfully evaded the MalConv model. Additionally, we investigated the simultaneous evasion of multiple detectors, observing evasion rates of 35% and 44% against KNN and Decision Tree machine learning detectors, respectively. Furthermore, evasion rates of 26% and 10% were achieved against Kaspersky and ESET commercial detectors. In order to prove the efficiency of our memetic algorithm in generating evasive adversarial examples, we compared it to the most used evolutionary-based attack: the genetic algorithm. Our method demonstrated significantly superior performance while utilizing fewer generations and a smaller population size.

Cite This Paper

Khadoudja Ghanem, Ziad Kherbache, Omar Ourdighi, "Enhancing Adversarial Examples for Evading Malware Detection Systems: A Memetic Algorithm Approach", International Journal of Computer Network and Information Security(IJCNIS), Vol.17, No.1, pp.1-16, 2025. DOI:10.5815/ijcnis.2025.01.01

Reference

[1]Ibitoye O., Abou-Khamis R., El Shehaby M., Matrawy A., Omair Shafiq M., "The Threat of Adversarial Attacks Against Machine Learning in Network Security: A Survey", 2020. DOI.org/10.48550/arXiv.1911.02621
[2]Alotaibi A., Rassam M.A. "Adversarial Machine Learning Attacks against Intrusion Detection Systems: A Survey on Strategies and Defense". Future Internet, vol.15, No.2, 62, 2023. DOI.org/10.3390/fi15020062
[3]Xiangjun L., Ke K., Su X., Pengtao Q., Daojing H., "Feature selection-based android malware adversarial sample generation and detection method". IET Information Security, Vol.15,No.6,pp.401-416, 2021. DOI.org/10.1049/ise2.12030
[4]Gupta S., Lamba S., Soni N., Priyadarshi P., "Evading Detection Systems by Generating Adversarial Malware Examples", In: Agrawal, R., Sanyal, G., Curran, K., Balas, V.E., Gaur, M.S. (eds), Cybersecurity in Emerging Digital Era. (ICCEDE), Communications in Computer and Information Science, vol. 1436, pp.51-60, 2020. DOI.org/10.1007/978-3-030-84842-2_4
[5]Park D., Yener B., "A survey on practical AEs for malware classifiers", In: Reversing and Offensive-oriented Trends Symposium, ACM, pp. 23–35, 2020. DOI.org/10.1145/3433667.3433670
[6]Deqiang L., Qianmu L., Yanfang Y., Shouhuai X., "Arms race in adversarial malware detection: A survey", ACM Computing Surveys (CSUR), vol.55,No1,pp.1–35, 2021. DOI.org/10.1145/3484491
[7]Xiang L., Lingfei W., Jiangyu Z., Zhenqing Q., Wei D., Xiang C., Yaguan Q., Chunming W., Shouling J., Tianyue L., Jingzheng W., Yanjun W., "Adversarial attacks against Windows PE malware detection: A survey of the state-of-the-art". Computers & Security, vol.128, 2023. DOI: 10.1016/j.cose.2023.103134
[8]Yanchen Q., Weizhe Z., Zhicheng T., Laurence T. Y., Yang L., Mamoun A., "Adversarial malware sample generation method based on the prototype of deep learning detector", Computers & Security, vol.119, pp.102762, 2022. DOI.org/10.1016/j.cose.2022.102762.
[9]Xiao G., Li J., Chen Y., Li K., "MalFCS: An effective malware classification framework with automated feature extraction based on deep convolutional neural networks", Journal of Parallel and Distributed Computing, vol.141,pp. 49-58, 2020. DOI:10.1016/j.jpdc.2020.03.012.
[10]Anderson H.S., Kharkar A., Filar B., Roth P., “Evading machine learning malware detection. In Black Hat” 2017. URL: www.blackhat.com/docs/us-17/thursday/us-17-Anderson-Bot-Vs-Bot-Evading-Machine-Learning-Malware-Detection-wp.pdf
[11]Alzantot M., Sharma Y., Chakraborty S., Zhang H., Hsieh C.-J., Srivastava M. B., "GenAttack: Practical Black-box Attacks with Gradient-Free Optimization", Genetic and Evolutionary Computation Conference (GECCO), pp. 1111–1119, 2019. DOI.org/10.1145/3321707.3321749
[12]Demetrio L., Coull S. E., Biggio B., Lagorio G., Armando A., Roli F., "Adversarial Examples: A Survey and Experimental Evaluation of Practical Attacks on Machine Learning for Windows Malware Detection", ACM Transactions on Privacy and Security, vol.24, No.4, Article No.: 27,pp. 1–31, 2021. DOI.org/10.1145/3473039.
[13]Xintong. L., Qi. L., "An IRL-based malware adversarial generation method to evade anti-malware engines". Computers & Security, vol.104, No.C,pp. 102118, 2021. DOI: 10.1016/j.cose.2020.102118.
[14]Grosse, K., Papernot, N., Manoharan, P., Backes, M., McDaniel, P. "Adversarial Examples for Malware Detection". In: Foley, S., Gollmann, D., Snekkenes, E. Computer Security – (ESORICS),Vol 10493, 2017. DOI.org/10.1007/978-3-319-66399-9_4
[15]Yang, C., Xu, J., Liang, S. et al. "DeepMal: maliciousness-Preserving adversarial instruction learning against static malware detection". Cybersecurity, vol. 4,Article No.16, 2021. DOI.org/10.1186/s42400-021-00079-5
[16]Yuan J., Zhou S., Lin L., Wang F., Cui J., "Black-box adversarial attacks against deep learning based malware binaries detection with GAN", In: European Conference on Artificial Intelligence IOS Press, vol. 325, pp. 2536–2542. 2020.
[17]Castro R. L., Schmitt C., Dreo G., "AIMED: Evolving Malware with Genetic Programming to Evade Detection", 18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/13th IEEE International Conference On Big Data Science And Engineering (TrustCom/BigDataSE), pp. 240-247, 2019., DOI: 10.1109/TrustCom/BigDataSE.2019.00040.
[18]Wang X., Miikkulainen R., "MDEA: Malware detection with evolutionary adversarial learning", 2020, DOI.org/10.48550/arXiv.2002.03331
[19]Demetrio L., Biggio B., Lagorio G., Roli F., Armando A., "Functionality-preserving black-box optimization of adversarial windows malware", 2020, DOI.org/10.48550/arXiv.2003.13526.
[20]Y. Javier, Pardo G.E., Tapiador J., "Optimization of code caves in malware binaries to evade machine learning detectors", Computers & Security,vol.116,PP.102643, 2022.DOI.org/10.1016/j.cose.2022.102643.
[21]Kolosnjaji B., Demontis A., Biggio B., Maiorca D., Giacinto G., Eckert C., Roli F., "Adversarial malware binaries: Evading deep learning for malware detection in executables", In: European Signal Processing Conference, IEEE, pp. 533–537, 2018. URL: https://www.eurasip.org/Proceedings/Eusipco/Eusipco2018/papers/1570440156.pdf.
[22]Aydogan E., Sen, S."Automatic Generation of Mobile Malwares Using Genetic Programming", In European Conference on the Applications of Evolutionary Computation, ser. EvoApplications. Springer, vol.9028, pp. 745–756, 2015. doi.org/10.1007/978-3-319-16549-3_60
[23]Liu X., Du X., Zhang X., Zhu Q., Wang H., Guizani M. "Adversarial Samples on Android Malware Detection Systems for IoT Systems". Sensors (Basel). Vol.19,No.4, 974, 2019. doi: 10.3390/s19040974.
[24]Guangquan X., Hongfei S., Jingyi C., Hongpeng B., Jiliang L., Guangdong B., Shaoying L., Weizhi M., Xi Z., "GenDroid: A query-efficient black-box android adversarial attack framework", Computers & Security, vol. 132, 103359, 2023. DOI.org/10.1016/j.cose.2023.103359.
[25]Raff E., Barker J., Sylvester J., Brandon R., Catanzaro B., Nicholas C., "Malware Detection by Eating a Whole EXE", 2017. DOI.org/10.48550/arXiv.1710.09435
[26]Ahandani M.A., Vakil-Baghmisheh MT., Talebi M., "Hybridizing local search algorithms for global optimization". Comput Optim Appl, vol.59, pp. 725–748, 2014. DOI.org/10.1007/s10589-014-9652-1
[27]"Memetic algorithm–An overview "ScienceDirect Topics URL : https://www.sciencedirect.com/topics/computer-science/memeticAlgorithm #:~:text=Memetic%20algorithms%20(MAs)%20are%20evolutionary, search%20processes%20 to%20refine%20individuals.
[28]Kreuk, F., Barak, A. , Aviv-Reuven, S. , Baruch, M. , Pinkas, B. , Keshet, J. "Deceiving end-to-end deep learning malware detectors using adversarial examples", 2018. DOI.org/10.48550/arXiv.1802.04528
[29]https://github.com/iosifache/DikeDataset#description-%EF%B8%8F [Dike Dataset]

International Journal of Computer Network and Information Security (IJCNIS)