Quick Links

International Journal of Advanced Innovative Technology in Engineering (IJAITE)


Artificial Intelligence for Malware Detection in Software-Defined Networks: A Comprehensive Systematic Literature Review

Sudhakar Honaji Yerme, Dr. Prabhakar L. Ramteke

Abstract :

Software-Defined Networking has emerged as a fundamental networking paradigm for cloud computing, Internet of Things, fifth-generation (5G) communication, and data-center infrastructures because of its centralized control, programmability, and flexible network management. However, the logical centralization of the control plane also introduces significant security vulnerabilities, making SDN increasingly susceptible to malware, botnets, ransomware, Distributed Denial-of-Service, and other sophisticated cyberattacks. Artificial Intelligence (AI)-based malware detection techniques have gained considerable attention due to their capability to identify complex and previously unseen attack patterns. This paper presents a comprehensive Systematic Literature Review of intelligent malware detection approaches for SDN by following the PRISMA 2020 framework and Kitchenham guidelines. A total of 30 primary studies published between 2020 and 2026 were systematically selected, assessed, and synthesized. The reviewed literature was classified into three major categories: Machine Learning, Deep Learning, and Hybrid AI approaches, followed by comprehensive comparative analyses of datasets, learning algorithms, feature engineering strategies, evaluation metrics, detection performance, and reported limitations. The quantitative synthesis indicates a clear research transition from conventional ML techniques toward deep learning and hybrid intelligence frameworks, with hybrid models consistently shows the highest detection performance, frequently exceeding 99% detection accuracy. The analysis further reveals that Random Forest, Support Vector Machine, Convolutional Neural Networks, Long Short-Term Memory networks, Deep Neural Networks, and CNN–LSTM hybrid architectures are among the most widely adopted algorithms, whereas NSL-KDD, InSDN, UNSW-NB15, CICIDS2017, and IoT-23 remain the dominant evaluation datasets. The review identifies several persistent challenges, including dependence on benchmark datasets, limited real-world SDN validation, class imbalance, high computational complexity, insufficient explainability, limited cross-dataset generalization, and the absence of standardized benchmarking protocols. The review outlines future research directions emphasizing lightweight and explainable AI models, graph neural networks, federated and continual learning, adaptive hybrid intelligence, and standardized real-world evaluation frameworks to support the development of accurate, scalable, robust, and deployable malware detection systems for next-generation Software-Defined Networks.

Keywords :

Software-Defined Networking; Malware Detection; Intrusion Detection; Artificial Intelligence; Machine Learning; Deep Learning; Hybrid Intelligence; Cy

Full Text :

Download PDF

DOI : 10.65809/IJAITE/26/v11i04/002

Cite this paper :

Sudhakar Honaji Yerme, Dr. Prabhakar L. Ramteke, "Artificial Intelligence for Malware Detection in Software-Defined Networks: A Comprehensive Systematic Literature Review", International Journal of Advanced Innovative Technology in Engineering, 11(4), 2026, PP 18-45. DOI: 10.65809/IJAITE/26/v11i04/002

References :

[1] Magnaye, N. A. (2024). Advancements in computer network technologies: A review. https://doi.org/10.54517/m.v5i1.2315

[2] Medappa, P. K. (2025). Software Defined Networking Transforming Traditional Network Architecture for the future. International Journal For Multidisciplinary Research, 7(2). https://doi.org/10.36948/ijfmr.2025.v07i02.49460

[3] Ikhioya, E. (2026). Security Threat Mitigation in SDN. British Journal of Computer, Networking and Information Technology. https://doi.org/10.52589/bjcnit-xzpz0sjz

[4] Ganame, K., Allaire, M. A., Zagdene, G., & Boudar, O. (2017). Network Behavioral Analysis for Zero-Day Malware Detection – A Case Study (pp. 169–181). Springer, Cham. https://doi.org/10.1007/978-3-319-69155-8_13

[5] Sonthalia, N., Reddy, E. V. A., Pagaria, H., & Jayasri, G. V. (2023). Using Machine Learning in Software Defined Networks to Recognize and Avoid DDOS Attacks. International Journal For Science Technology And Engineering, 11(3), 1045–1050. https://doi.org/10.22214/ijraset.2023.49565

[6] Ali, Q. I., Assim, O. M., Talal, Z., & Younis, N. Th. (2025). Software Defined Networks with Artificial Intelligence: A Comprehensive Analysis and Review. Journal of Advances in Computer Networks, 13(2), 31–36. https://doi.org/10.18178/jacn.2025.13.2.296

[7] Jouilili, A., Hantouti, H., & El Ouazzani, R. (2024). Enhancing AI-Driven Intrusion Detection in SDN: Exploring the Power of Dimensionality Reduction Techniques. https://doi.org/10.1109/iraset60544.2024.10549433

[8] Tavangari, S. (2024). A Comparative Analysis of Deep Learning Architectures for Real-Time Anomaly Detection in Software-Defined Networks. https://doi.org/10.20944/preprints202410.1050.v1

[9] Malik, J., Akhunzada, A., Bibi, I., Imran, M., Musaddiq, A., & Kim, S. W. (2020). Hybrid Deep Learning: An Efficient Reconnaissance and Surveillance Detection Mechanism in SDN. IEEE Access, 8, 134695–134706. https://doi.org/10.1109/ACCESS.2020.3009849

[10] Rzym, G., Masny, A., & Chołda, P. (2024). Dynamic Telemetry and Deep Neural Networks for Anomaly Detection in 6G Software-Defined Networks. Electronics. https://doi.org/10.3390/electronics13020382

[11] Danang, D., Dianta, I. A., & Santoso, A. B. (2025). Hybrid CNN GRU Framework for Early Detection and Adaptive Mitigation of DDoS Attacks in SDN using Image Based Traffic Analysis. 2(2), 57–70. https://doi.org/10.62951/ijies.v2i2.292

[12] Elijah, T. D., & Familusi, O. B. (2025). AI-Powered Intrusion Detection and Prevention Systems for the Next Generation Network. Traektoriâ Nauki, 11(10), 2001. https://doi.org/10.22178/pos.123-3

[13] Hidayat, M., Kusrini, K., Utami, E., Setyanto, A., & Karim, A. (2025). Advancements in Machine Learning and Deep Learning for Malware Detection Challenges Breakthroughs. 1–6. https://doi.org/10.1109/iccit65724.2025.11167593

[14] Anbar, M. (2023). A Systematic Literature Review on Machine Learning and Deep Learning Approaches for Detecting DDoS Attacks in Software-Defined Networking. Sensors, 23(9), 4441. https://doi.org/10.3390/s23094441

[15] Zmaimita, H., Madani, A., & Zine-Dine, K. (2025). Machine and Deep Learning for Intrusion Detection: A PRISMA-Guided Systematic Review of Recent Advances. Register: Jurnal Ilmiah Teknologi Sistem Informasi, 11(1), 66–74. https://doi.org/10.26594/register.v11i1.5589

[16] Alsmadi, I., & Zarour, M. (2017). Empirical Evidences in Software-Defined Network Security: A Systematic Literature Review (pp. 253–295). Springer, Cham. https://doi.org/10.1007/978-3-319-44257-0_11

[17] Gaurav, A., Gupta, B. B., Chui, K. T., Arya, V., & Wu, J. (2024). Enhancing Intrusion Detection in Software Defined Networks with Optimized Feature Selection and Logistic Regression. 1809–1815. https://doi.org/10.1109/iccworkshops59551.2024.10615911

[18] Hassan, H. A., Hemdan, E. E.-D., El-Shafai, W., Shokair, M., & Abd El-Samie, F. E. (2023). Detection of attacks on software defined networks using machine learning techniques and imbalanced data handling methods. Security and Privacy. https://doi.org/10.1002/spy2.350

[19] Shinde, S. R., & Pujeri, U. (2025). Malware Detection Using Machine Learning: A Neural Network-Based Approach. 1–5. https://doi.org/10.1109/i4tech64670.2025.11277839

[20] Alzahrani, A. O., & Alenazi, M. J. F. (2021). Designing a Network Intrusion Detection System Based on Machine Learning for Software Defined Networks. Future Internet, 13(5), 111. https://doi.org/10.3390/FI13050111

[21] Shukla, U., Sapkota, B., & Dawadi, B. R. (2025). Reinforcement Learning based Malware mitigation in balanced Multicontroller Software Defined Networking enabled Internet of Things Networks. Journal of Innovations in Engineering Education. https://doi.org/10.3126/jiee.v8i1.86281

[22] Sengayo, N., & Basikolo, T. (2024). Multistage classification in SDN security: a machine learning approach using real-world data for enhanced intrusion and vulnerability detection. ITU Journal, 5(4), 433–446. https://doi.org/10.52953/bzoj6066

[23] Dalgade, D., Patyane, S., Matey, A., Singh, S., & Godbole, A. (2024). Malware Detection using Machine Learning. International Journal of Innovative Science and Research Technology. https://doi.org/10.38124/ijisrt/ijisrt24apr1102

[24] Hirsi, A., Audah, L., Salh, A., Sahar, N. M., & Alhartomi, M. A. (2024). DDoS Anomaly Detection in Software-Defined Networks: An Evaluation of Machine Learning Techniques for Traffic Classification and Prediction. 100–105. https://doi.org/10.1109/icftss61109.2024.10691328

[25] Nguyen, V.-T., Hoang, V.-C., Nguyen, X.-H., & Le, K. H. (2022). Towards a high-performance threat-aware system for software-defined networks. International Conference on Autonomic and Trusted Computing, 280–285. https://doi.org/10.1109/ATC55345.2022.9942972

[26] Agrawal, A., Bhushan, B., Sharma, H., Hameed, A. A., & Jamil, A. (2025). Advancing Intrusion Detection in Software-Defined Networks. 1–6. https://doi.org/10.1109/satc65530.2025.11136867

[27] Li, C., Feng, C., Cai, X., Yang, F., & Wang, Y. (2026). Research on deep learning-based malicious traffic detection and defense mechanism. 110. https://doi.org/10.1117/12.3086415

[28] Yang, J. (2024). A new Attacks Intrusion Detection Model Based on Deep Learning in Software-Defined Networking Environments. 430–436. https://doi.org/10.1109/mlise62164.2024.10674546

[29] Ingle, D. (2025). Enhancing malware detection and classification in network traffic using deep learning techniques. Journal of Forensic Sciences. https://doi.org/10.1111/1556-4029.70189

[30] Saher, M. A. (2022). A Novel Approach To Network Intrusion Detection System Using Deep Learning For Sdn: Futuristic Approach. https://doi.org/10.48550/arxiv.2208.02094

[31] Hadi, M. R., & Mohammed, A. S. (2022). A Novel Approach To Network Intrusion Detection System Using Deep Learning For Sdn: Futuristic Approach. abs/2208.02094. https://doi.org/10.5121/csit.2022.121106

[32] Jeebodh, M. R., & Baliyan, N. (2024). IoT Malware Detection Using Deep Learning. 1–6. https://doi.org/10.1109/icccnt61001.2024.10724403

[33] Korsten, H. H. M., & Reddy, K. S. (2025). DeepSDN: Deep Learning Based Software Defined Network Model for Cyberthreat Detection in IoT Network. ACM Transactions on Internet Technology. https://doi.org/10.1145/3737875

[34] Elubeyd, H., & Yiltas-Kaplan, D. (2023). Hybrid Deep Learning Approach for Automatic Dos/DDoS Attacks Detection in Software-Defined Networks. Applied Sciences, 13(6), 3828. https://doi.org/10.3390/app13063828

[35] Sharma, A., & Saxena, P. (2024). Distributed Denial of Service Attack Detection and Prevention Using Pipit Fox-Attentional Deep Learning in Software-Defined Networking. International Journal of Image and Graphics. https://doi.org/10.1142/s0219467826500294

[36] Choobdar, P., Naderan, M., & Naderan, M. (2021). Detection and Multi-Class Classification of Intrusion in Software Defined Networks Using Stacked Auto-Encoders and CICIDS2017 Dataset. Wireless Personal Communications, 1–35. https://doi.org/10.1007/S11277-021-09139-Y

[37] Souza, C. H. M., & Arima, C. H. (2024). A hybrid approach for malware detection in SDN‐enabled IoT scenarios. Internet Technology Letters. https://doi.org/10.1002/itl2.534

[38] Elubeyd, H., & Yiltas-Kaplan, D. (2023). Hybrid Deep Learning Approach for Automatic Dos/DDoS Attacks Detection in Software-Defined Networks. Applied Sciences, 13(6), 3828. https://doi.org/10.3390/app13063828

[39] Latah, M., & Toker, L. (2020). An efficient flow-based multi-level hybrid intrusion detection system for software-defined networks. 3(3), 261–271. https://doi.org/10.1007/S42045-020-00040-Z

[40] Shihab, D., Abdulhameed, A. A., & Gaata, M. T. G. (2025). Optimized Hybrid CNN-LSTM Framework with Multi-Feature Analysis and SMOTE for Intrusion Detection in SDN. https://doi.org/10.61710/kjcs.v3i4.132

[41] Kaur, P. (2025). Malware detection using classification technique and hybrid optimization technique. International Journal of Apllied Mathematics, 38(9s), 1393–1406. https://doi.org/10.12732/ijam.v38i9s.866

[42] Kanimozhi, R., & Ramesh, P. S. (2025). Deep reinforcement learning-based intrusion detection scheme for software-defined networking. Dental Science Reports, 15(1), 38827. https://doi.org/10.1038/s41598-025-24869-w

[43] Alasmari, S., & Mubarak, G. (2024). Android Malware Detection using TripleGuard Neural Network and Hybrid Bird Mating with Battle Royal Optimization. https://doi.org/10.21203/rs.3.rs-5434673/v1

[44] Abdallah, M., Khac, N. A. L., Jahromi, H. Z., & Jurcut, A. D. (2021). A Hybrid CNN-LSTM Based Approach for Anomaly Detection Systems in SDNs. Availability, Reliability and Security. https://doi.org/10.1145/3465481.3469190

[45] Logeswari, G., Bose, S., Vijayaraj, G., Gokulraj, G., Maheswaran, N., & Varshini, I. C. (2025). RHFS-SEC: A Hybridized Feature Ranking and Soft-Ensemble Approach for Intelligent Intrusion Detection in SDN Environments. 957–963. https://doi.org/10.1109/icces67310.2025.11336910

[46] Nawshin, S., Islam, S., & Shatabda, S. (2024). PCA-ANN: Feature selection based hybrid intrusion detection system in software defined network. https://doi.org/10.3233/jifs-236340