Data-driven evaluation of intrusion detectors: a methodological framework

Solayman Ayoubi (1) , Gregory Blanc (2, 3, 4, 5) , Houda Jmila (2, 6, 7, 5) , Thomas Silverston (1) , Sébastien Tixeuil (8, 9)

1 SIMBIOT - SIMulating and Building IOT
2 IP Paris - Institut Polytechnique de Paris
3 TSP - RST - Département Réseaux et Services de Télécommunications
4 SCN-SAMOVAR - Sécurité et Confiance Numérique
5 SAMOVAR - Services répartis, Architectures, MOdélisation, Validation, Administration des Réseaux
6 TSP - RS2M - Département Réseaux et Services Multimédia Mobiles
7 R3S-SAMOVAR - Réseaux, Systèmes, Services, Sécurité
8 NPA - Networks and Performance Analysis
9 LINCS - Laboratory of Information, Network and Communication Sciences

Intrusion detection systems are an important domain in cybersecurity research. Countless solutions have been proposed, continuously improving upon one another. Yet, and despite the introduction of distinct approaches, including machine-learning methods, the evaluation methodology has barely evolved. In this paper, we design a comprehensive evaluation framework for Machine Learning (ML)-based intrusion detection systems (IDS) and take into account the unique aspects of ML algorithms, their strengths and weaknesses. The framework design is inspired by both i) traditional IDS evaluation methods and ii) recommendations for evaluating ML algorithms in diverse application areas. Data quality being the key to machine learning, we focus on data-driven evaluation by exploring data-related issues. Our approach goes beyond evaluating intrusion detection performance (also known as effectiveness) and aims at proposing standard data manipulation methods to tackle robustness and stability. Finally, we evaluate our framework through a qualitative comparison with other IDS evaluation approaches from the state of the art.

Mots clés: Intrusion Detection System; Machine learning; Data-driven Evaluation; Evaluation Framework

Domaines: Informatique [cs]

Solayman Ayoubi, Gregory Blanc, Houda Jmila, Thomas Silverston, Sébastien Tixeuil. Data-driven evaluation of intrusion detectors: a methodological framework. FPS 2022 - 15th International Symposium on Foundations & Practice of Security, Dec 2022, Ottawa, ON, Canada. pp.142-157, ⟨10.1007/978-3-031-30122-3_9⟩. ⟨hal-04055085⟩