Authors: Asif Ali Syed, Dr. Manav Thakur

Certificate: View Certificate

Abstract

Machine learning intrusion detectors are often judged on benchmarks with redundant features, severe class imbalance and evaluation shortcuts. This study builds and evaluates a hybrid framework that combines a correlation filter, a rank-aggregated feature ranking based on mutual information and LightGBM split gain, softened square-root cost-sensitive weighting, and a regularised LightGBM classifier with early stopping. Every design choice was settled on a validation subset carved from the official UNSW-NB15 training partition, and the official test partition of 82,332 flows was used once for final measurement. The pipeline retained 15 of 42 features. Square-root weighting gave the best validation macro-F1 (0.634 against 0.601 without treatment) and outperformed SMOTE and hybrid resampling. On the test partition, the framework reached a ten-class macro-F1 of 0.530, the highest of seven models, and a binary F1 of 0.890 with an AUC of 0.977. Its accuracy was, however, significantly lower than that of five of six baselines under McNemar's test, and its false alarm rate was 25.6%. An ablation study showed that stabilising the learner with regularisation and early stopping contributed 7.0 of the 9.4 points of macro-F1 gained over default LightGBM. Feature selection cut training time by 37.3%. SHAP analysis identified the source time-to-live as the dominant feature, yet removing it changed binary F1 by only 0.001. A nine-point gap between validation and test binary F1 was traced to distribution shift, not duplicate leakage. The framework improves rare-class detection at low training cost, but benchmark results on this dataset must be read with caution.

Introduction

1.1 Network Security in a Growing Digital Environment

Modern organisations depend on networks that carry ever larger volumes of traffic between cloud services, remote users and connected devices. Intrusion detection systems (IDS) are the main defensive layer that monitors this traffic and raises alerts when activity departs from expected behaviour (Khraisat et al., 2019). Signature-based systems miss novel attacks, while anomaly-based systems can flag unseen behaviour at the cost of more false alarms.

1.2 Machine Learning as a Basis for Outlier Detection

Machine learning has become the dominant approach to anomaly-based detection, because classifiers can learn the statistical signatures of attack families from labelled flow records instead of relying on hand-written rules. Surveys document a wide spread of methods, from decision trees and support vector machines to ensembles and deep neural networks (Ahmad et al., 2021; Liu & Lang, 2019). In this setting an attack flow can be viewed as an outlier with respect to normal traffic, and a supervised multi-class model can additionally identify which family it belongs to. Benchmark datasets such as UNSW-NB15 and CICIDS2017 provide common ground for comparing these methods (Moustafa & Slay, 2015; Sharafaldin et al., 2018).

1.3 Redundant Features and Imbalanced Traffic

Two properties of flow data complicate learning. First, flow features are often redundant: byte counters, packet counters and loss counters tend to rise together, and connection-count features computed over overlapping windows are strongly correlated. Redundant features add cost without adding information and can make models harder to interpret, which is why feature selection is regarded as one of the most influential design decisions in IDS research (Li et al., 2017; Thakkar & Lohiya, 2022). Second, attack families are highly imbalanced. In UNSW-NB15 the largest training class outnumbers the smallest by more than 400 to 1, so a classifier can achieve high accuracy while almost never recognising rare attacks. Resampling methods such as SMOTE and algorithm-level cost-sensitive learning are the two main remedies, but their benefits vary by dataset and neither is guaranteed to help (Fernández et al., 2018; Jiang et al., 2020).

1.4 Gradient Boosting for Efficient Detection

Gradient boosting decision trees have emerged as strong learners for tabular data, often matching or outperforming deep networks on heterogeneous, heavy-tailed features (Grinsztajn et al., 2022). XGBoost popularised regularised boosting at scale (Chen & Guestrin, 2016), and LightGBM extended it with histogram-based split finding, leaf-wise tree growth, Gradient-based One-Side Sampling and Exclusive Feature Bundling, which together reduce training cost substantially (Ke et al., 2017). LightGBM also supports per-sample weights and early stopping natively, which makes it a natural base for a pipeline that combines feature reduction with cost-sensitive learning.

1.5 Evaluation Rigour in Security Machine Learning

A growing body of work warns that many reported IDS results are optimistic. Common pitfalls include tuning on test data, leaking information through resampling performed before data splitting, relying on accuracy under class imbalance, and learning artefacts of the testbed instead of attack behaviour (Arp et al., 2022; Engelen et al., 2021). Benchmarks also have known limits in age and realism (Ring et al., 2019; Sarhan et al., 2022). Against this background, this paper presents a hybrid LightGBM framework that integrates a correlation filter, a rank-aggregated hybrid feature ranking, softened cost-sensitive weighting and a regularised, early-stopped learner, and evaluates it under a strict protocol on UNSW-NB15. It reports where the framework wins, ties and loses against six baselines, measures each component's contribution, examines interpretability with SHAP, and quantifies the gap between validation and test performance.

Conclusion

This study built and evaluated a hybrid LightGBM framework that integrates a correlation filter, a rank-aggregated hybrid feature ranking, softened cost-sensitive weighting and a regularised, early-stopped learner for network outlier detection on UNSW-NB15. Under a strict protocol that kept model selection separate from final evaluation, the framework reduced the feature set from 42 to 15, achieved the highest ten-class macro-F1 (0.530) and balanced accuracy (0.589) among seven models, and reached a binary F1 of 0.890 with an AUC of 0.977. Feature selection cut training time by 37.3% at a fixed number of trees, and square-root weighting outperformed SMOTE and hybrid resampling without adding training cost. The evidence also sets clear limits on these claims. The framework was significantly less accurate than five of six baselines at the flow level, it was not better than XGBoost or the random forest in binary detection, its false alarm rate of 25.6% is too high for direct deployment, and its inference was slow in this implementation. Rare classes such as Analysis and Backdoor remained poorly detected under every strategy. The ablation showed that most of the gain came from stabilising the learner, and a nine-point drop from validation to test exposed a distribution shift that no component of the framework addresses. Practitioners should therefore report macro-averaged and per-class metrics on the official test partition, never run gradient boosting with default fixed rounds on such data, soften rather than fully balance class weights, and test whether dominant explanatory features are true signals or testbed shortcuts. Future work should complete a full CICIDS2017 evaluation on a common feature set to test cross-dataset generalisation (Sarhan et al., 2022), apply the same class weights to strong baselines for a stricter comparison, repeat experiments across multiple random seeds, evaluate SMOTE variants designed for mixed data, and investigate threshold calibration and two-stage cascades to reduce false alarms.

References

1. Ahmad, Z., Shahid Khan, A., Wai Shiang, C., Abdullah, J., & Ahmad, F. (2021). Network intrusion detection system: A systematic study of machine learning and deep learning approaches. Transactions on Emerging Telecommunications Technologies, 32(1), Article e4150. https://doi.org/10.1002/ett.4150 2. Apruzzese, G., Colajanni, M., Ferretti, L., Guido, A., & Marchetti, M. (2018). On the effectiveness of machine and deep learning for cyber security. In 2018 10th International Conference on Cyber Conflict (CyCon) (pp. 371–390). IEEE. https://doi.org/10.23919/CYCON.2018.8405026 3. Arp, D., Quiring, E., Pendlebury, F., Warnecke, A., Pierazzi, F., Wressnegger, C., Cavallaro, L., & Rieck, K. (2022). Dos and don'ts of machine learning in computer security. In Proceedings of the 31st USENIX Security Symposium (pp. 3971–3988). USENIX Association. 4. Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 785–794). ACM. https://doi.org/10.1145/2939672.2939785 5. Engelen, G., Rimmer, V., & Joosen, W. (2021). Troubleshooting an intrusion detection dataset: The CICIDS2017 case study. In 2021 IEEE Security and Privacy Workshops (SPW) (pp. 7–12). IEEE. https://doi.org/10.1109/SPW53761.2021.00009 6. Fernández, A., García, S., Herrera, F., & Chawla, N. V. (2018). SMOTE for learning from imbalanced data: Progress and challenges, marking the 15-year anniversary. Journal of Artificial Intelligence Research, 61, 863–905. https://doi.org/10.1613/jair.1.11192 7. Grinsztajn, L., Oyallon, E., & Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on typical tabular data? In Advances in Neural Information Processing Systems 35 (pp. 507–520). Curran Associates. 8. Jiang, K., Wang, W., Wang, A., & Wu, H. (2020). Network intrusion detection combined hybrid sampling with deep hierarchical network. IEEE Access, 8, 32464–32476. https://doi.org/10.1109/ACCESS.2020.2973730 9. Kasongo, S. M., & Sun, Y. (2020). Performance analysis of intrusion detection systems using a feature selection method on the UNSW-NB15 dataset. Journal of Big Data, 7(1), Article 105. https://doi.org/10.1186/s40537-020-00379-6 10. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017). LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems 30 (pp. 3146–3154). Curran Associates. 11. Khraisat, A., Gondal, I., Vamplew, P., & Kamruzzaman, J. (2019). Survey of intrusion detection systems: Techniques, datasets and challenges. Cybersecurity, 2(1), Article 20. https://doi.org/10.1186/s42400-019-0038-7 12. Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R. P., Tang, J., & Liu, H. (2017). Feature selection: A data perspective. ACM Computing Surveys, 50(6), Article 94. https://doi.org/10.1145/3136625 13. Liu, H., & Lang, B. (2019). Machine learning and deep learning methods for intrusion detection systems: A survey. Applied Sciences, 9(20), Article 4396. https://doi.org/10.3390/app9204396 14. Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30 (pp. 4765–4774). Curran Associates. 15. Moustafa, N., & Slay, J. (2015). UNSW-NB15: A comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set). In 2015 Military Communications and Information Systems Conference (MilCIS) (pp. 1–6). IEEE. https://doi.org/10.1109/MilCIS.2015.7348942 16. Moustafa, N., Slay, J., & Creech, G. (2019). Novel geometric area analysis technique for anomaly detection using trapezoidal area estimation on large-scale networks. IEEE Transactions on Big Data, 5(4), 481–494. https://doi.org/10.1109/TBDATA.2017.2715166 17. Ring, M., Wunderlich, S., Scheuring, D., Landes, D., & Hotho, A. (2019). A survey of network-based intrusion detection data sets. Computers & Security, 86, 147–167. https://doi.org/10.1016/j.cose.2019.06.005 18. Sarhan, M., Layeghy, S., & Portmann, M. (2022). Towards a standard feature set for network intrusion detection system datasets. Mobile Networks and Applications, 27(1), 357–370. https://doi.org/10.1007/s11036-021-01843-0 19. Sharafaldin, I., Lashkari, A. H., & Ghorbani, A. A. (2018). Toward generating a new intrusion detection dataset and intrusion traffic characterization. In Proceedings of the 4th International Conference on Information Systems Security and Privacy (ICISSP 2018) (pp. 108–116). SciTePress. https://doi.org/10.5220/0006639801080116 20. Thakkar, A., & Lohiya, R. (2022). A survey on intrusion detection system: Feature selection, model, performance measures, application perspective, challenges, and future research directions. Artificial Intelligence Review, 55(1), 453–563. https://doi.org/10.1007/s10462-021-10037-9