Public Thesis defense - ICTEAM

Analysis and classification of malware based on symbolic execution and machine learning by Charles-Henry Bertrand Van Ouytsel

In recent years, the development of new information and communication technologies has been accompanied by the emergence of new cyber threats. Malware, in particular, poses significant challenges to cybersecurity engineers. New obfuscation and evasion techniques are constantly being developed to modify existing malware and bypass security solutions. Security researchers must therefore constantly improve their detection tools to protect cyberspace.

New approaches based on machine learning could help researchers develop new security tools. However, most of these techniques rely on features that can be easily modified by malware authors. It is therefore necessary to explore new ways of representing malware behavior. Symbolic execution seems to be a good candidate to explore the different execution paths of malware and to reveal its malicious behavior.

This thesis is dedicated to the exploration of symbolic execution and machine learning for malware analysis. After reviewing existing related approaches, we introduce SEMA (Symbolic Execution for Malware Analysis), which uses symbolic execution to explore malware execution paths and extract System Call Dependency Graphs (SCDGs). SEMA uses various machine learning techniques to detect and identify malware based on SCDGs. We then provide a detailed analysis of machine learning for packing detection, a common obfuscation technique used by malware. Finally, we extend SEMA to support federated learning, enabling collaborative model training without direct dataset sharing.

Jury members:

Prof. Axel Legay (UCLouvain), promoteur

Prof. Peter Van Roy (UCLouvain), président

Prof. Charles Pecheur (UCLouvain), secrétaire

Prof. Olivier Bonaventure (UCLouvain)

Prof. Emilio Coppa (Sapienza University of Rome, Italie)

Dr. Thomas Given-Wilson (Diffblue, Royaume-Uni - collaborateur scientifique UCLouvain)

Prof. Johannes Kinder (LMU Munich, Allemagne)

Pay attention :

The public defense of Charles-Henry Bertrand Van Ouytsel scheduled for Thursday February 29 at 4:15 p.m. will also take place in the form of a video conference

Télécharger l'annonce.