The amount and the complexity of malicious activity increasing and evolving day by day. Typical static code analysis is futile when challenged by diverse variants. The prolog of new malware samples every day is not uncommon and the malware designed by the attackers have the ability to change as they propagate. Thus, automated dynamic malware analysis becomes a widely preferred technique for the identification of unknown malware.
In this paper, an automated malware detection system is presented based on dynamic malware analysis approach. The behavior of malware is observed in the controlled environment of the popular malware analysis system. It uses the clustering and classification of embedded malware behavior reports to identify the presence of malicious behavior. Based on the experimentation and evaluation it is evident that the proposed system is able to achieve better F-measures, FPR, FNR, TPR and TNR values resulting in accurate classification leading to more efficient detection of unknown malware compared to the traditional hierarchical classification approach.