Christophe Devine, Nicolas Richaud
EICAR 18th Annual Conference
May 2009
This study presents the evaluation of twelve anti-virus products with regards to programs not known from the signature files that show different kinds of malicious behavior. In practical terms, a set of twenty-one tests implementing various actions were developed; they cover key-logging, injection of code into other processes, network evasion, rootkit-like behaviour and exploitation of software vulnerabilities. The test programs were then run against each anti-virus program, and results were collected and consolidated. It was shown that all products tested here show deficiencies in at least one area, and some in all areas. For example, eleven anti-virus programs out of twelve still do not detect one code injection technique, which has been known for more than five years. Programs that spy on the user, such as recording the microphone, are not detected at all. Finally, this study provides recommendations to anti-virus vendors to enhance the capabilities of their products to detect malware, and improve safeguards against known attack techniques.
Detection of malicious programs has traditionnaly relied on signature-based analysis. This method has the advantage of providing, in most cases, precise identification of the threat and relieves the user from the burden of making an informed decision. However, signatures may prove inadequate in several situations:
A new trend which has recently emerged is black-box detection of malware activity based on their behaviour, as exemplified by in the works of [4] and [5]. This method has the advantage of being able to detect malware in a more proactive fashion, at the cost of generating an higher number of false positives.
Rather than focusing on theoritical aspects of behavioural detection, this study concentrates on single test cases, each showing one particular method for performing a malicious action. Most surveys of anti-virus products only tested their signature-based detection engines; nevertheless, a few studies similar to this one do exist (such as [6]).
The choice of products to be tested was based on their popularity, in order to cover the largest installed based as possible. Furthermore, time constraints would not have allowed testing the full range of all available anti-virus programs. Considering no recent and freely available study of anti-virus market share could be found, we relied on three denominators to make our decision:
An initial list of thirty-eight products was retrieved from Virustotal [8]. This list was then narrowed down to twelve products chosen for subsequent testing, and are shown as follows:
Product name | Version tested |
---|---|
avast! professional edition | 4.8.1296 |
AVG Internet Security | 8.0.200 |
Avira Premium Security Suite | 8.2.0.252 |
BitDefender Total Security 2009 | 12.0.11.2 |
ESET Smart Security (NOD32) | 3.0.672.0 |
F-Secure Internet Security 2009 | 9.00 build 149 |
Kaspersky Anti-Virus For Windows Workstations | 6.0.3.837 |
McAfee Total Protection 2009 | 13.0.218 |
Norton 360 Version 2.0 | 2.5.0.5 |
Panda Internet Security 2009 | 14.00.00 |
Sophos Anti-Virus & Client Firewall | 7.6.2 |
Trend Micro Internet Security Pro | 17.0.1305 |
Table 1: List of evaluated anti-virus programs
A Windows XP operating system (english version) with SP3 integrated was installed inside a VMware virtual machine. Two accounts were created, one with administrative rights (named "localadmin"), and another without ("localuser"). No additional patches or configuration changes were applied. A snapshot of the virtual machine state was then made, which served as the install base as well as the control subject.
Then, each anti-virus was installed as a leaf of the snapshot made previously, and fully updated to the latest version of the signatures. After this step, access to the internet was removed by switching the network adapter from "NAT" to "Host-only" mode, to ensure the tests could be reproduced identically for all anti-virus programs. It is important to note the anti-virus programs were left in their default configuration. In a few cases, the user was asked about the type of network he was connected to; we always chose the most restrictive setting ("public network", "internet", etc.).
The installation phase was conducted between the 10th and 12th of December 2008.
Figure 1: VMware snapshot tree
Tests to be run were selected as being able to represent a wide range of malicious behaviors that may be found "in the wild". For this purpose, research articles documenting specific malware were consulted (notably [9] and [10]), as well as "hacking" tutorials available on the Internet.
Identified malicious behaviors were implemented as series of single tests. Each test only contained the strictest number of operations required for the action to be completed successfully (for example, capturing keystrokes). After a test was run, the virtual machine was reset to the current snapshot, to prevent unwanted interaction between tests.
Three checks were added in each test program to prevent accidental execution outside the virtual machine:
Some tests did require administrative privileges and are shown with [A] in from of them. Tests, which have been run from a non-privileged user account, are shown with [U].
The testing phase was conducted between the 14th and 19th of December 2008. Final tests were performed between the 7th and 9th of January 2009.
This series of tests includes six keylogging techniques, three of which can be run from user space and do not require administration privileges. Others require the loading of a kernel driver, and were originally developed by Thomas Sabono [12] for the purpose of testing anti-rootkit programs.
The tests were run for one minute, during which keys were entered. The output of each sample was then checked.
Product name | testA01 | testA02 | testA03 | testA11 | testA12 | testA13 |
---|---|---|---|---|---|---|
avast! | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. |
AVG | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. |
Avira | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. |
BitDefender | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. |
ESET | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. |
F-Secure | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. |
Kaspersky | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. |
McAfee | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. |
Norton | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. |
Panda | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. |
Sophos | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | User alerted; keys logged. | User alerted; keys logged. | User alerted; keys logged. |
Trend Micro | No alert; keys logged. | Program blocked; user alerted and prompted for action. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. | No alert; keys logged. |
Table 2: Results of testing keyloggers
Figure 2: Kaspersky's "Proactive Defense" default configuration screen
This series of tests stresses the capabilities of anti-virus programs to prevent the unauthorized hijacking of a process by another, as well as attempts to access the network (for example, to upload gathered information or send spam).
It should be noted that both testA22 and testA31 do not require the use of WriteProcessMemory(). Instead, the string "l32.dll" is searched inside the target executable, and the directory containing the DLL is added to the user's PATH environment variable.
Product name | testA21 (bind shell) | testA21 (reverse connect) | testA22 | testA23 | testA31 | testA32 |
---|---|---|---|---|---|---|
avast! | No alert; but incoming connection blocked. | No alert; shell connected successfully. | No alert; shell connected successfully. | No alert; packets captured. | No alert; DLL injected. | No alert; DLL injected. |
AVG | Incoming connection detected and blocked; user alerted and prompted for action. | Outgoing connection detected and blocked; user alerted and prompted for action. | No alert; shell connected successfully. | No alert; packets captured. | No alert; DLL injected. | No alert; DLL injected. |
Avira | Listening socket blocked; user alerted and prompted for action. | Outgoing connection detected and blocked; user alerted and prompted for action. | No alert; shell connected successfully. | Access to raw sockets blocked; user alerted and prompted for action. | Program blocked; user alerted and prompted for action. | No alert; DLL injected. |
BitDefender | Listening socket blocked; user alerted and prompted for action. | Outgoing connection detected and blocked; user alerted and prompted for action. | No alert; shell connected successfully. | No alert; packets captured. | No alert; DLL injected. | No alert; DLL injected. |
ESET | No alert; but incoming connections blocked. | No alert; shell connected successfully. | No alert; shell connected successfully. | No alert; packets captured. | No alert; DLL injected. | No alert; DLL injected. |
F-Secure | Listening socket blocked; user alerted and prompted for action. | No alert; shell connected successfully. | No alert; shell connected successfully. | No alert; packets captured. | No alert; DLL injected. | No alert; DLL injected. |
Kaspersky | No alert; but incoming connection blocked. | CMD shell execution blocked; user alerted and prompted for action. | CMD shell execution blocked; user alerted and prompted for action. | No alert; packets captured. | Program blocked; user alerted and prompted for action. | No alert; DLL injected. |
McAfee | Listening socket blocked; user alerted and prompted for action. | No alert; shell connected successfully. | No alert; shell connected successfully. | No alert; packets captured. | No alert; DLL injected. | No alert; DLL injected. |
Norton | No alert; shell connected successfully. | No alert; shell connected successfully. | No alert; shell connected successfully. | No alert; packets captured. | No alert; DLL injected. | No alert; DLL injected. |
Panda | Incoming connection detected and blocked; user alerted and prompted for action. | No alert; shell connected successfully. | No alert; shell connected successfully. | No alert; packets captured. | No alert; DLL injected. | No alert; DLL injected. |
Sophos | Listening socket blocked; user alerted and prompted for action. | Outgoing connection detected and blocked; user alerted and prompted for action. | Access to network blocked; user alerted and prompted for action. | No alert; packets captured. | No alert; DLL injected. | No alert; DLL injected. |
Trend Micro | Listening socket blocked; user alerted and prompted for action. | Outgoing connection detected and blocked; user alerted and prompted for action. | Attempt to execute Internet Explorer blocked; user alerted and prompted for action. | No alert; packets captured. | Program blocked; user alerted and prompted for action. | Program blocked; user alerted and prompted for action. |
Table 3: Results of testing code injection and network access
This section contains three tests covering various user-monitoring activities, developed at Thales in 2008 by Jean-Jamil Khalifé during his internship. It also features another set of techniques representative of classic rootkit behavior.
Product name | testA41 | testA42 | testA43 | testA51 | testA52 | testA53 |
---|---|---|---|---|---|---|
avast! | No alert; clipboard contents captured. | No alert; microphone recorded. | No alert; screen captured. | No alert; RAM modified; backdoor functional. | No alert; MBR modified; backdoor functional. | No alert; file hidden. |
AVG | No alert; clipboard contents captured. | No alert; microphone recorded. | No alert; screen captured. | No alert; RAM modified; backdoor functional. | No alert; MBR modified; backdoor functional. | No alert; file hidden. |
Avira | No alert; clipboard contents captured. | No alert; microphone recorded. | No alert; screen captured. | Detected as TR/Dropper.GEN | No alert; MBR modified; backdoor functional. | No alert; file hidden. |
BitDefender | No alert; clipboard contents captured. | No alert; microphone recorded. | No alert; screen captured. | No alert; RAM modified; backdoor functional. | No alert; MBR modified; backdoor functional. | No alert; file hidden. |
ESET | No alert; clipboard contents captured. | No alert; microphone recorded. | No alert; screen captured. | No alert; RAM modified; backdoor functional. | No alert; MBR modified; backdoor functional. | No alert; file hidden. |
F-Secure | No alert; clipboard contents captured. | No alert; microphone recorded. | No alert; screen captured. | No alert; RAM modified; backdoor functional. | No alert; MBR modified; backdoor functional. | No alert; file hidden. |
Kaspersky | No alert; clipboard contents captured. | No alert; microphone recorded. | No alert; screen captured. | Program blocked; user alerted and prompted for action. | No alert; MBR modified; backdoor functional. | No alert; file hidden. |
McAfee | No alert; clipboard contents captured. | No alert; microphone recorded. | No alert; screen captured. | No alert; RAM modified; backdoor functional. | No alert; MBR modified; backdoor functional. | No alert; file hidden. |
Norton | No alert; clipboard contents captured. | No alert; microphone recorded. | No alert; screen captured. | No alert; RAM modified; backdoor functional. | No alert; MBR modified; backdoor functional. | No alert; file hidden. |
Panda | No alert; clipboard contents captured. | No alert; microphone recorded. | No alert; screen captured. | No alert; RAM modified; backdoor functional. | No alert; MBR modified; backdoor functional. | No alert; file hidden. |
Sophos | No alert; clipboard contents captured. | No alert; microphone recorded. | No alert; screen captured. | No alert; RAM modified; backdoor functional. | No alert; MBR modified; backdoor functional. | User alerted; file hidden. |
Trend Micro | No alert; clipboard contents captured. | No alert; microphone recorded. | No alert; screen captured. | No alert; RAM modified; backdoor functional. | No alert; MBR modified; backdoor functional. | No alert; file hidden. |
Table 4: Results of testing user-mode and kernel-mode malicious activities
Finally, this series of tests covers the exploitation of three relatively recent vulnerabilities.
Product name | testA61 | testA62 | testA63 |
---|---|---|---|
avast! | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. |
AVG | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. |
Avira | No alert; vulnerability exploitation successful. | Detected as HTML/Shellcode.Gen; user alerted and prompted for action. | No alert; vulnerability exploitation successful. |
BitDefender | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. |
ESET | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. |
F-Secure | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. |
Kaspersky | CMD shell execution blocked; user alerted and prompted for action. | No alert; vulnerability exploitation successful. | No alert; but exploit failed silently. |
McAfee | Program blocked; user alerted and prompted for action. | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. |
Norton | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. |
Panda | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. |
Sophos | User alerted; but vulnerability exploitation successful. | Detected as Troj/PDFJs-B and quarantined; user alerted. | No alert; vulnerability exploitation successful. |
Trend Micro | No alert; vulnerability exploitation successful. | No alert; vulnerability exploitation successful. | Program blocked; user alerted and prompted for action. |
Table 5: Results of testing the exploitation of vulnerabilities
Figure 3: Trend Micro's warning after exploiting the VLC vulnerability
One main disadvantage of our testing methodology was the requirement to perform all tests by hand. This made running the test suite against the panel of anti-virus programs very time-consuming. A possible evolution will be to run each test automatically using a predefined script; this poses the problem of detecting if the malicious action completed successfully, as well as detecting if the anti-virus picked up the threat.
It may be tempting to add an increasing number of tests in the future. Those may not be pertinent however, as malware authors will generally use the simplest method not detected by anti-virus programs. Why use advanced code injection techniques when a classic windows hook remains undetected? As such, this study hopes to raise the bar for malware authors, by encouraging companies that produce anti-malware products to take into account the different techniques presented in this study.
Adding new behavioural patterns will of course pose the problem of false positives; this may be mitigated using whitelisting, as well as providing users with correct and informative alert messages.
Finally, it is to be hoped the problems and lost revenue caused by malware will loose relevance as more secure computing architecture come forward, such as those base on sandboxed virtual machine (Java, Flash...) and more fine-grained access control. In this regard, the addition of UAC in Windows Vista, however flawed it may be [21], is a step in the right direction.