Maximize
Bookmark

VX Heavens

Library Collection Sources Engines Constructors Simulators Utilities Links Forum

A study of anti-virus' response to unknown threats

Christophe Devine, Nicolas Richaud
EICAR 18th Annual Conference
May 2009

PDFDownload PDF (142.45Kb) (You need to be registered on forum)
[Back to index] [Comments (0)]
Christophe Devine <devine(@t)bob.cat>
Nicolas Richaud <nicolas.richaud(@t)lab.b-care.net>

Abstract

This study presents the evaluation of twelve anti-virus products with regards to programs not known from the signature files that show different kinds of malicious behavior. In practical terms, a set of twenty-one tests implementing various actions were developed; they cover key-logging, injection of code into other processes, network evasion, rootkit-like behaviour and exploitation of software vulnerabilities. The test programs were then run against each anti-virus program, and results were collected and consolidated. It was shown that all products tested here show deficiencies in at least one area, and some in all areas. For example, eleven anti-virus programs out of twelve still do not detect one code injection technique, which has been known for more than five years. Programs that spy on the user, such as recording the microphone, are not detected at all. Finally, this study provides recommendations to anti-virus vendors to enhance the capabilities of their products to detect malware, and improve safeguards against known attack techniques.

Introduction

Detection of malicious programs has traditionnaly relied on signature-based analysis. This method has the advantage of providing, in most cases, precise identification of the threat and relieves the user from the burden of making an informed decision. However, signatures may prove inadequate in several situations:

A new trend which has recently emerged is black-box detection of malware activity based on their behaviour, as exemplified by in the works of [4] and [5]. This method has the advantage of being able to detect malware in a more proactive fashion, at the cost of generating an higher number of false positives.

Rather than focusing on theoritical aspects of behavioural detection, this study concentrates on single test cases, each showing one particular method for performing a malicious action. Most surveys of anti-virus products only tested their signature-based detection engines; nevertheless, a few studies similar to this one do exist (such as [6]).

Methodology

Selection of anti-virus programs to be tested

The choice of products to be tested was based on their popularity, in order to cover the largest installed based as possible. Furthermore, time constraints would not have allowed testing the full range of all available anti-virus programs. Considering no recent and freely available study of anti-virus market share could be found, we relied on three denominators to make our decision:

An initial list of thirty-eight products was retrieved from Virustotal [8]. This list was then narrowed down to twelve products chosen for subsequent testing, and are shown as follows:

Product nameVersion tested
avast! professional edition4.8.1296
AVG Internet Security8.0.200
Avira Premium Security Suite8.2.0.252
BitDefender Total Security 200912.0.11.2
ESET Smart Security (NOD32)3.0.672.0
F-Secure Internet Security 20099.00 build 149
Kaspersky Anti-Virus For Windows Workstations6.0.3.837
McAfee Total Protection 200913.0.218
Norton 360 Version 2.02.5.0.5
Panda Internet Security 200914.00.00
Sophos Anti-Virus & Client Firewall7.6.2
Trend Micro Internet Security Pro17.0.1305

Table 1: List of evaluated anti-virus programs

A Windows XP operating system (english version) with SP3 integrated was installed inside a VMware virtual machine. Two accounts were created, one with administrative rights (named "localadmin"), and another without ("localuser"). No additional patches or configuration changes were applied. A snapshot of the virtual machine state was then made, which served as the install base as well as the control subject.

Then, each anti-virus was installed as a leaf of the snapshot made previously, and fully updated to the latest version of the signatures. After this step, access to the internet was removed by switching the network adapter from "NAT" to "Host-only" mode, to ensure the tests could be reproduced identically for all anti-virus programs. It is important to note the anti-virus programs were left in their default configuration. In a few cases, the user was asked about the type of network he was connected to; we always chose the most restrictive setting ("public network", "internet", etc.).

The installation phase was conducted between the 10th and 12th of December 2008.

Figure 1: VMware snapshot tree

Figure 1: VMware snapshot tree

Selection of the tests to be performed

Tests to be run were selected as being able to represent a wide range of malicious behaviors that may be found "in the wild". For this purpose, research articles documenting specific malware were consulted (notably [9] and [10]), as well as "hacking" tutorials available on the Internet.

Identified malicious behaviors were implemented as series of single tests. Each test only contained the strictest number of operations required for the action to be completed successfully (for example, capturing keystrokes). After a test was run, the virtual machine was reset to the current snapshot, to prevent unwanted interaction between tests.

Three checks were added in each test program to prevent accidental execution outside the virtual machine:

Some tests did require administrative privileges and are shown with [A] in from of them. Tests, which have been run from a non-privileged user account, are shown with [U].

The testing phase was conducted between the 14th and 19th of December 2008. Final tests were performed between the 7th and 9th of January 2009.

Limits of this study

Test results

Keyloggers

This series of tests includes six keylogging techniques, three of which can be run from user space and do not require administration privileges. Others require the loading of a kernel driver, and were originally developed by Thomas Sabono [12] for the purpose of testing anti-rootkit programs.

The tests were run for one minute, during which keys were entered. The output of each sample was then checked.

Product nametestA01testA02testA03testA11testA12testA13
avast!No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.
AVGNo alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.
AviraNo alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.
BitDefenderNo alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.
ESETNo alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.
F-SecureNo alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.
KasperskyNo alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.
McAfeeNo alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.
NortonNo alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.
PandaNo alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.
SophosNo alert; keys logged.No alert; keys logged.No alert; keys logged.User alerted; keys logged.User alerted; keys logged.User alerted; keys logged.
Trend MicroNo alert; keys logged.Program blocked; user alerted and prompted for action.No alert; keys logged.No alert; keys logged.No alert; keys logged.No alert; keys logged.

Table 2: Results of testing keyloggers

Figure 2: Kaspersky's "Proactive Defense" default configuration screen

Figure 2: Kaspersky's "Proactive Defense" default configuration screen

Code injection and network access

This series of tests stresses the capabilities of anti-virus programs to prevent the unauthorized hijacking of a process by another, as well as attempts to access the network (for example, to upload gathered information or send spam).

It should be noted that both testA22 and testA31 do not require the use of WriteProcessMemory(). Instead, the string "l32.dll" is searched inside the target executable, and the directory containing the DLL is added to the user's PATH environment variable.

Product nametestA21 (bind shell)testA21 (reverse connect)testA22testA23testA31testA32
avast!No alert; but incoming connection blocked.No alert; shell connected successfully.No alert; shell connected successfully.No alert; packets captured.No alert; DLL injected.No alert; DLL injected.
AVGIncoming connection detected and blocked; user alerted and prompted for action.Outgoing connection detected and blocked; user alerted and prompted for action.No alert; shell connected successfully.No alert; packets captured.No alert; DLL injected.No alert; DLL injected.
AviraListening socket blocked; user alerted and prompted for action.Outgoing connection detected and blocked; user alerted and prompted for action.No alert; shell connected successfully.Access to raw sockets blocked; user alerted and prompted for action.Program blocked; user alerted and prompted for action.No alert; DLL injected.
BitDefenderListening socket blocked; user alerted and prompted for action.Outgoing connection detected and blocked; user alerted and prompted for action.No alert; shell connected successfully.No alert; packets captured.No alert; DLL injected.No alert; DLL injected.
ESETNo alert; but incoming connections blocked.No alert; shell connected successfully.No alert; shell connected successfully.No alert; packets captured.No alert; DLL injected.No alert; DLL injected.
F-SecureListening socket blocked; user alerted and prompted for action.No alert; shell connected successfully.No alert; shell connected successfully.No alert; packets captured.No alert; DLL injected.No alert; DLL injected.
KasperskyNo alert; but incoming connection blocked.CMD shell execution blocked; user alerted and prompted for action.CMD shell execution blocked; user alerted and prompted for action.No alert; packets captured.Program blocked; user alerted and prompted for action.No alert; DLL injected.
McAfeeListening socket blocked; user alerted and prompted for action.No alert; shell connected successfully.No alert; shell connected successfully.No alert; packets captured.No alert; DLL injected.No alert; DLL injected.
NortonNo alert; shell connected successfully.No alert; shell connected successfully.No alert; shell connected successfully.No alert; packets captured.No alert; DLL injected.No alert; DLL injected.
PandaIncoming connection detected and blocked; user alerted and prompted for action.No alert; shell connected successfully.No alert; shell connected successfully.No alert; packets captured.No alert; DLL injected.No alert; DLL injected.
SophosListening socket blocked; user alerted and prompted for action.Outgoing connection detected and blocked; user alerted and prompted for action.Access to network blocked; user alerted and prompted for action.No alert; packets captured.No alert; DLL injected.No alert; DLL injected.
Trend MicroListening socket blocked; user alerted and prompted for action.Outgoing connection detected and blocked; user alerted and prompted for action.Attempt to execute Internet Explorer blocked; user alerted and prompted for action.No alert; packets captured.Program blocked; user alerted and prompted for action.Program blocked; user alerted and prompted for action.

Table 3: Results of testing code injection and network access

User-mode and kernel-mode malicious activities

This section contains three tests covering various user-monitoring activities, developed at Thales in 2008 by Jean-Jamil Khalifé during his internship. It also features another set of techniques representative of classic rootkit behavior.

Product nametestA41testA42testA43testA51testA52testA53
avast!No alert; clipboard contents captured.No alert; microphone recorded.No alert; screen captured.No alert; RAM modified; backdoor functional.No alert; MBR modified; backdoor functional.No alert; file hidden.
AVGNo alert; clipboard contents captured.No alert; microphone recorded.No alert; screen captured.No alert; RAM modified; backdoor functional.No alert; MBR modified; backdoor functional.No alert; file hidden.
AviraNo alert; clipboard contents captured.No alert; microphone recorded.No alert; screen captured.Detected as TR/Dropper.GENNo alert; MBR modified; backdoor functional.No alert; file hidden.
BitDefenderNo alert; clipboard contents captured.No alert; microphone recorded.No alert; screen captured.No alert; RAM modified; backdoor functional.No alert; MBR modified; backdoor functional.No alert; file hidden.
ESETNo alert; clipboard contents captured.No alert; microphone recorded.No alert; screen captured.No alert; RAM modified; backdoor functional.No alert; MBR modified; backdoor functional.No alert; file hidden.
F-SecureNo alert; clipboard contents captured.No alert; microphone recorded.No alert; screen captured.No alert; RAM modified; backdoor functional.No alert; MBR modified; backdoor functional.No alert; file hidden.
KasperskyNo alert; clipboard contents captured.No alert; microphone recorded.No alert; screen captured.Program blocked; user alerted and prompted for action.No alert; MBR modified; backdoor functional.No alert; file hidden.
McAfeeNo alert; clipboard contents captured.No alert; microphone recorded.No alert; screen captured.No alert; RAM modified; backdoor functional.No alert; MBR modified; backdoor functional.No alert; file hidden.
NortonNo alert; clipboard contents captured.No alert; microphone recorded.No alert; screen captured.No alert; RAM modified; backdoor functional.No alert; MBR modified; backdoor functional.No alert; file hidden.
PandaNo alert; clipboard contents captured.No alert; microphone recorded.No alert; screen captured.No alert; RAM modified; backdoor functional.No alert; MBR modified; backdoor functional.No alert; file hidden.
SophosNo alert; clipboard contents captured.No alert; microphone recorded.No alert; screen captured.No alert; RAM modified; backdoor functional.No alert; MBR modified; backdoor functional.User alerted; file hidden.
Trend MicroNo alert; clipboard contents captured.No alert; microphone recorded.No alert; screen captured.No alert; RAM modified; backdoor functional.No alert; MBR modified; backdoor functional.No alert; file hidden.

Table 4: Results of testing user-mode and kernel-mode malicious activities

Exploitation of vulnerabilities

Finally, this series of tests covers the exploitation of three relatively recent vulnerabilities.

Product nametestA61testA62testA63
avast!No alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.
AVGNo alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.
AviraNo alert; vulnerability exploitation successful.Detected as HTML/Shellcode.Gen; user alerted and prompted for action.No alert; vulnerability exploitation successful.
BitDefenderNo alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.
ESETNo alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.
F-SecureNo alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.
KasperskyCMD shell execution blocked; user alerted and prompted for action.No alert; vulnerability exploitation successful.No alert; but exploit failed silently.
McAfeeProgram blocked; user alerted and prompted for action.No alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.
NortonNo alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.
PandaNo alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.
SophosUser alerted; but vulnerability exploitation successful.Detected as Troj/PDFJs-B and quarantined; user alerted.No alert; vulnerability exploitation successful.
Trend MicroNo alert; vulnerability exploitation successful.No alert; vulnerability exploitation successful.Program blocked; user alerted and prompted for action.

Table 5: Results of testing the exploitation of vulnerabilities

Figure 3: Trend Micro's warning after exploiting the VLC vulnerability

Figure 3: Trend Micro's warning after exploiting the VLC vulnerability

Conclusion and future work

One main disadvantage of our testing methodology was the requirement to perform all tests by hand. This made running the test suite against the panel of anti-virus programs very time-consuming. A possible evolution will be to run each test automatically using a predefined script; this poses the problem of detecting if the malicious action completed successfully, as well as detecting if the anti-virus picked up the threat.

It may be tempting to add an increasing number of tests in the future. Those may not be pertinent however, as malware authors will generally use the simplest method not detected by anti-virus programs. Why use advanced code injection techniques when a classic windows hook remains undetected? As such, this study hopes to raise the bar for malware authors, by encouraging companies that produce anti-malware products to take into account the different techniques presented in this study.

Adding new behavioural patterns will of course pose the problem of false positives; this may be mitigated using whitelisting, as well as providing users with correct and informative alert messages.

Finally, it is to be hoped the problems and lost revenue caused by malware will loose relevance as more secure computing architecture come forward, such as those base on sandboxed virtual machine (Java, Flash...) and more fine-grained access control. In this regard, the addition of UAC in Windows Vista, however flawed it may be [21], is a step in the right direction.

References

  1. Zesheng Chen, Chuanyi Ji: An Information-Theoretical View of Network-Aware Malware Attacks. CoRR abs/0805.0802: (2008)
  2. "The Microsoft Security Response Center (MSRC) : Update on Microsoft Excel Vulnerability", as retrieved from http://blogs.technet.com/msrc/archive/2006/06/17/436860.aspx
  3. Shobha Venkataraman, Avrim Blum, Dawn Song: Limits of Learning-based Signature Generation with Adversaries. Proceedings of the 15th Annual Network and Distributed Systems Security Symposium (2008)
  4. Eric Filiol, Grègoire Jacob, Mickaël Le Liard: Evaluation methodology and theoretical model for antiviral behavioural detection strategies. Journal in Computer Virology 3(1): 23-37 (2007)
  5. Sèbastien Josse: Rootkit detection from outside the Matrix. Journal in Computer Virology 3(2): 113-123 (2007)
  6. Guillaume Kaddouch: Firewall Leak Tester, http://www.firewallleaktester.com/
  7. http://www.softpedia.com/get/Antivirus/
  8. http://www.virustotal.com/sobre.html
  9. Heng Yin, Zhenkai Liang, Dawn Song: HookFinder: Identifying and Understanding Malware Hooking Behaviors. Proceedings of ISOC NDSS 2008.
  10. Jamie Butler and Kris Kendal: Blackout: What Really Happened. Black Hat USA 2008
  11. Joanna Rutkowska: Red Pill... or how to detect VMM using (almost) one CPU instruction, retrieved from http://www.invisiblethings.org/papers/redpill.html
  12. Thomas Sabono: La fiabilitè des logiciels anti-rookits Windows 32 bits. SSTIC 2007
  13. "crazyload": Playing with Windows /dev/(k)mem. Phrack 59 (2002)
  14. Greg Hoglund: A real NT Rootkit, patching the NT Kernel. Phrack 55 (1999)
  15. Derek Soeder and Ryan Permeh: eEye BootRoot. Black Hat USA 2005.
  16. Greg Hoglund, James Butler: Rootkits: Subverting the Windows Kernel. Addison Wesley, ISBN 0-321-29431-9 (2006)
  17. Vulnerability in Server Service Could Allow Remote Code Execution, as retrieved from http://www.microsoft.com/technet/security/Bulletin/MS08-067.mspx
  18. http://www.metasploit.com/framework/download/
  19. CVE-2008-1104: http://cve.mitre.org/cgi-bin/cvename.cgi?name=2008-2992
  20. CVE-2008-4654: http://cve.mitre.org/cgi-bin/cvename.cgi?name=2008-4654
  21. Robert Paveza: User-Prompted Elevation of Unintended Code in Windows Vista, as retrieved from http://www.robpaveza.net/VistaUACExploit/UACExploitWhitepaper.pdf
[Back to index] [Comments (0)]
deenesitfrplruua