Malwares – Types & Defense

1. Malwares – Types & Defense Raghunathan Srinivasan Sept 25, 2007 CSE 466/598 Computer Systems Security

2. Malware • How to define malware? • Over a broad sense, any malicious program • Types • Viruses • Trojans • Rootkits • Spyware

3. Virus • A program that can attach itself to another program • Can replicate • Encrypted • How to prevent them? • Anti–virus??? • How do they work • No – real Answer

4. Types of Viruses [Evolution?] • Parasitic Viruses • Also known as file infectors • Date / Logic bomb • Michaelangelo, Sunday, Century • Macro Viruses • Infect macro utility feature in word • Encrypted Virus • cascade • Polymorphic Virus • 1260 • Stealth Virus

5. Encrypted Viruses Decryption engine • Viruses have certain patterns present in them • Signatures • AV looks for these patterns in files • To avoid detection, the virus encrypts itself • Mov • Fetch • ###$$$ • &&^^^^ Encrypted virus body

6. Encrypted virus • It is not possible to find out what the encrypted text is • So how to find if an encrypted entity is a virus? • Look in previous slide • Next step – polymorphic viruses

7. Polymorphic • Can change form from infection to infection • There is a mutation engine present in the virus body • During run time – the virus loads the mutation engine • The ME changed the decryption routine • The virus changes form on every encryption • Now the virus is difficult to spot

8. Detection • Creating random encryption – decryption routines is difficult • See how many badly designed encryption algorithms are present • CSS • Hence encryption is weak, can be broken • Can this be reliable? • No • Then what to do?

9. Detection • AV scanners use what is known as simulation • They create a virtual PC in the RAM • Load the program in the Virtual PC • The program executes, and shows its true behavior eventually • You can read the following paper for further details • Understanding and Managing polymorphic viruses – • Google it, it’s a white paper by Symantec

10. A new trend in Virus • Viruses have become complex • Anti-Virus programs are running powerful engines • game of cat and mouse • What further can virus writers do to prevent detection • Go stealth • Install rootkits • Install portions of program in various other executables • Disable detectors?

11. Disabling detectors • If you don’t have a defense mechanism, you cant escape infection • Kill all security processes • Works, but a smart user can figure something • Patch on the definitions • Patch on the program policies • How does that help • Disable updates?

12. Examples of such viruses • SpamThru • Locates existing AV in the machine • Patches them to prevent updates • Installs its own virus scanner • Why? • Beast • Kills all existing security services • Hooks on to winlogon.exe • What is winlogon.exe

13. Implications? • AV does not function • No method to detect the presence of viruses • How to solve this? • Borrow some virus tricks • Hide the AV • Move the program code • Hide files • Hide Process name

14. Other Miscellaneous Malware • Worms • Self replicating program • Does not require host to replicate • It uses the network to send copies of itself • They use the bandwidth and harm the network • Viruses harm the computer (host) • Does worm not harm the PC? • Not necessarily • Worms for ATM’s • Slammer, Nachi

15. Trojan • USC Trojans? • People from the affair of Helen of Troy? • NO • Program that enters a system disguised as something else • Never trust the gifts from Greeks (lesson learned from trojan war) • Trojan perhaps looks harmless • Or useful • Allow installation • Backdoors • Rootkits

16. Rootkits • Term derived from UNIX account ‘root’ • Patches on to host kernel libraries, routines • Place hooks on API’s, OS services, Routines, etc • A good rootkit cannot be detected • Does the statement sound too strong?

17. Shadow Walker • Designed to deceive in signature scanners • That is how Anti-Virus and most Rootkit detectors work • Hides its presence in the system • It hooks on to the page table entries & the page fault handler • It flushes the TLB • No page can be accessed bypassing the page fault handler initially

18. Shadow Walker …. • So how does that help • A scanner attempts to read a page • A fault is generated • This causes a fetch • The rootkit ensures that the scanenr never gets any access to infected pages

19. BluePill • Rootkit designed for Vista running on AMD pacifica technology • Has special mode for VM executions • Allocate memory for a process • More than required • What does this do? • Rootkit writes on the paged drivers • When the drivers are loaded back, you have infected drivers in memory • Allows Vista to be moved in guest environment • Rootkit becomes a hypervisor

20. Bluepill .. contd • So what happens due to that? • The Vista OS becomes the guest, and is completely under the control of the rootkit. Any scanner working from within the OS can never see the rootkit. • Why? • An OS process cannot have access to the layer below the OS • So if we placed something below the OS, the OS cannot find out about it.

21. Scenario 1 Application level malware – easy/slightly difficult to detect Applications kernel level malware – very difficult to detect OS Kernel Hardware

22. Scenario 2 Applications OS Kernel Not possible to detect from within the OS. Requires Hardware detection VMM layer malware Hardware

23. Solutions • VM based rootkit detectors • Hardware based rootkit detectors

24. VMM based detector • Type I VMM [XEN] • The VMM runs on top of the hardware. • Root of trust mechanism • VMM checks the privileged VM • The PVM checks the SM • SM checks the other VM’s

25. VMM • The VMM runs 1 Privileged VM(VM0), and many other guest VM’s • The VMM checks the VM0 over periods of time • Ensures the kernel of VM0 is not tampered with • VM0 runs the SM • It contains the integrity values of SM, to detect tampering

26. VMM detector - contd • The SM can access the states of all applications running on all the Guest VM’s • Guest VM’s run OS’s that run user applications • So what has this achieved? • Layered Software

27. The Trusted VM • What has to be done to penetrate the VMM layer • Attack the applications • Attack the guest OS • Attack the Guest VM • Finally attack the VMM • SM detects these before the final step

28. VMM layer • Is a micro kernel • What is a micro kernel • Answer: Best left to OS classes • Hence not a general purpose OS • Does not execute third party software • Due to this, it is secure • Too strong a statement? • Ok, has fewer vulnerabilities (due to less code) • Has fewer loopholes to exploit • Does not suffer from infected third party drivers

29. What does the VMM do? • Isolation between programs in an Operating System’s is a very difficult process • Many researches on it, fairly inconclusive • VMM provides isolation between the Guest VM’s • VMM also allows us to sandbox an OS and monitor it

30. VM0 • Monitors the SM • It can also allow and prevent other VM’s from accessing certain memory locations • It can protect sections in memory • It can prevent other VM’s from accessing some I/O devices • Why is this important?

31. SM • Checks the VM • Provides secure communication to User • Why is this important? • The SM has access to the state of registers, memory and instructions being executed by each Guest Vm • This helps to monitor the GVM’s

32. SM - contd • Checks the integrity values of Guest OS’s during boot • Allows detection of boot sector infections, rootkits • Can this help us detect VM based rootkits? • Checks kernel integrity, OS text section, interrupt vectors, etc

33. Last step • Can a rootkit impersonate a user • Yes, at least it will attempt to do so • So how can this be prevented? • The last module Secure I/O device • Do you see the answer to a question regarding I/O device access 3 slides back?

34. Secure I/O • Provides a trusted mode of communication between user and VMM • It should be a separate device • Why? Why cant it be a software channel

35. Why do we need secure I/O • Are human validations really true • What happens if this step is not followed • A viral program can trick the guest OS into sending a message that an update was performed • Allows changing of integrity values • The malware gets certified by the SM

36. Hardware detectors • Separate hardware device • Attached to the PCI slots • Can be attached in other places also • Some implementations involve placing a co-processor on the motherboard

37. Hardware detectors • This is also a root of trust device • The hardware device runs an OS • Its resources and state are not accessible by host CPU/HW • It is capable of accessing the host’s memory • It can halt a system if required

38. Heirachical checking • Each level stored the integrity values of the level above it • The SecCore contains the integrity values of certain critical sections of the kernel

39. SecCore • The critical sections of the kernel is responsible for checking the rest of the kernel • It is also responsible for checking the applications • The kernel is responsible for maintaining the integrity of the User level programs

40. Advantages • The Coprocessor does not have to attest the entire OS • Keeps load low • It stores information only about a small space • Memory requirements low • Most of the checking is offloaded to the Host CPU

41. Problems • Many integrity values reside inside the kernel • Can be infected • Solution? • Sign them • Digital Signatures

42. Thanks