A Policy Analysis of Phishing Countermeasures

1. A Policy Analysis of Phishing Countermeasures Committee Lorrie Cranor (chair) Alessandro Acquisti Jason Hong Adrian Perrig Steve Sheng Thesis Presentation Sept 8, 2009

3. Subject: eBay: Urgent Notification From Billing Department

4. We regret to inform you that you eBay account could be suspended if you don’t update your account information.

5. https://signin.ebay.com/ws/eBayISAPI.dll?SignIn&sid=verify&co_partnerid=2&sidteid=0https://signin.ebay.com/ws/eBayISAPI.dll?SignIn&sid=verify&co_partnerid=2&sidteid=0

6. This entire process known as phishing

7. Phishing a widespread problem • Estimated 0.4% of emails are phishing (Messagelabs 2009) • Average 27,000 unique phishing sites each month (APWG 2008) • Estimated $ 350m – $ 3.2b direct loss a year (Moore and Clayton 2007, Gartner 2007) • Attacks methods and targets evolving More profitable to phish than rob the bank!

8. Research Question As phishing continues to evolve, what can and should stakeholders do to fight it better?

9. Thesis Statement This thesis analyzes phishing stakeholders and their stakes, generates recommendations to fight phishing better through expert interviews. Case studies on the effectiveness of web browser anti-phishing toolbars and anti-phishing education inform this analysis.

10. Outline • Anti-Phishing expert interview study • Empirical study on browser phishing warnings • Anti-Phishing Phil • Phishing Susceptibility Study • Conclusions

11. Outline • Anti-Phishing expert interview study • Empirical study on browser phishing warnings • Anti-Phishing Phil • Phishing Susceptibility Study • Conclusions

12. Anti-Phishing expert interview study • 31 experts from academia, CERT, the Anti- Phishing Working Group (APWG), law enforcement, and key industry stakeholders • Examined the incentives of various stakeholders • Gathered information about the current and future state of phishing attacks • Ranked and prioritized countermeasures to be focused SHENG, S. , KUMARAGURU, P. , ACQUI STI , A. , CRANOR, L. , AND HONG, J . Improving phishing countermeasures: An analysis of expert interviews. In eCrime Researchers Summit 2009 (Tacoma, WA, October 20 – 21 2009).

13. Experts

14. Methodology • Interview Protocol • 60 minute in-person or phone interviews • a set of open-ended questions about how phishing impacts their organizations, amount of losses, current and future state of phishing, and the effectiveness of different countermeasures • comment and prioritize on a set of 31 recommendations • Analysis • Full interviews transcribed • Themes labeled and clustered

15. Overview of Findings • Evolving threat • Stakeholder incentives • What stakeholders should do • Issues of false positives • Law enforcement • Education

16. Evolving threats • Phishing is evolving to be more organized • Majority of phishing messages suspected to come from organized phishing gangs (Moore and Clayton 09) • Phishing is becoming more targeted and evolving • Phishing and malware increasingly blended together

17. “You see social engineering aspects of malware and high automation aspects of phishing. At some point, it might be hard to tell them apart . . .To the attackers, it doesn’t matter what they use. They know social engineering has an effect on the end user, they know script and code and have some effect on the user’s machine. It is just a matter of putting what they know and what they have.” from an academic expert

18. Misalignment of incentives • Consumer made liable for phishing in some countries • ISPs have little incentives to clean up compromised machines • A major US ISP has 10% computers infected with malware

19. Recommendations • Financial institutions should produce and report accurate estimates of phishing losses • A major US bank lost $4.5 million to phishing and malware in 12 months • without reliable statistics, difficult for corporations to manage the risk and for law enforcement to prioritize cases • Financial institutions lack incentives to report • Key: collaborate with law enforcements to gather statistics and regulators should require mandatory reporting

20. Recommendations • Fixing compromised machines • Recommend the US government institute a notice and takedown regime for bot C&C • Research on automatically cleaning compromised machines needed

21. What stakeholders should do • OS vendors are doing a lot of good work, but web applications insecurity is a major hurdle • Browsers are at a strategic place to warn users • Highly concentrated market (NetApps 2009) • Effective warning (Egelman et al 2008) • ISPs in the best position to clean up compromised hosts

22. Recommendations • Web browser continue to improve the performance of integrated browser anti-phishing warning systems in order to catch 85-95% of phishing URLs within an hour after they go online • Key: more diverse feed and use heuristics to complement blacklist

23. Law enforcement • Experts agreed law enforcement is crucial and needs to be emphasized • Challenges: • lack of necessary analytical capabilities in determining investigative priorities • International nature of the crime • Sophistication of criminals to hide traces

24. “ We get a lot of information in, but we are overloaded. People can share data now, that’s occurring, but what’s not happening is the analysis piece. We have limited resources . . . We do it manually. We need resources, software and hardware to enable that, also more bodies looking at it. There is no magic about the data, but the magic is in the analysis. . . taking institutional knowledge and applying some data mining algorithms.” A law enforcement expert

25. Recommendations • Law enforcement • Recommend US government invest in tools for better case management and better digital evidence processing • Expand scholarship programs to recruit graduates in computer science

26. Education • Education and awareness are important • Experts disagree on the effectiveness of education

27. “There needs to be some accountability on Internet users …. People still click on URLs they shouldn’t. So we need to stress user education, and a little bit of common sense.” From an industry expert

28. “My experience of education is that it won’t make that much difference. You have to do it, because if you don’t, consumers will get mad at you. . . . . However, education doesn’t impact phishing losses, or make it less. It doesn’t do any of that, what it does is making people feel safer.” From an industry expert

29. Related work • Technical phishing analysis • FSTC Phishing white paper 2005 • Identity theft Technology Council report 2005 • DHS and APWG report 2006 • Best Practices (APWG 2006 2008 2009) • Economics of Security • Security Economics and the Internal Market (Anderson et al 2008) • Economics of malware (OECD 2008)

30. Outline • Anti-Phishing expert interview study • Empirical study on browser phishing warnings • Anti-Phishing Phil • Phishing Susceptibility Study • Conclusions

31. Empirical study on browser phishing warning • Browser is a strategic place to protect users, and major browsers provide phishing warnings based on blacklists and heuristics • What is the accuracy and update speed of blacklists? • What is the accuracy of heuristics? • How to improve protection? SHENG, S., WARDMAN, B., WARNER, G., CRANOR, L., HONG, J., AND ZHANG, C. An empirical analysis of phishing blacklists. In 6th Conference in Email and Anti-Spam (Mountain view, CA, July 16 - 17 2009).

32. Methodology • Used an automated test bed with fresh phish to evaluate 8 popular phishing toolbars • Conducted two tests in October and December 2008 • Tested 191 fresh phish and 13,458 HAM URLs • For each phish tested 9 times in 48 hours for blacklist update

33. Automated Testbed Y. Zhang, S. Egelman, L. Cranor, and J. Hong Phinding Phish: Evaluating Anti-Phishing Tools. In Proceedings of the 14th Annual Network & Distributed System Security Symposium (NDSS 2007), San Diego, CA, 28th February - 2nd March, 2007.

35. Summary of results • 70% of phishing campaigns in our dataset lasted less than one day • Zero hour protection for blacklists is between 15- 40% • Most tools detect > 60% after 5 hours • Tools that use heuristics caught significantly more phish • False positive rates are very low for both blacklists and heuristics

36. Blacklist performance - October

37. Heuristics

38. Recommendations • Clarify the legal issues surrounding false positives of heuristics • Focus heuristics research on reducing false positives • Leverage heuristics to improve blacklist

39. Leverage heuristics to improve blacklist

40. Related work • Phishing toolbar test • Zhang et al (2006) • Ludl et al (2006) • HP study (2006) • Effectiveness of spam blacklists • J. Jung and E. Sit (2006) • A. Ramachandran and N. Feamster (2006)

41. Outline • Anti-Phishing expert interview study • Empirical study on browser phishing warnings • Anti-Phishing Phil • Phishing Susceptibility Study • Conclusions

42. Design Motivations • Security is secondary task • Learning by doing • Fun and engaging • Better strategies

43. Anti-Phishing Phil • Online game • Teaches people how to protect themselves from phishing attacks • identify phishing URLs • use web browser cues • find legitimate sites with search engines SHENG, S., MAGNIEN, B., KUMARAGURU, P., ACQUISTI, A., CRANOR, L. F., HONG, J., AND NUNGE, E. Anti-phishing phil: the design and evaluation of a game that teaches people not to fall for phish. In SOUPS ’07: Proceedings of the 3rd symposium on Usable privacy and security (New York, NY, USA, 2007)

44. Design Methodology • Design Principles • Reflection principle, story-based agent environment principle, conceptual and procedural knowledge principle • Multiple iterations • Early iterations made use of paper and Flash prototypes • Play-testing and feedback from research group • Developed a working prototype that we tested with actual users • Iterated on the design several more times based on user feedback and behavior • Polish with attractive images and enticing sounds

45. Anti-Phishing Phil