Honeypots as a Tool to Improve Incident Response Readiness at USP

1. Honeypots as a Tool to Improve Incident Response Readiness at USP Alberto Camilli Isabel Chagas Centro de Computação Eletrônica Universidade de São Paulo Educause Security Professionals Conference 2007 Denver 12 April 2007

2. Agenda • University of São Paulo, numbers and IT organization • USP and the national Honeynet project • USP honeypot-based Early Notification procedure and results • Q&A

3. Presentation Objectives • Show how honeypots take part in the incident notification process, how they are configured and managed at USP. • Main incident statistics for USP. • Show what changes in the management of honeypots at USP lead to a change in the campus profile of incidents, with reduction in quantity and in the solution times.

4. University of São PauloResearch Extensive • State University (São Paulo) • 185 in ISI rank: • 31.548 indexed papers • 185.110 citations • 2.270 PhD, 3.218 MSc • 48.530 Undergraduate students • 4.884 faculty • 14.952 staff • 32.059 graduate students • 8 main campuses, 60% based in São Paulo (city) • 85 “federation-like” Units (Teaching, Services and Extension) • Annual budget near US$ 700 M

5. Centro de Computação Eletrônica da USP • Main USP Computer Center • NOC and CSIRT 24x7 operation • 400m2 data center • HPC main facilities for the University (e.g. computer 363o. in Top500 list)

6. USP Network Connections Internal GEANT ( EUROPE / SPAIN ) MG External RJ SC Internet Commodity USA and Brazil POP Clara ( Global Crossing ) RS Internet 2 USA POP RNP ( USP ) ANSP SP Internet USP Enclave Kyatera Test Bed Registro BR METRO IX ( Non Commercial ) TV GIGA Test Bed USP Ribeirão Preto • USPnet 600Mbps I/O Internet commodity traffic 1,000 Buildings; 50,000 network points 25.000 (80.000) e-mail accounts (@usp.br) 850,000 e-mail/day (20% valid) USPnet Telefonica Metro SP Medical Schools USP Bauru Hospitals USP São Carlos USP ( Cidade Universitária ) USP Pirassununga USP Piracicaba

7. USP IT organization 143.107.xxx.xxx /16 200.136.xxx.xxx /20 200.144.xxx.xxx /20 Adm. Dean USP business rules Internal Units Administration CIO IT services and infrastructure campus 1 Business IT CSIRT USP Internal-External Campus Notifications ID_#inc Local CSIRT1 Campus Computer Centers External CSIRTs Notifications campus n Local CSIRT2 Internal Campus Notifications ID_#inc ID_#inc

8. USP and the Brazilian Honeypot Alliance USP Ribeirão Preto USP Piracicaba USP São Carlos USP Cidade Universitária Coordination by CERT.br: http:///www.honeypots-alliance.org.br/

9. The ProjectBrazilian Honeypots AllianceDistributed Honeypots Project • Coordination: CERT.br and CenPRA Research Center • Use of low interation honeypots • Based on voluntary work of research partners • 37 research partner instituitions • Industry, telcos, academic (USP and others), government and military networks • Each partner provides • Hardware and network • Honeypot(s) maintenance

10. USP Motivation • To increase USP’s capacity of: • Incidents identification and knowledge • Incident detection • Event correlation with other Entities • Trend analysis • Which Units are more vulnerable? • Very Useful for Incident Response • Sensors distributed in several campuses

11. USP Honeypots Location

12. Brazilian Honeypot Alliance Architecture Netblock range from /28 to /24

13. Data Usage • Incident response (CERT.br): • Identify well known malicious/abuse activities • Worms, bots, scans, spams and malwares in general • Notify the Brazilian networks´contacts • Including recovery tips • Partners: • Observe trends and scans for new vulnerabilities • Detect promptly: • Outbreaks of new worms/bots • Compromised servers • Network configuration errors • What about USP usage?

14. Honeypots project: how good it was (july06) after 3 years? 3 years Gigabit backbone July 06 Honeypot (CERT.br) But ... how better can it become?

15. A closer look at our honeypot data... 1.Threats from the outside Different External IPs Port 80 Protect applications! Protect backbone routers!

16. A closer look at our honeypot data ... 2.Where and how the internal network is being attacked same subnet type of attack, e.g.: 135(tcp), 445(tcp) Protect Windows desktop! Protect subnet!

17. Worm Propagation Times 300 min = 5 hs Typical log for a honeypot at USP Worm propagation Model [1] ~ 10 hs then ... early notification? [1] Zou C., Gao L., Gong W. Monitoring and Early Warning for Internet Worms. CCS’03

18. Timeline logic in Early Notification Tni Tca Tp1 Tne Notification CSIRT-USP Notification Cert.br Correction Action Propagation Time CERT.br CSIRT Handling Local CSIRT Handling CSIRT • Contamination (int) • Improper Action (ext) End of Incident Tn = max (Tne,Tni) Tp2 Tca Time Notification CERT.br OR Notification CSIRT-USP Tp2 = Tp1 – some measured average Tca ??

19. Early Notification (EN) procedure • Hypothesis • Unnoticed attacks should now begin to be identified. • CSIRT-USP is able to notify attacks in advance. • Units will be able to react accordingly to block these attacks. • What we did? • Notifiy the victims as soon as an internal attack is being observed • No further considerations about the nature of the attack. • Why? • We want better incident scores and honeypot logs are at our disposal. • How and when? • A daily script generates a summary of the attacks. Each attacked Unit receives the summary notification from CSIRT-USP, as a new security incident ticket.

20. Internal notifications message format • CSIRT USP messages (daily summary): • Subject: [Honeypot] Máquina(s) suspeita(s) (20070309) • Content: 143.107.zzz.yyy : 139(247) 143.107.nnn.mmm : 135(202) 137(573) 139(3041) 1433(183) 445(1302) 80(568) • Cert.br messages (on IP basis): • Subject:143.107.zzz.yyy: host(s) infectado(s) com Agobot/Phatbot • Content: Apr 27 13:30:15.918576 143.107.zzz.yyy.3683 > xxx.xxx.xxx.22.139: S [tcp sum ok] (src OS: Windows XP SP1, Windows 2000 SP4) 904637194:904637194(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) (ttl 123, id 20447, len 48) number of attempts

21. Internal notifications results after 6 months of EN adoption Antecipated 6hs by CSIRT-USP Same data, Different analysis criteria,Different Interpretations

22. Overview of EN results CSIRT USP/Cert.BR internal notifications 2002-july 2006: internal notifications only from Cert.br Notifications Increase

23. Classification of Incidents at USP

24. Top 10 Units solution time(before and after EN) 2006 Period Unit ID

25. Top 10 Units incidents (before and after EN) 2006 Period Unit ID now, a closer look to the profiles of incidents ...

26. Incident Profile ID_146 Before EN (69): 16% 12% 16% 49% After EN (77): Tca ~ 20+ day F(t)=1-eλt 25% 65% Tca ~12 day

27. Incident profile ID_112 Before EN (36): January-July 2006 After EN (77): July-December 2006 6% 14% 8% 39% 28% 86% 19% External (Spam, Open-Proxy, Other) “vanished”

28. EN limitations CSIRT USP Notification rate Local team internal Overloaded Capacity (λ´) Spare Capacity (λ) external internal F(t)=1-eλt internal internal external external internal internal Incident solution rate Feedback to CSIRT • Local Responses are limited by local Capacities • Capacity  (skills, technology, staff/bw, staff/computers, ...) • Local Capacities are related to the Local Incident Profiles (symptom)

29. Other (very) interesting profiles ... Linux and good firewall management • Minimum contamination by worms • Little interaction to CSIRT-USP, no influence from notification process. Similar Units profile (BW utilization, Staff, Technology, ....) none from CSIRT USP !

30. Incident Response Readiness at USP • Early notification is essentially a CSIRT procedurethat relys on: • Honeypots, for the localization and identification of the problem • Available local internal capacities, for problem solving • Long term Incident reduction and better responses can be achieved with: • Education • Specializing local CSIRT managers • Training of local teams, to improve correction actions • General User Education (especially on Windows):diversified public: students, professors, administration • Preventive actions, to keep volume of internal notifications under manageble limits • Anti-virus distribution • Bandwith control • Network access control • Institutional network scanning • Other Specific tools on-going under study

31. Summary

32. Conclusions • End-user’s freedom is normally obtained with some degree of computers contamination. • Honeypot is an effective way to detect early stages of contamination and to support the development of actions against later stages of the worm’s cycles. • Honeypot monitoration is centralized and demands minumum infrastructure support • Honeypots permit suggest local actions according to Unit’s profiles • Gobal worm mitigation doesn’t necessarily mean local worm mitigation. • Honeypot-based Early Notifications by CSIRT-USP changed the profile of security incidents at USP • Incidents are closed in shorter times • External incidents has been reduced

33. Special Thanks CISRT USP TEAM • Marta Bazzo Cilento • Hamilton Jun Higashizono • André Gerhard • Rogério Herrera Mendonça • Luis Ferreira • Bruno Darigo • Fernando Fugita • Solange Vieira • Olavo Rodrigues

34. References • Brazilian Honeypots Alliance – Distributed Honeypots Project http://www.honeypots-alliance.org.br • CCE http://www.usp.br/cce • CERT.br http://www.cert.br • Honeyd http://www.honeyd.org/ Several papers about the project http://www.honeynet.org.br/papers/ • USP http://www.usp.br • Other Zou C., Gao L., Gong W.;Monitoring and Early Warning for Internet Worms. CCS’03 Dagon D., Zou C., Lee W.; Modeling BotNet Propagation Using Time Zones. NDSS’06 [1]

35. Q&A Thank you!