Processing Intelligence Feeds with Open Source Software

1. Processing Intelligence Feeds with Open Source Software Chris Horsley, SC Leung, Tomas Lima, L. Aaron Kaplan, Raphael Vinot

2. Overview • Current topics in automatic incident handling for CERTs • IFAS • HKCERT , IFAS and use-cases • IHAP project • ContactDB project • Current R&D

3. IFAS • Information Feed Analysis System

4. Knowing what’s going on

5. How do national CSIRTs know what’s happening? • National CSIRTs need visibility on network in their economy • However, many national CSIRTs don’t operate networks themselves, and normally don’t have global (or any) direct visibility • How does the CSIRT know what’s going on in their country?

6. The kindness of strangers • Luckily, there are a lot of network operators, research teams, vendors, and other CSIRTs out there that collect information, and will share it with national CSIRTs. • And here comes the “but”...

7. So much data, so many formats • There are many feeds, all with their own data formats and mediums: • Formats: CSV, JSON, XML, STIX, IODEF • Mediums: HTML, RSS, email, HTTP APIs • While there are efforts to standardise data formats, this will take a long time, and will likely never cover 100% of feeds • We can’t change the format of remote feeds - we can only change what we do with the data.

8. The need for standards • Different feeds use many terms to mean the same thing: • ip, source_ip, src_ip, endpoint, attacker_ip, cnc_ip... • If we receive events from many feeds, we need to normalise so we can compare them together.

9. The need for storage • As a national CSIRT, we’re concerned with the health of national networks: which means measurement. • We can only measure longterm if we store events, enabling us to analyse them. • We also want to search through events, like: • C&C servers in domestic networks in last week • Bots infected with Trojan.abc on BigISP • Defaced web sites targeting gov.zz

10. Need for automation • There’s way too much network event data out there to manually process • Options: • a) use lots of analyst time doing tedious log processing • b) write lots of small, independent scripts • c) ignore inbound logs completely • d) use an automated processing system

11. So what do we need? • We need something which automatically: • Gathers many different types of feeds • Normalises the data in those feeds • Stores that data somewhere • Allows search and performs statistical analysis

12. IFAS • IFAS = Information Feed Analysis System • Project sponsored by HKCERT and developed by HKCERT and CSIRT Foundry • An integration of open source tools, released as open source for CSIRTs

13. Architecture

14. Architecture • Abusehelper: gather, process, and enrich feeds, generate events • Logstash: process and normalise feeds • Elasticsearch: store events in schema-free index server • Kibana: search through events • IFAS Reporter: get overall statistics, build realtime dashboards

16. Kibana event searches

17. Freeform statistical reporting

18. Nesting, filtering, deduplication

19. IFAS – Dashboard *Drill down right at the chart • Visualize information

20. What you need to start

21. Software • Open source under Apache 2.0 License • Only possible with the hard work released under open source licenses from Abusehelper and Elasticsearch teams • Contributions, bug reports, feature requests most welcome!

22. Hardware • Production: 8-16GB memory machine • Dev: 4GB possible • Multi-core machine (4+ ideal) • Runs in a VM no problem

23. Out of the box feeds Out of Box Feed Plugins (4 publicly available) • Abuse.ch • CleanMX • Millersmiles • Phishtank Other developed Plugins • Malc0de • Malicious Domain List • Arbor SRF • Shadowserver • Zone-H Future … more, and your own

24. Where to get it • Currently under closed pilot to trusted CSIRTs • Eventually public release • Please contact contact@ifas.io for details

25. Demos

26. IFAS and Use Cases SC Leung, HKCERT

27. Give a sense of Today’s Events

28. IFAS - Log Search • Powerful search on all the information collected Feed Details Keywords here Add columns of interests

29. IFAS - Reporter • Statistical analysis-Trends & Distributions • Free form statistical reports 3. 4. 5. 2. 1. 6.

30. Nesting, filtering, deduplication Number of phishings in “.AU” in each ASN by brand

31. IFAS - Alert • Set tracking criteria – get notify ASAP • domain: *.gov.hk • Alert lists : educational institutions (hkedu), NGOs (hkorg) !

32. Dashboard Real-time situational awareness for CERT management

33. Public Situational Awarenesson Compromised Servers / PCs

34. Hong Kong Security Watch Report

37. Analysis of Trend with Events • Correlate Cryptolocker 2013-Oct with Zeus

38. Engage ISPs for large scale incident handling ISP • Data do help HKCERT engaging ISPs (their sales team) • Data do help a server hosting SP understand their customers’ security problems

39. Converting security events into incident reports • Defacement • Phishing •  Export to CSV for batch processing, with some other scripts • Malware hosting – a bit difficult • Large volume of incidents – need prioritisation

40. Future of IFAS - a collaboration platform • All you can use • All you can contribute • Add input filters for new feeds • Add new plug-in modules • Add new chart and visualization • Integrate with other systems, e.g. RTIR • … • Standard language: STIX, taxonomy of ENISA

41. DSMS (Decision Support & Monitoring System) • An ongoing project that turn security events into Actionable Data • Set Priority, Choose Monitors, Consolidate Results Monitoring Services Profile Status Check (HTTP, DNS) via proxy Input  URL Tasks Status ? Interface Module Decision Support Sub-system (online /offline) Interfaces to Monitors Interface Module Public analysis sys (VirusTotal, ThreatExpert) IFAS Request to monitor Interface Modules Story Private analysis sys Interface Modules Web reputation (D-Shield) Incident Mgmt Output Consolidated Results

42. Incident handling automation project IHAP

43. IHAP • Very similar to IFAS, developed in parallel by CERT.pt, CERT.at • Also uses Logstash, Elastic Search and Abusehelper • Less work on the Webinterface, more work on Ontology, „Data harmonisation document“

44. IHAP - History • Discussions about CERT.AT developments/documents • Discussions about cooperation between CERTs • ENISA support

45. IHAP - Goals • Open Source • Maintainable • Flexible and Modular - must be possible to integrate existing software and modules (Pastemon, AbuseHelper, etc..) • Reusable • Easily Extendable - should require little knowledge and basic programming skills • Easily Deployable • Easily Updatable – easy to share new developments with other CERTs and update the system with that new code • Easily Configurable - config files that can be easily modified to fit CERT‘s needs • Documented - must be well documented

46. Links & Code http://www.enisa.europa.eu/activities/cert/support/incident-handling-automation

47. Common field names for AH • https://bitbucket.org/clarifiednetworks/abusehelper/wiki/Data%20Harmonization%20Ontology • A standard set of well defined field names within Abusehelper (AH) • Allows CERTs to: • Write bots which are interoperable within AH • Measure in identical ways • Easier to parse different feeds („generic santizer bot“) : you just have to define the mappings

48. contactDB

49. Background/ problem • abuse@ lookups suck (IRT object not in use, no standard; Just now RIPE DB is changing with abuse-c:) • Getting the right lookup is non-trivial, complex • Many (national) CERTs create their own abuse contact lookup DBs. • National CERT DB, TI directory, FIRST data can not be looked up automatically via scripts.

50. Idea • A caching contact database with more specific internal data • Some of this data (tel nos, etc) will never be in the public whois • Unify with TI, FIRST etc data • Make it query-able by scripts