1 / 25

ENAVis: Enterprise Network Activities Visualization

ENAVis: Enterprise Network Activities Visualization. Qi Liao, Andrew Blaich, Aaron Striegel, and Douglas Thain Department of Computer Science & Engineering University of Notre Dame. cse.nd.edu. Problem. Complex systems are hard to understand and visualize. Plenty of micro-level tools

teagan
Download Presentation

ENAVis: Enterprise Network Activities Visualization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ENAVis: Enterprise NetworkActivities Visualization Qi Liao, Andrew Blaich, Aaron Striegel, and Douglas Thain Department of Computer Science & Engineering University of Notre Dame cse.nd.edu

  2. Problem • Complex systems are hard to understand and visualize. • Plenty of micro-level tools • Host level (syslog, ssh log, etc) • Network level (MRTG, Netflow, etc) • Need macro-level picture of network • Not just in raw network connectivity • Need to know: • Who (users) are responsible • What (applications) are running on the network. 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  3. Context vs. Content • Packet content • (protocol:IP address:port number) • local context • (Network connection, user, application, arguments, file accesses) Who? What? NetFlow & sFlow Analyzer 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  4. ENAVis Host Users Applications 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  5. IP/Port ≠ User/App • Logging usually in form of • Network addressesUser identity • Port numbers Application identity • Network addresses and port numbersare NOTgood identifier for network activities • Two problems: • Lack of a mechanism to collect this missing Local Context. • Lack of a tool to correlate the huge amount of local contextdata. • Visually and interactively explore the data. • Visualization is the key. 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  6. Highlights • Local Context Data Collection • Light-weight, easy-to-deploy, monitoring agent • Scalable central data processing • Heterogeneous Graph • Hierarchical graph representation of data • HUA: Hosts, Users, Applications • Local-context aware connection chaining • Visualization • Statistics report and chart plotting • Visualize HUA graphs and perform queries • Interactive exploration 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  7. Data Collection • Need to know 4W for each network connection • who (users) • what (applications) • when (time) • where (hosts) • Proof of concept: • An easy to deploy and lightweight agent written in bash script • Only calls commonly available system tools • KISS • A hierarchy of local contextgathering • Tier one: netstat • Tier two: ps • Tier three: lsof (optional) 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  8. Built-in System Tools • netstat • Displays network connections and configurations • Whois for network connectivity • Proto, src/dst IP/port, State • -e UID, -p  PID • ps • Currently running processes. • PID  GID, PPID, argument list • lsof • All open files • Location (full path) of application, libs, files • diff • Difference of two consecutive outputs • > start of a new record • < end of an existing record 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  9. Directories/files structure days hosts 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  10. Agent Performance • 300+ machines on our campus since April 2007 • Over 400 GB data • Mix of CSE faculty / students, scientific grid, engineering lab. • Linux, Solaris, Mac OS X, (Windows) • CPU • <100 ms CPU time every 5 sec (2%) • Bandwidth • Total data size sent to the server: < 3 MB / day • 1000 hosts: 240 Kbps 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  11. Server Performance • Disk • Sun Fire X2100, AMD Opteron dual core (2.2GHz), 2 GB SDRAM, 1TB SATA disk. • 1000 hosts, window size = one year: 1 TB disk • 300 hosts, window size = past month: 30 GB • Processing Time • Up to 4500 hosts 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  12. HUA Graph Model • Heterogeneous graph • 4D space • Hosts, Users, Applications (HUA), Time • A meta-graph illustrating states Host-to-Host (similar to Netflow) Host-to-User Host-to-App User-to-User User-to-App Application-to-Application 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  13. Example: HU graph • A HUA graph • uses “User” nodes to glue “Host” and “Application” nodes • use “Application” nodes to glue local and remote parties. 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  14. Identities Linking • Perform connection chaining and bipartite matching. • Mapping src/dst identifiers in O(n) time. • Allow explicit identity linking between any pair of nodes. • User and Application identities is no longer inferred from host addresses or port numbers. 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  15. Implementation • Tool developed using Java, JFreeChart, and Prefuse. • Load n days’ connection data whose state = established. 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  16. Graph View Users Monitored hosts Graph controls hops Apps External Domains Node selection 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  17. Applications Time window Enterprise users Clusters/subdomains Local users Hosts 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  18. Applications User IDs Top Users User Info 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  19. Applications Top Apps Application Names 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  20. Applications Finance System Trusted Host Violation 2 Violation 1 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  21. Summary ENAVis approach Traditional approach • Centralized correlation and visualizationmake life easier for admins  • Augmented local-contextdata (Users and Applications), which are not available in existing schemes. 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  22. Conclusion • It is important to know who is responsible and what is running on an enterprise network. • Local-context (users and applications) is useful. • Network management, security policy auditing, fault localization, forensic, etc. • ENAVis: • Collects, fuses, and visualizes the missing local-context data. • Interesting HUA network connectivity graph. • Interactive exploration tool. • Future works • Windows agent as a service • Data mining modules built into the tool. 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  23. Acknowledgements • This work was supported in part by • the National Science Foundation (CNS-03-47392, CNS-05-49087), as well as • Sun Academic Excellence Grant (AEG) (EDUD-7824-080234-US). 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  24. Visit http://netscale.cse.nd.edu/Lockdown/ Thank You ! 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

  25. Demo 22nd Large Installation System Administration Conference (LISA '08), Nov. 9-14, 2008. San Diego, CA

More Related