1 / 13

Storage Architectures

Network Monitoring Data Storage and Processing. Storage Architectures. Introduction.

meara
Download Presentation

Storage Architectures

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Network Monitoring Data Storage and Processing Storage Architectures

  2. Introduction • Given a starting pool of data objects (e.g. raw packet files, database table of meta-data) various process can be applied to create new derived objects (either at capture time, as a batch process, or on demand). A hierarchy of processes can be applied as an automated workflow when an object (e.g. graph of throughput) is requested.

  3. Outline • Conceptual Model • Data Types and Objects • Transformations • Dependency Hierarchies and Workflows • Example • Implementation Considerations • Flow Throughput

  4. Data Types • Basic data types are those imported into the system e.g. packet traces, capture meta-data • Derived data types are those created from basic data types e.g. flow packet trace, netflow record • Some form of object-oriented hierarchy of types e.g. flow packet sequence is a packet sequence

  5. Transformations • Objects from the data store can be used to create new objects • Multiple objects can be used to create a new object • Multiple objects can be created by one transformation • Transformations can use objects inheriting the required data type e.g. flow packet sequence when packet sequence needed

  6. Transformation Parameters • Parameters for transformations are also objects • These parameter objects can be used for indexing, searching or as meta-data for the created object

  7. Dependence Hierarchies • Can request an object and if it is not in the data store then it can be created through a chain of transformations of data objects • The intermediate objects created will then be in the store, available for repeated queries or similar or overlapping queries

  8. Implementation Considerations • Performance • the transformation functions will have to be efficiently written as many TB of data are being dealt with • pre-emptive generation of objects when the object is used for other transformation • creating of common data in advance • Garbage collection due to storage limitations

  9. Example Implementation • Flow throughput line graph • Netflow records are created from pcap files for the desired time window • User selects desired flows and bin size • Throughput calculated from pcap files • Graph created from throughput data • Implemented as a Perl/CGI web app • Results cached for multiple uses

  10. Related Work • Bashir, O., Phillips, I.W., Parish, D.J., Adams, J.L. & Spencer, T. (1998), The Management and Processing of Network Performance Information, BT Technology Journal , 16(4) , October 1998, pp 203-212, ISSN: 1358-3948. • Mike Fisk, George Varghese (2002), Agile and Scalable Analysis of Network Events, IMW 2002, pp 285-290. • Root: An Object-Oriented Data Analysis Framework, CERN, http://root.cern.ch/

More Related