1 / 35

Péter Hága Eötvös Loránd University

Building the Measurement Infrastructure and Virtual Data Observatory in the European Future Internet Experimental Facility. Péter Hága Eötvös Loránd University. w hy to build a new experimental facility?.

akiva
Download Presentation

Péter Hága Eötvös Loránd University

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Building the Measurement Infrastructure and Virtual Data Observatory in the European Future Internet Experimental Facility Péter Hága Eötvös Loránd University

  2. why to build a new experimental facility? • in the Internet it is often not possible to measure traffic flows and other aspects of usage • models and analysis of measurement data should playcrucial role in the research on a Future Internet • the design of new network architectures should be amenable to modeling and measurement the network itself (in contrast of the today’s Internet) • the Experimental Facility should provide a platform with a rich set of tools for measurement and monitoring • the EF should offer full observability of the experiment and the related data • theoreticalanalysis and modeling can benefit from the rich capture and logging of data FuturICT 2009, Budapest, Hungary, 29 June 2009.

  3. FI research initiatives GENI infrastructure • GENI/FIND/NetSE, US • AKARI Project, Japan • Future Internet Forum, Korea • ANR, France • G-Lab Initiative, Germany • SHOK, Fin • Internet del Futuro, Spain • EU Future Internet Research and Experimentation (FIRE) FuturICT 2009, Budapest, Hungary, 29 June 2009.

  4. etomic history • The European Traffic Observatory Measurement InfrastruCture (etomic) was created in 2004within the Evergrow Integrated Project. • Since 2005 also supported by the Hungarian Office for Research and Technology • Its goals: • to provide open access, public testbed for researchers experimenting the Internet • to serve as a Virtual Observatory active measurement data on the European part of the Internet • Since 2008 in Onelab2 Etomic is federated in the Advanced Network Monitoring Equipment Best Testbed Award FuturICT 2009, Budapest, Hungary, 29 June 2009.

  5. The GENI connection FuturICT 2009, Budapest, Hungary, 29 June 2009.

  6. Best Testbed Award Infrastructure stations

  7. Future Internet Research and Experimentation (FIRE) programme • OneLab2: An Open Federated Laboratory Supporting Network • OneLab2 EU Integrated Project 2008-2010 • 29 participants Europe wide + 1 from Japan • budget 7.5 M€ • aims to federate several measurement infrastructures (including Planetlab Europe) • includes building (extending) and maintaining a monitoring and measurement infrastructure (based on the Etomic) • partners involved: • Université Pierre et Marie Curie, France • Universidad Autonoma de Madrid, Spain • Tel Aviv University, Israel • Eötvös Loránd University, Hungary FuturICT 2009, Budapest, Hungary, 29 June 2009.

  8. Advanced Network Monitoring Equipment • Web interface available via www.etomic.org • Account application -> own measurement design • Free access to periodic measurement end-to-end data • Measurement time slot reservation for registered users, unique slot (experiment are not affected by other users) • Programming DAG and ARGOS cards via a user friendly API • Controlling APE box • Fully configurable active measurement scenarios • Measurements are distributed automatically to the measurement stations • Measurement data is stored in a VO fashion FuturICT 2009, Budapest, Hungary, 29 June 2009.

  9. Advanced Network Monitoring Equipment (ANME) Hardware setup FuturICT 2009, Budapest, Hungary, 29 June 2009.

  10. Main components FuturICT 2009, Budapest, Hungary, 29 June 2009.

  11. ANME hardware components • Etomic • Precise active measurements with DAG 3.6GE and ARGOS FPGA • CoMo • Monitoring the traffic of Planetlab nodes • APE • lightweight measurement box • standalone • GPS receiver to provide time synchronization FuturICT 2009, Budapest, Hungary, 29 June 2009.

  12. Etomic/CoMo architecture • Server PC architecture • Linux OS • Endace DAG 3.6 GE card or • ARGOS FPGA measurement card • with packet sending capability (packetoffset ~60ns) • GPS antenna for time synchronization FuturICT 2009, Budapest, Hungary, 29 June 2009.

  13. APE lightweight measurement box • standalone • based on Blackfin programmable board • dedicated IP packet timestamping module (< μsec)‏ • low cost (300 €) FuturICT 2009, Budapest, Hungary, 29 June 2009.

  14. etomic Central Management System Infrastructure management FuturICT 2009, Budapest, Hungary, 29 June 2009.

  15. Central Management System • Etomic CMS • IBM blade server • User management • Node maintenance • Experiment scheduling • Storing experiment results (temporally) • Web GUI FuturICT 2009, Budapest, Hungary, 29 June 2009.

  16. Slices vs. unique timeslots no virtualization • balancing sliceability and fidelity is one of the most fundamental challenges facing the EF • virtualization allows many researchers to share a common set of resources • while virtualization introduces too much unpredictability in timing measurements • dedicated measurement hardware elements can be allocated to some slices. Measurement hardware should operate under temporal partitioning principle.

  17. realization FuturICT 2009, Budapest, Hungary, 29 June 2009.

  18. Some ongoing basic topics Experimental use cases FuturICT 2009, Budapest, Hungary, 29 June 2009.

  19. Use cases • one way delay (60nanosec resolution) • tracking topology changes • available bandwidth meter • transport protocol testing • queuing delay tomography • geolocation experiments • … FuturICT 2009, Budapest, Hungary, 29 June 2009.

  20. Store & share Data handling FuturICT 2009, Budapest, Hungary, 29 June 2009.

  21. Experimental facilities • most network measurement projects: • use a single dedicated infrastructure • scan only narrow subsegments • analyze a limited set of networkcharacteristics • centralized and separated fromeach other • key idea: try to interconnectseparate measurement data! • large-scale behaviour • long-term evolution

  22. Traditional approach Traditionally measurements are designed to collect only specific data, important from the point of view of the researcher’sagenda. FuturICT 2009, Budapest, Hungary, 29 June 2009.

  23. Sharing science Genomedatabases Astronomy

  24. Related work: CAIDA/DatCat

  25. Related work: MoMe database

  26. Related work: MAWI repository

  27. Data publication efforts raw data isnot stored raw data from single infrastructure • DatCat (USA): • searchable catalog of metadata about measurements • passive traffic traces, traceroutes, BGP tables, virus propagation studies • MOME (EU): • database for meta-measurement data • packet & flow traces, routing data, HTTP traces • standardization efforts • sharing of analysis tools is possible (e.g. jitter calculation) • MAWI (Japan): • repository of passive traces from the WIDE backbone (collected since 1999)

  28. FTP and GREP are not adequate • FTP ~ 1 MB/sec • FTP ~ 1 GB/min (= 1 $/GB) • FTP ~ 1 TB/ days and 1K$ • FTP ~ 1 PB/ 3 years and 1M$ • you can: GREP 1 MB in ~ seconds GREP 1 GB in ~ minutes GREP 1 TB in ~ days GREP 1 PB in ~ years • for 1 PB you need ~ 1000 disks • at some point you need • to limit search • parallel data search and analysis • move the analysis to the data storage • this is where databases can help

  29. Key ideas in data handling • store & share raw data • joint analysis of different types of measurement data • reanalysis (with new evaluation methods) • reference data (historical comparison) • share analysis tools • server side processing simplifies client applications • no need to transfer bulk data packages: online processing • standardization, network XML Network Measurement Virtual Observatory (nmVO) FuturICT 2009, Budapest, Hungary, 29 June 2009.

  30. VO approach • The modern approach is to collect and store all measurable data and make it available for „virtual observation”. Virtual measurements can have set of goals different from the original FuturICT 2009, Budapest, Hungary, 29 June 2009.

  31. Unified interface • VO can be realized by collecting measurement data from different infrastructures. Data structures should be standardized → netXML FuturICT 2009, Budapest, Hungary, 29 June 2009.

  32. Casjobs - database interface • thenmVOconcept: efficientwaytoorganize and sharedifferenttypes of measurementdata and correspondinganalysistools • prototypenodeimplementation • clientapplicationexample • a semanticextensiontointegratemultipledatasources

  33. Summary • Experimental facility: • „old” etomic is running • „new” etomic will be launched this summer • Main features: • Precise active measurements • Unique time slot reservation • Easy to use GUI • Data is collected in VO fashion • European deployment (planned): FuturICT 2009, Budapest, Hungary, 29 June 2009.

  34. FuturICT 2009, Budapest, Hungary, 29 June 2009.

  35. Thanks and register! Visit: www.etomic.org E-mail: haga@etomic.org

More Related