1 / 32

DuDE : A D istributed Computing System u sing a D ecentralized P2P E nvironment

DuDE : A D istributed Computing System u sing a D ecentralized P2P E nvironment. The 4th International Workshop on Architectures, Services and Applications for the Next Generation Internet (WASA-NGI-IV) Bonn , Germany , October 4th, 2011. J. Skodzik , P. Danielis, V. Altmann,

shanna
Download Presentation

DuDE : A D istributed Computing System u sing a D ecentralized P2P E nvironment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DuDE: A Distributed Computing System using aDecentralized P2P Environment The 4th International Workshop on Architectures, Services and Applications for the Next Generation Internet (WASA-NGI-IV) Bonn, Germany, October 4th, 2011 J. Skodzik, P. Danielis, V. Altmann, J. Rohrbeck, D. Timmermann • University of Rostock, Germany • Institute of Applied Microelectronics • and Computer Engineering T. Bahls, D. Duchow • Nokia Siemens Networks • Broadband Access Division • Greifswald, Germany

  2. Outline • Introduction & Motivation • DuDE in General • The DuDE Algorithm in Detail • Test Scenario and Evaluation • Summary andFuture Work

  3. Situation today Does an ANhaveenoughresources? Doesitprovidesufficientstatisticsat all? • Increasingnumberof Internet usersandtraffic data • Internet Service Providers (ISPs) wanttoensure: • Quality of Service (QoS) • The detectionofbottlenecks • The detectionofattacks • Howtoensuretheseissues?  Statisticsgeneratedfromexisting log data

  4. Introduction & Motivation Resources utilization CPU MEM 60% 60%

  5. Introduction & Motivation Supported Not supported • Processorutilization • RAM utilization • Drops • Numberofpackets Short term statistics (STS)for singleANs Long termstatistics (LTS) Creationof global statistics Simultaneous computationof multiple statistics Processing ofincreasinglog datavolumes Simple supportofnewstatisticstypes

  6. Introduction & Motivation One AN does not haveenoughhardwareressources  Usageof multiple ANs tocomputestatistics Efficientresourcesharingwith high resilienceandscalability  Utilizationof P2P technology DuDE: Exploitation of already available resources  No extra costsfor additional equipment

  7. DuDE in General Node2 ID Node3 ID Node1 ID Node4 ID Logical P2P ring

  8. DuDE in General Log data (somehundredsof KBs) Data chunk (ca. 100 KBs) 8

  9. DuDE in General • Objective: High log dataavailability = 99.999 % • Simple replicationwastesmemoryressources  Reed-Solomon Codes • Split log dataofeach AN into m datachunks

  10. DuDE in General • Objective: High log dataavailability = 99.999 % • Simple replicationwastesmemoryressources  Reed-Solomon Codes • Split log dataofeach AN into m datachunks • Encoding: Add k interleavedcodingchunks n=m+kchunks

  11. DuDE in General • Objective: High log dataavailability = 99.999 % • Simple replicationwastesmemoryressources  Reed-Solomon Codes • Split log dataofeach AN into m datachunks • Encoding: Add k interleavedcodingchunks n=m+kchunks • Decoding: Restore log datafromany m of n chunks

  12. DuDE in General Log data (somehundredsof KBs) Data chunk (ca. 100 KBs) Howtoapplyourapplicationto P2P?

  13. DuDE in General Whichstepsarenecessarytocomputestatistics? Task Job Task Task Admin. Job = collectionof STS and/or LTS tasks Task = partofjob, e.g., requestfor „CPU“ statistics Jobscheduler (JS): Receptionandmonitoringofjob Taskwatcher (TW): Receptionandprocessingoftask

  14. The DuDE Algorithm in Detail Stage 1: Resourcecollection 10% 60% 50% 30% 60% 50% 30% 10% Admin. …Job …Global statistics …Task …Log data

  15. The DuDE Algorithm in Detail Stage 2: Jobscheduler determination 1. Resource collection Admin. …Job …Global statistics …Task …Log data

  16. The DuDE Algorithm in Detail Stage 3: Resourcere-collection 2. Jobscheduler determination 1. Resource collection Admin. …Job …Global statistics …Task …Log data

  17. The DuDE Algorithm in Detail Stage 4: Task assignment 3. Resource re-collection 2. Jobscheduler determination 1. Resource collection Request forProcessorutilization STS Request for RAM utilization LTS Request for Drops LTS Admin. …Job …Global statistics …Task …Log data

  18. The DuDE Algorithm in Detail Stage 4: Task assignment 3. Resource re-collection 2. Jobscheduler determination 1. Resource collection Howto find all log datafor global statisticscomputation? Admin. …Job …Global statistics …Task …Log data

  19. The DuDE Algorithm in Detail Global Peer Data Discovery Algorithm - Thresholdvalue A = 2 Node2 ID Node1 ID 1 yes 0 no Node1 Taskwatcher 2 yes 0 no Node2 3 yes 0 no Node3 4 no 1 no Node4 5 yes 0 no Node5 5 no 2 yes Node5 Algorithmisdone 6 no 1 no Node6 7 yes 2 yes Node7 Node3 ID Node5 Algorithmisdone Request for global statistics All dataneeded

  20. The DuDE Algorithm in Detail Stage 5: Log datacollection 4. Task assignment 3. Resource re-collection 2. Jobscheduler determination 1. Resource collection Admin. …Job …Global statistics …Task …Log data

  21. The DuDE Algorithm in Detail Stage 6: Statisticscomputation 4. Task assignment 3. Resource re-collection 2. Jobscheduler determination 1. Resource collection 5. Log data collection Processorutilization stat. RAM utilization stat. Drops stat. Admin. …Job …Global statistics …Task …Log data

  22. The DuDE Algorithm in Detail Stage 7: Send resultsanddisplaythem 4. Task assignment 6. Statistics computation 3. Resource re-collection 2. Jobscheduler determination 1. Resource collection 5. Log data collection Admin. Admin. …Job …Global statistics …Task …Log data

  23. The DuDE Algorithm in Detail Stage 7: Send resultsanddisplaythem 6. Compute statistics 4. Assigntasks 3. Resource recollection 2. Determinejob scheduler 1. Resource collection 5. Restore log data Admin. Admin. …Job …Global statistics …Task …Log data

  24. Test Scenario and Evaluation PC Configuration Pentium 4 (1.5 GHz) 512 MB RAM  Equivalent to AN HW

  25. Test Scenario and Evaluation • Parameters: • Numberoftasksinsidejob • Numberoflog datasets in the P2P network • Computational loadforstatisticscomputation • Measurements: • Time forfinishing a job • Memory utilization

  26. Test Scenario and Evaluation Linear IncreaseofNeeded Time Time is Constant 

  27. Test Scenario and Evaluation Linear IncreaseofNeeded Time Time isConstant 

  28. Test Scenario and Evaluation Linear Increaseof Memory Utilization Constant Memory Utilization

  29. Test Scenario and Evaluation Memory utilizationincreasesmoreatthesingle AN thanatthetaskwatcher

  30. Test Scenario and Evaluation Memory utilizationisconstantandindependentofthe computational load

  31. Summary and Future work • P2P-based systemfordistributedcomputingof statistics • STS and LTS • Statisticsfor a single AN andthewholenetwork • Global Peer Data Discovery Algorithm • Successfullydeveloped prototype (demosession) • Investigation offurtherusecases

  32. Thanks for your attention! Questions?

More Related