1 / 32

Accelerating Distributed Machine Learning by Smart Parameter Server

Explore how Smart Parameter Server improves distributed machine learning efficiency through selective updates, proactive pushes, and intelligent strategies. Experiment results show significant reductions in training time compared to baseline methods.

lapp
Download Presentation

Accelerating Distributed Machine Learning by Smart Parameter Server

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AcceleratingDistributedMachineLearningbySmartParameterServer JinkunGeng, Dan Li and Shuai Wang

  2. Background • Distributedmachinelearningbecomesthecommonpractice,becauseof: • 1.Theexplosivegrowthofdatasize

  3. Background • Distributedmachinelearningbecomesthecommonpractice,becauseof: • 2.Theincreasingcomplexityoftrainingmodel ImageNetCompetition: <10(Hinton, 2012), 22 (Google, 2014), 152 (Microsoft, 2015), 1207 (SenseTime, 2016)

  4. Background • ParameterServer(PS)-basedarchitectureiswidelysupportedbymainstreamDMLsystems.

  5. Background • However,thepowerofPSarchitecturehasnotbeenfullyexploited. • 1.Communicationredundancy • 2.Stragglerproblem

  6. Background • Adeeperinsight… • 1.Worker-centricdesignislessefficient • 2.PScanbemoreintelligent(i.e.SmartPS) SmartPS

  7. Background • TomakePSmoreintelligent… • Dependency-Aware • Straggler-Assistant

  8. ASimpleModelofParameters

  9. WorkflowofPS-basedDML

  10. WorkflowofPS-basedDML

  11. WorkflowofPS-basedDML

  12. DesignStrategies • TomakePSmoreintelligent… • 1.Selectiveupdate() • 2.Proactivepush() • 3.Prioritizedtransmission() • 4.Unnecessarypushblockage()

  13. Strategy1:SelectiveUpdate

  14. Strategy1:SelectiveUpdate

  15. Strategy1:SelectiveUpdate

  16. Strategy1:SelectiveUpdate

  17. Strategy2:Proactive Push

  18. Strategy3:Straggler-Assistant

  19. Strategy3:Straggler-Assistant

  20. Strategy3:Straggler-Assistant

  21. Strategy4:Blocking Unnecessary Pushes

  22. Evaluation • ExperimentSetting： • 17Nodeswithdifferentperformanceconfigurations:1PS+16Worker • 2Benchmarks: • MatrixFactorizationandPageRank • 5Baselines： • BSP， ASP，SSP（slack=1）， SSP（slack=2），SSP （slack=3）

  23. Evaluation MFBenchmark: Withacommonthreshold,SmartPSreducesthetrainingtimeby68.1%~90.3%comparedwiththebaselines.

  24. Evaluation PRBenchmark: Withacommonthreshold,SmartPSreducesthetrainingtimeby65.7%~84.9%comparedwiththebaselines.

  25. FurtherDiscussion • Comparisontosomerecentworks: Bothleveragetheknowledgeofparameterdependency 2.Bothleverageprioritizedtransmission forDMLacceleration

  26. FurtherDiscussion • Comparisontosomerecentworks:

  27. OngoingWork • AdeeperinsightintoPS-basedarch… • FunctionofPS: • 1.ParameterDistribution • 2.ParameterAggregation • FunctionofWorker: • 1.ParameterRefinement ->DataAccessControl ->DataOperation ->DataOperation

  28. OngoingWork ParameterDistribution ParameterAggregation ParameterRefinement

  29. OngoingWork DataAccessControl DataOperation DataOperation

  30. OngoingWork DataAccessControl Token Token Token DataOperation

  31. NextGenerationofSmartPS • ParameterServer->TokenServer • 1.Decoupledata(access)controlanddataoperation • 2.Alight-weightandsmartTokenServerinsteadofParameterServer. TokenServer ParameterServer

  32. Thanks! NASPResearchGroup https://nasp.cs.tsinghua.edu.cn/ https://www.gengjinkun.com/

More Related