1 / 17

Large-scale Machine Learning using DryadLINQ

Large-scale Machine Learning using DryadLINQ. Mihai Budiu Microsoft Research, Silicon Valley Ambient Intelligence: From Sensor Networks to Smart Environments and Social Media Workshop Stanford, June 11, 2019. Goal of DryadLINQ. Software Stack. Applications. . Net + LINQ. DryadLINQ.

randi
Download Presentation

Large-scale Machine Learning using DryadLINQ

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Large-scale Machine Learning using DryadLINQ Mihai Budiu Microsoft Research, Silicon Valley Ambient Intelligence: From Sensor Networks to Smart Environments and Social Media Workshop Stanford, June 11, 2019

  2. Goal of DryadLINQ

  3. Software Stack Applications .Net + LINQ DryadLINQ Dryad Cluster storage Cluster services Windows Server Windows Server Windows Server Windows Server

  4. Dryad = Execution Layer Job (application) Pipeline ≈ Dryad Unix Shell Cluster Machine

  5. LINQ Data Model .NET objects of type T Collection

  6. LINQ Language Summary Input Where (filter) Select (map) GroupBy OrderBy (sort) Aggregate (fold) Join

  7. LINQ => DryadLINQ Dryad

  8. DryadLINQ Data Model .Net objects Partition Collection

  9. DryadLINQ = LINQ + Dryad Collection<T> collection; static boolIsLegal(Key c); var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value}; Code Dryad job Data collection C# C# C# C# results

  10. Example: Natal Training

  11. Natal Problem • Recognize players from depth map • At frame rate • Low resource usage

  12. Learn from Data Rasterize Training examples Motion Capture (ground truth) Machine learning Classifier

  13. Running on Xbox

  14. Cluster-based training Classifier Training examples Machine learning DryadLINQ Dryad

  15. You can have it! • Dryad+DryadLINQ available for download • Academic license • Commercial evaluation license • Runs on Windows HPC platform • Dryad is in binary form, DryadLINQ in source • Requires signing a 3-page licensing agreement • http://connect.microsoft.com/site/sitehome.aspx?SiteID=891

  16. Conclusions = 17

More Related