LF Deep Learning Foundation Technical Advisory Council Meeting

1. LF Deep Learning FoundationTechnical Advisory Council Meeting January 17, 2019

2. Recording of Calls • This is a reminder that we have decided to record TAC meetings and store them on the TAC Wiki.

3. Antitrust Policy Notice • Linux Foundation meetings involve participation by industry competitors, and it is the intention of the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. • Examples of types of actions that are prohibited at Linux Foundation meetings and in connection with Linux Foundation activities are described in the Linux Foundation Antitrust Policy available at http://www.linuxfoundation.org/antitrust-policy. If you have questions about these matters, please contact your company counsel, or if you are a member of the Linux Foundation, feel free to contact Andrew Updegrove of the firm of Gesmer Undergone LLP, which provides legal counsel to the Linux Foundation.

4. Dial-in Information • Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/430697670 • Or iPhone one-tap: • US: +16465588656,,430697670# or +16699006833,,430697670# • Or Telephone: • Dial(for higher quality, dial a number based on your current location): • US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free) • Meeting ID: 430 697 670 • International numbers available: https://zoom.us/u/achYtcw7uN

5. Agenda • Roll Call • Project Contribution Proposal Review & Discussion: • Pyro • Next Meeting Information • Discussion / Q&A

6. TAC Member Directory *TAC Chairperson

7. TAC Overview • LF Deep Learning Foundation is an umbrella project of The Linux Foundation with the mission of supporting AI, ML and DL open source projects. • LF DL currently has three projects and we are accepting contributions of additional projects. • The TAC is a committee within LF DL that is responsible for assisting in the coordination and communication of projects within LF DL. It is not a technical oversight body, all technical oversight is handled by the individual projects themselves according to their own governance. • Since the March 2018 launch of LF DL, activity of the TAC has included: • approving a project lifecycle for projects of LF DL (incubation, graduation, archive stages); • reviewing and then approving contribution proposals for the Angel and EDL projects; • starting work on a landscape analysis diagram for open source AI projects overall; and • identifying the area of model interchange formats for exploration by the TAC.

8. TAC Wiki • We have established a Wiki for the TAC which can be found here (login to lists.deeplearning.org required): • https://lists.deeplearningfoundation.org/g/tac-general/wiki/home

9. Project Contribution Proposal Review & Discussion: Pyro • Project Pyro • Deep universal probabilistic programming with Python and PyTorch • http://pyro.ai/ • https://github.com/uber/pyro (As of Jan 14th: 50 contributors, 4870 stars) • Presenter: • Fritz Obermeyer, Pyro Engineering Lead, Uber AI Labs

10. Pyro: Deep Universal Probabilistic Programming Fritz Obermeyer, Pyro engineering lead, Uber AI Labs

11. ML Problems at Uber Uber does not have one machine learning problem, it has thousands.

12. ML Solutions in Practice

13. ML Solutions in Practice

14. Probabilistic programming Build custom solutions that combine the best of all worlds:- probabilistic models (handling uncertainty)- high-level programming languages (means of abstraction)- scalable deep learning (learning from large data)

15. Example application: semi-supervised learning Suppose we want to classify handwritten digits... but we only have a few labeled examples

16. Pyro: Python with two extra primitives x = pyro.sample(“x”, Bernoulli(0.5))assertisinstance(x, torch.Tensor)pyro.sample(“data”, Normal(0., 1.),obs=data)theta = pyro.param(“theta”, torch.ones(100),constraint=positive)

17. Pyro models generate data style digit digit decoder = ... # nndefmodel(image=None): pyro.module("decoder", decoder) style = pyro.sample("style", Normal(zeros(20), ones(20))) digit = pyro.sample("digit", Categorical(ones(10) *0.1)) image = pyro.sample("image", Bernoulli(decoder(style, digit)),obs=image)return image image Semi-supervised Learning with Deep Generative Models, Diederik P Kingma, Danilo Jimenez Rezende, Shakir Mohamed, Max Welling

18. Approximate inference as optimization We can do inference by learning a model of the posterior, called a guide: Variational Bayes and Beyond: Bayesian Inference for Big Data, Broderick, T.

19. Pyro guides infer latent variables style digit digit defmodel(image=None): ...defguide(image): pyro.module("encoder_digit", encoder_digit) pyro.module("encoder_style", encoder_style) digit = pyro.sample("digit", Categorical(encoder_digit(image))) loc, scale = encoder_style(image, digit) style = pyro.sample("style", Normal(loc, scale))return digit, style image

20. Pyro SVI trains the guide style digit digit defmodel(image): ...defguide(image): ...svi = SVI(model, guide, optim.Adam(), TraceEnum_ELBO())for epoch inrange(10):for batch in partition(data): svi.step(batch) image

21. The guide now serves as a classifier style digit digit defmodel(image): ...defguide(image): ...# ...train...digit, style = guide(image) image

22. Results style digit digit image

23. Pyro stack Generative models Custom inference components Inference building blocks (e.g. loss functions, MCMC kernels) Pyro Runtime Ops, distributions Effect handlers PyTorch 1.0: JIT, autograd, distributions, tensor math Python C++ CUDA

24. Research advances must interact correctly Subsampling Reparameterization Sticking the Landing SVI Neural Baselines RSVI ADVI Discrete Enumeration Message Passing Rao- Blackwellization DiCE

25. Documentation catalyzes community growth Design interfaces to be easy to document Document internals, e.g.minipyro.py Follow Python & PyTorch idioms Tutorials, examples, reference, forum, tech reports http://pyro.ai/exampleshttp://docs.pyro.ai

26. Community

27. Community We collaborate on PyTorch Distributions Open Design Process

28. Community pyro.contrib.oed pyro.contrib.tracking

29. Community

30. Results at Uber and beyond Pyro is used by large companies like Siemens, IBM, and Uber, and startups like Noodle.AI Pyro is widely used in research institutions includingHarvard, MIT, Stanford, Oxford, Cambridge, and The Broad Institute Pyro solves a range of Uber problems, spanning: sensor fusion, time series forecasting, ads campaign optimization, and data augmentation for deep image understanding Joint Mapping and Calibration via Differentiable Sensor Fusion, Chen et al. (2018)

31. 2019H1 Roadmap Engineering foundations:- Improve support for deployment/serving with PyTorch 1.0- Pay down tech debt and simplify user interfaces- OSS citizenship: push code upstream to PyTorch and GPyTorch Supporting applications, at Uber and beyond:- Optimal experimental design- Sensor fusion & perception- Time series segmentation Core PPL research:- Automating inference patterns- Improving inference algorithm scalability and performance

32. Thank you! For documentation, examples, and more: https://pyro.ai

33. TAC Vote on Project Proposal ›Proposed Resolution: The TAC approves the Pyro Project as an incubation project of the LF Deep Learning Foundation.

34. TAC Vote on Project Proposal RESOLVED: The TAC approves the Pyro Project as an incubation project of the LF Deep Learning Foundation.

35. Next Meeting Information • Upcoming Meetings • January 31, 2019: Invited Presentation: Pachyderm • February 14, 2019: Regular TAC Call (No invited speaker)

36. Questions / Discussion

37. Legal Notices The Linux Foundation, The Linux Foundation logos, and other marks that may be used herein are owned by The Linux Foundation or its affiliated entities, and are subject to The Linux Foundation’s Trademark Usage Policy at https://www.linuxfoundation.org/trademark-usage, as may be modified from time to time. Linux is a registered trademark of Linus Torvalds. Please see the Linux Mark Institute’s trademark usage page at https://lmi.linuxfoundation.org for details regarding use of this trademark. Some marks that may be used herein are owned by projects operating as separately incorporated entities managed by The Linux Foundation, and have their own trademarks, policies and usage guidelines. TWITTER, TWEET, RETWEET and the Twitter logo are trademarks of Twitter, Inc. or its affiliates. Facebook and the “f” logo are trademarks of Facebook or its affiliates. LinkedIn, the LinkedIn logo, the IN logo and InMail are registered trademarks or trademarks of LinkedIn Corporation and its affiliates in the United States and/or other countries. YouTube and the YouTube icon are trademarks of YouTube or its affiliates. All other trademarks are the property of their respective owners. Use of such marks herein does not represent affiliation with or authorization, sponsorship or approval by such owners unless otherwise expressly specified. The Linux Foundation is subject to other policies, including without limitation its Privacy Policy at https://www.linuxfoundation.org/privacy and its Antitrust Policy at https://www.linuxfoundation.org/antitrust-policy. each as may be modified from time to time. More information about The Linux Foundation’s policies is available at https://www.linuxfoundation.org. Please email legal@linuxfoundation.org with any questions about The Linux Foundation’s policies or the notices set forth on this slide. 1/3/18 The Linux Foundation Internal Use Only