Beyond the OS textbook - getting started in OS research

1. Beyond the OS textbook - getting started in OS research Kim Keeton HP Labs kkeeton@hpl.hp.com Dilma Da Silva IBM Research dilmasilva@us.ibm.com http://www.hpl.hp.com/personal/Kimberly_Keeton http://www.research.ibm.com/people/d/dilma/ 2007 SOSP Women's Workshop

2. Getting started - a steep uphill climb? Completed Research Project Textbook Lessons 2007 SOSP Women's Workshop

3. A little bit about Kim • Education • PhD, Computer Science, UC Berkeley (1999) • MS, Computer Science, UC Berkeley (1994) • BS, Computer Engineering and Engineering and Public Policy, Carnegie Mellon (1991) • Career • HP Labs, 1999 - present • Research interests • Storage systems, storage management, dependability, information management 2007 SOSP Women's Workshop

4. A little more about Kim 2007 SOSP Women's Workshop

5. A little bit about Dilma • Education • PhD, Computer Science, Georgia Tech (1997) • MS, Computer Science, Univ Sao Paulo, Brazil (1990) • BS, Computer Science, Univ of Sao Paulo, Brazil (1986) • Career • IBM Research, 2000-present • Assistant Professor, University of Sao Paulo, 1996-2000 • Research interests • Operating systems, distributed systems • Parallel computing, software engineering, mobile computing 2007 SOSP Women's Workshop

6. A little more about Dilma • Musical activities: embracing lack of “natural talent” • Passions: books , knitting • Navigating politics: large extended family  2007 SOSP Women's Workshop

7. Outline • Getting prepared • Picking the “right” problem • Finding a creative solution • Quantitatively evaluating your solution • Communicating your results • The lessons along the way 2007 SOSP Women's Workshop

8. Getting prepared: system building • Sharpening your system-building skills • Find teams and partners that are right for you • Learning by hand-on work, learning by example (Knuth’s literary programming) • Develop aesthetic for elegant, simple solutions; fight temptation (yours, others) to build Taj Mahals! • Your arsenal: • Tool chains, scripting, debugging, performance evaluation, tracing • Model checking • Simulation 2007 SOSP Women's Workshop

9. Getting prepared: Map the territory • Read papers to learn … and to reflect on the approach taken • Identify the good, the bad, and the ugly • Reading groups can be useful • Be curious … with focus • Challenge your pride & prejudice tendencies • Common ineffective behavior: • Blindness to whole bodies of work • Drowning on papers 2007 SOSP Women's Workshop

10. There are lots of right problems, not just one! Many ways to find a great problem to work on: Understand technology trends and changing usage patterns Run experiments to understand real problems Attend talks, including interesting ones in other areas Look for open questions in future work section Carve out niche in larger group project Do an internship to understand “real world” problems Solve a problem that impacts you personally Stumble across it while solving another problem Picking the “right” problem 2007 SOSP Women's Workshop

11. Picking the “right” problem • Do “real stuff”: solve problems that someone cares about • Exposure to industrial-strength real-world problems • Pick a small problem and let it grow • Course projects are a great starting point • Easier to make contributions if you arrive early to a field • Part of the contribution is framing the research question • “Good research is done with a shovel, not with tweezers.” -- Roger Needham • Things to remember: • It doesn’t need to be ACM-Dissertation-Award perfect • You’re not going to work on it for your entire career • Make sure you are excited enough to work on it for several years 2007 SOSP Women's Workshop

12. Keep things simple unless there’s a good reason not to Pick innovation points carefully; be compatible everywhere else Best results are obvious in retrospect Borrow techniques from other fields Machine learning Data mining Optimization Control theory User interfaces Finding a creative solution 2007 SOSP Women's Workshop

13. Computer systems ideas must be quantitatively evaluated Apply the scientific method Use experiments to answer questions Use standard benchmarks Use measurements of real systems Understand your results Quantitatively evaluating your solution 2007 SOSP Women's Workshop

14. Apply the scientific method • Goal: (dis)prove hypothesis • Not to measure what’s easy to instrument • Think of it as minimizing the effort to prove your point • Useful exercise #1: back-of-the-envelope calculations to see potential of an idea • Useful exercise #2: draw graph you expect to see • If experimental results differ: • Flaw in intuition? Flaw in system? Flaw in measurement? • Uncovered interesting behavior? • Use intuition to ask questions, not to answer them 2007 SOSP Women's Workshop

15. Use experiments to answer questions • Microbenchmarks • Focus on a particular aspect of system • Easy to reason about • Macrobenchmarks • Do not focus on particular aspect of system • More realistic evaluations of idea 2007 SOSP Women's Workshop

16. Use standard benchmarks • “For better or worse, benchmarks shape a field.” – Dave Patterson • Benefits of standard workloads: • Workload is well-defined, agreed upon by community • Provides comparability with other systems • May provide load generation tools • Transaction Processing Performance Council (TPC) (http://www.tpc.org) • Standard Performance Evaluation Corporation (SPEC) (http://www.spec.org) 2007 SOSP Women's Workshop

17. Use measurements of real systems • Traces of actual system behavior will ground your work in reality • Storage Networking Industry Association (SNIA) IOTTA Repository (http://iotta.snia.org/repository/resources/) • System call traces: Coda, Drew Roselli's traces and others • Network file system traces: Berkeley Auspex, Harvard NFS • Block-level storage traces: HP Labs cello • Computer Failure Data Repository (CFDR) (http://cfdr.usenix.org/) • Node outages in more than 20 large HPC clusters • Other options: • Work with industrial partners • Measure your local systems 2007 SOSP Women's Workshop

18. Understand your results • Use knowledge of statistics to evaluate results • Example: perform multiple iterations of experiment and calculate confidence intervals • Give yourself enough time to collect and understand the data • Things always take longer than you think • You won’t always get the measurements right the first time • You may decide you need measure something new to make the point 2007 SOSP Women's Workshop

19. Communicating your results • Report in sufficient detail for others to reproduce results • Deadlines: friend or foe? • Ask colleagues for feedback • Early drafts of papers • Practice talks • Benefits • You’ll get useful feedback • People will be more aware of what you’re doing 2007 SOSP Women's Workshop

20. Learning from mistakes • Innovation involves risk • “Would you like the formula for success?" TJ Watson asked. "Double your rate of failure." • Being smart: early failure detection • Inventory of “failures” and “successes” • We learn from both 2007 SOSP Women's Workshop

21. Summary Completed Research Project • It’s a learning process • We can make the journey smoother: • Improve system building skills • Learn from past and current efforts in the research community • Learn about trends and problems in the field • Adhere to scientific methods • Practice careful evaluation • Communicate Textbook Lessons 2007 SOSP Women's Workshop

22. More importantly … HAVE FUN! 2007 SOSP Women's Workshop

23. Eric Anderson Carla Ellis Manish Gupta Jim Gray Arif Merchant Jeff Mogul Brad Morrey Dave Patterson Sharon Perl Anna Povzner Craig Soules Alistair Veitch Janet Wiener John Wilkes Jay Wylie Acknowledgments 2007 SOSP Women's Workshop