Developing Tools for Scientific Communities in the Apache Software Foundation: Beyond Open Source

1. Experiences Developing Tools for Scientific Communities in the Apache Software Foundation: Beyond Open Source Marlon Pierce Indiana University February 14, 2012

2. IU Science Gateway Group Overview • IU Science Gateway Group members • Marlon Pierce: Group Lead • Suresh Marru: Principal Software Architect • Raminder Singh, Chathura Herath, Yu Ma, Lahiru Gunathilake: Senior team members • Five interns and research assistants • NSF SDCI funding of Open Gateway Computing Environments project • TACC (M. Dahan), SDSC (N. Wilkins-Diehr), SDSU (M. Thomas), NCSA (S. Pamidighantam), UIUC (S. Wang), Purdue (C. Song), UTHSCSA (E. Brookes) • We participate in two Apache incubators • Apache Rave: http://incubator.apache.org/rave/ • Apache Airavata: http://www.airavata.org

3. Key Problems for Science Gateway, CI Software • Reusability • Reuse or write own gateway software? • Sustainability • Small teams • Successful gateways become infrastructure • Hard to fund • Cyberinfrastructure Software Sustainability and Reusability: Report from an NSF-funded workshop • https://scholarworks.iu.edu/dspace/handle/2022/6701 • Governance • How are design decisions made? • Who decides if the software is suitable for release? • How do you handle contributions? • How do you add people to the development and project management teams?

4. Proposed Solution: Open Community Software • More than SourceForge, GitHub, Google Code, etc • Those provide excellent Web tools to help developers. • Here we are concerned with community building. • Diverse community of developers means reusability and sustainability • But diverse communities require governance

5. Some Open Model Examples in CI • NSF-funded CDIGS project • http://confluence.globus.org/display/CDIGS/CDIGS+Home+Page • HUBzero Consortium, Sakai Foundation • Institutional level organizations • Eclipse Parallel Tools Platform • Jay Alameda, NCSA: NSF SI2 funding to develop HPC tools workbench • http://www.eclipse.org/ptp/ • Enzo Project • Excellent talk at TG11 by Prof. Brian O’Shea on their open community efforts • http://enzo-project.org/ • Apache Software Foundation • OODT and TIKKA Data Management Projects at NASA JPL

6. Two Apache Software Foundation Case Studies Apache Rave and Airavata Incubators

7. Apache Airavata • Science Gateway software framework to • Compose, manage, execute, and monitor computational workflows on Grids and Clouds • Web service abstractions to legacy command line-driven scientific applications • Modular software framework to use as individual components or as an integrated solution. • More Information • AiravataWeb Site: airavata.org • Developer Mailing Lists: airavata-dev@incubator.apache.org

8. Apache Airavata High Level Overview

9. Example Workflow: Nuclear Physics Courtesy of collaboration with Prof. James Vary and team, Iowa State

10. Apache Rave • Open Community Software for Enterprise Social Networking, Shareable Web Components, and Science Gateways • Founding members: • Mitre Software • SURFnet • Hippo Software • Indiana University • More information • Project Website: http://incubator.apache.org/rave/ • Mailing List: rave-dev@incubator.apache.org 1

11. Gadget Dashboard View Gadget Store View

12. Extending Rave for Science Gateways • Rave is designed to be extended. • Good design (interfaces, easily pluggable implementations) and code organization are required. • It helps to have a diverse, distributed developer community • How can you work on it if we can’t work on it? • Rave is also packaged so that you can extend it without touching the source tree. • GCE11 paper presented 3 case studies for Science Gateways

13. The Apache Way and Science Gateways

14. Why Apache for Gateway Software? • Apache Software Foundation is a neutral playing field • 501(c)(3) non-profit organization. • Designed to encourage competitors to collaborate on foundational software. • Includes a legal cell for legal issues. • Foundation itself is sustainable • Incorporated in 1999 • Multiple sponsors (Yahoo, Microsoft, Google, AMD, Facebook, IBM, …) • Proven governance models • Projects are run by Program Management Committees. • New projects must go through incubation. • Provides the social infrastructure for building communities. • Opportunities to collaborate with other Apache projects outside the usual CI world.

15. The Apache Way • Projects start as incubators with 1 champion and several mentors. • Making good choices is very important • Graduation ultimately is judged by the Apache community. • +1/-1 votes on the incubator list • Good, open engineering practices required • DEV mailing list design discussions, issue tracking • Jira contributions • Important decisions are voted on • Properly packaged code • Build out of the box • Releases are signed • Licenses, disclaimers, notices, change logs, etc. • Releases are voted • Developer diversity • Three or more unconnected developers • Price is giving up sole ownership, replace with meritocracy

16. Apache and Science Gateways • Apache rewards projects for cross-pollination. • Connecting with complementary Apache projects strengthens both sides. • New requirements, new development methods • Apache methods foster sustainability • Building communities of developers, not just users • Key merit criterion • Apache methods provide governance • Incubators learn best practices from mentors • Releases are peer-reviewed

17. Apache Contributions Aren’t Just Software • Apache committers and PMC members aren’t just code writers. • Successful communities also include • Important users • Project evangelists • Content providers: documentation, tutorials • Testers, requirements providers, and constructive complainers • Using Jira and mailing lists • Anything else that needs doing.

18. How To Get Involved • Join the DEV mailing lists. • Grab the software and start complaining. • Post Jira tickets • Add your patches to Jira if you want to solve a problem. • Frequent patch submission is the best way to get voted in as a committer.