1 / 26

Systematic Literary Reviews: experience and challenges ISERN discussion session

Systematic Literary Reviews: experience and challenges ISERN discussion session 14 Sept. 2010, Bolzano, Italy Reidar Conradi 1), Mohammad Ali Babar 2) 1) NTNU, NO-7491, Trondheim, Norway 2) ITU, Copenhagen, Denmark http://www.idi.ntnu.no/grupper/epos/isern/slr-comments-isern-14sep10.ppt

niel
Download Presentation

Systematic Literary Reviews: experience and challenges ISERN discussion session

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Systematic Literary Reviews: experience and challenges ISERN discussion session 14 Sept. 2010, Bolzano, Italy Reidar Conradi 1), Mohammad Ali Babar 2) 1) NTNU, NO-7491, Trondheim, Norway 2) ITU, Copenhagen, Denmark http://www.idi.ntnu.no/grupper/epos/isern/slr-comments-isern-14sep10.ppt conradi@idi.ntnu.no, baba@itu.dk Systematic Literary Reviews: Experience and Advice

  2. Table of contents • 0. Some challenges of SRLs • What is so special about SE and ESE/EBSE? • Availability and Utility of current SLRs • Utility of SLRs for Agile incl. Pair Programming • SLR: status, challenges, future advice • Future: OSS and networked development • References X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  3. 0. Some challenges of SRL Summary of challenges: What is currently known about the strengths and limitations of systematic reviews for software engineering? How rigorous and relevant are the systematic reviews reported so far? What are the challenges of the adoption of this methodology in software engineering? What are the improvements proposed for maturing this methodology? What is the role of systematic reviews in software engineering research and practice in the coming ten years? How can ISERN members contribute to the evidential assessment of the rigor, effectiveness and relevance of systematic reviews in software engineering? What kinds of Methods are required to ensure the rigor and relevance of systematic reviews for software practice? What can be done to address the most common challenges (such as forming optimal search strategy, assessing primary studies, and synthesizing the evidence) involved in adopting systematic reviews in software engineering research and practice. X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  4. 1.1 What is so special about SE and ESE/EBSE? • Still looking for the silver bullet; cf. Conradi’s own “odyssee”: PLs, ADT/OO, PEs, SE DBs, SCM, Process, Quality/SPI, Reuse/CBSE and gradually open source software (OSS) -- plus ESE/EBSE. • SE is both “hard” and “soft“ – so tough to improve predictibly: still 5+-2 SLOCs per person-hour for large software; 4+2+4 rule. • Five trends: • Massive global reuse/CBSE by OSS - half million software components available • New cooperation and innovation processes around OSS • Flood of mobile/media applications w/ fancy and fast-moving technologies • Some new SE technologies: Agile methods, SOA, cloud computing, … • Empirical track: what works how, where and why – takes time w/ narrow focus – so are we not studying the most relevant issues?  X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  5. 1.2 Don’t forget relevance: PITL vs. PITS • Relevance vs. rigor: PITL vs. PITS, case studies vs. experiments? • Ex. top-10 project killers are PITL-related (Boehm89, Jones02, Charette05): • Goals/commitment • Project management • Communication • Bad requirements • Changing requirements • Estimation, tracking • Product quality • Technical personnel • Development technology • Insufficient software reuse • So try mix agile + OSS + mobile … – and less experiments? X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  6. 1.3 The EBSE challenge How to synthesize relevant experience in a meta-analysis? How to find and retrieve relevant experience – via data mining and semantic searches e.g. by Google or Case-Based reasoning. Costly, but effort not reported! Ex. Medline portal receives 2000 new journal articles per day! Industry: how to motivate for an empirical approach.And yet much turbulence in companies and their contact persons. COMMENTS: Status of available SLRs….. Comments on SLR on Agile methods (Dybå&al.2007). Detailed comments on SLR with a statistical meta-analysis of Pair Programming (Hannay&al.2009). Recommendations. X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  7. 2.1 The utility of existing SLRs, v1 Controlled Experiments (Sjøberg2005): Found 121 such, or 2% of all empirical studies, 20 repeated experiments: 7 of 11 had same results internally, but only 2 of 9 had same results externally (unclear context factors?). Quasi-experiments (Kampenes20xx): need randomized subjects. Effect size (Dybå200x): often too small to avoid Type-II errors, i.e. need a β between 60 and 80. Theory use (Hannay20xx): marginal, as most experiments have their own agenda, and it is hardly not “feasible” for an experiment to be repeated N > 100 times. Cf. 3p physics papers with 150 co-authors, and similar in medicine with over 100,000 subjects. => Merge above insights in a shared recommendation!! Means that future EBSE experiments must be more closely coordinated – but by whom and how? X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  8. 2.2 The utility of existing SLRs, v2 Estimation of effort/schedule (Jørgensen20xx): stable 30% overruns in the last 30 years. Domain expert will usually beat a general and formal tool. Knowledge management (Dingsøyr&Bjørnson20xx): not covering actual use. Agile methods (Dybå&Dingsøyr2007): see later. PP meta-analysis (Hannay&al.2009): see later. Open Source Software (Hauge&al.2010): Industrial OSS adoption. I.e. a Scandinavian discipline??  X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  9. 2.3 The utility of existing SLRs, v3 From the Univ. Keele portal: ++??add more Design inspections (Runeson20xx): generally more effective than unit testing. Testing (Juristo20xx): Unit testing showing diverse results. Software Architecture (Travassos20xx): ... Requirements engineering (20xx): ... ... X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  10. 2.4 Some quasi SLRs, v4 SCM: SCM overview (Mili and Mili and Mili) Version models (Conradi&Westfechtel2007) ICSE IMPACT paper on SCM (Estublier&al.2007) Reuse: HP?? (Wayne Lim) Industrial reuse (Mohagheghi&Conradi2008) Inspections and testing: ICSE IMPACT paper on defect discovery (Biffl&al.2007) Grady’s book?? Gilb&Graham book?? X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  11. 2.5 More conventional SLRs, v5 ICSE: SE roadmap book (2000), FOSE roadmap book (2007). IMPACT papers ESE community: laws: Endres/Rombach Dagstuhl seminars 2007 (Basili et al.) Tutorials, keynotes, textbooks, industrial seminars Make comparison table of some of this? X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  12. 2.6 How to find more SLRs: what queries work? Repeat for i=1..n over the entire Google universe (not limited to ACM, IEEE, …) the following query: Software and SE-subarena_i and (Literature Review or Review or Overview or Survey of or Lessons learned or Experience With or …). Where SE_subarena_i: <empty>, engineering, requirements engineering, design, architecture, coding, implementation, programming, testing, project management, configuration management, … But much manual post-processing of the query results! X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  13. 3.1 Two SLRs on Agile Methods Agile manifesto: improvisation and quick changes, not plan-based and stiff waterfall regime. Agile technologies: XP (12 techniques incl. PP and TDD), Scrum, EVO, DSDM, rapid prototyping, … Much overselling, with an anti-metrics culture. Recent challenges with global, distributed/mobile teamwork. (Dybå&Dingsøyr2007): SLR of 33 primary research papers on Agile technologies. (Hannay&al.2009): Meta-analysis of 18 primary research papers on PP. X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  14. 3.2 SLR on Agile Methods (Dybå&Dingsøyr2007): SLR w/ no clear conclusion of XP benefits, but with perceived positive attitudes. X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  15. 3.3 Reflections on the two Agile SLRs • In Agile: Project management and requirements engineering most importent – not PITS-inspired factors like productivity or defect density. • In first SLR on Agile: 3 XP (PP?) experiments in 2003-2005. • In second SLR on PP: 18 other experiments, 9 from 1998-2005, 9 from 2006-2007. X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  16. 3.4 SLR on Pair Programming (PP): intro PP: One of the 12 methods in XP, with roots in generous teamwork elsewhere – cf. book by (Weinberg1971). So, what do we know on PP: either less effort, shorter duration, better “quality”– or just happier people with more shared knowledge and experience, learning from another. Moderate effects on programming-related factors: +- 15%. (Hannay&al.2009): A statistical meta-analysis of PP experiments in 18 primary research papers. Technical emphasis: quality, duration, effort – not longitudinal studies of social/cognitive factors to promote learning. X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  17. ”3.5 SLR must consider ”Project-triangle” – three competing variables or aspects Effort Duration/ deadline X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  18. 3.6 SLRs on Pair Programming (PP) – in detail No common experiment organization or treatment wrt.: Goals and objectives. Team composition: software expertise (student vs. practitioner, programming skills), soloists vs. pairs, application task, use of external reviewers, … Software technology: Eclipse; UML; Java, C, C++ Randomized vs. self-chosen team composition. And here: no social/cognitive aspects studied. Tree major dependent variables: Quality (hmm?) – see next slides Duration (hours) – elapsed time Effort (person-hours) – often just named hours No volume / productivity measures (SLOC, FP, …), calculated by tools. X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  19. 3.7 SLRs on PP – in detail X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  20. 3.8 SLRs on PP – the Quality results X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  21. 4.1 The future: how to consider OSS? OSS:changes the whole paradigm of software development and associated economic patterns and interactions – by cooperative, distributed innovation. Norwegian software-intensive companies and public institutions must undergo this economic and cultural revolution: novel Innovation Models: new products and services novel Business Models: make money on these Need professional partnership communities to establish and evolve their needed software, itself being OSS. Move away from “unpredictable” volonteers working for free. OSS: adopted by private and public policy makers: e.g., IBM and Sun Microsystems, IKT-Norge, …, Skattedirektoratet, KS, … St.meld. 17 (2006-07) by Norwegian Government recommends OSS and open standards. www.FriProg.no competence center. Also www.FriNett.no (NFR). X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  22. 4.2 From linear to networked paradigm Before: Now: X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  23. 4.3 Networked development actors Emerging network paradigm for OSS will affect: • Long-term vs. short-term planning, • Cross-company cooperation and dependence: (re)negotiation vs. dictate, • Control mechanisms: many risky parts, • (Open) Innovation: new combinations, spin-offs, … • Legal: IPRs and licensing, • Economical: new business models, • Marketing: close to development, • => Requirement Engineering reborn! • See [Ayala09] [Hauge10] [Lindman09]. X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  24. 4.4 OSS Ecosystem for support - by Friprog-.no X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  25. 4.5 Summary of X-disciplinary aspects in OSS • Situated selection of promising components, pragmatic approach, later auto-generated ontologies for functionality properties. • New ways of cooperation and sharing: new work organization, very interesting paradigm – “The world is flat”, global cooperation. Ex. Cooperate on shared, domain-specific middleware. • Open Innovation model: altruism gives the highest long-term profit, by combining several open technologies into new products and services [Browning08] [Chesbrough03] [Chesbrough03a] [Chesbrough06]. • Business model: and making money of this - proven to work! • Apache: open and shared source supported by a cooperative foundation (“IBM”); separate payable services. • eZ: dual model with free previous version, payable current version plus services and support. 25 X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

  26. 6. References (1) [Ayala09] C. Ayala, Ø. Hauge, R. Conradi, X. Franch, et al.: ”Challenges Using the OSS Component Marketplace in the Industry”, Proc. OSS'09. [Bowker99] Geoffrey C. Bowker and Susan Leigh Star: ”Sorting Things Out - Classification and Its Consequences”, MIT Press, Boston, USA, Oct. 1999, 389 pages, ISBN 978-0-262-02461-7. [Browning08] L. D. Browning, Alf Steinar Sætre, K. Stephens and J.-O. Sørnes: ”Information and Communication Technologies in Action: Linking Theory and Narratives of Practice”. New York: Routledge, 2008. [Chesbrough03] Henry W. Chesbrough: OPEN Innovation: The new imperative for creating and profiting from technology, Harvard Business School Press, 2003. [Chesbrough03a] Henry W. Chesbrough: “The era of open innovation”, MIT Sloan Management Review, 44(3):35-41 (2003). [Chesbrough06] Henry W. Chesbrough: OPEN Business Models – How to Thrive in the New Innovation Landscape, Harvard Business School Press, 2006. [Hauge10] Ø. Hauge, D. Cruzes, R. Conradi et al.:"Risks and Risk Mitigation in Open Source Software Adoption", Proc. OSS'2010. [Lindman09] Juho Lindman et al.: “Beyond the Business Model: Incentives for Organizations to Publish Software Source Code”, Proc. OSS'2009. X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

More Related