Information Retrieval for High-Quality Systematic Reviews: The Basics

1. Information Retrieval for High-Quality Systematic Reviews:The Basics 6.0

2. 5.0 The Eight Steps of a High Quality Systematic Review • Formulate Review Questions • Define inclusion and exclusion criteria • Locate Studies • Select Studies • Assess Study Quality • Extract Data • Analyze and present results • Select Studies

3. Finding Evidence: A Conceptual Framework Grey Literature Published Literature Electronic Searches Hand Searches Invisible College RCTs CRTs Electronic Registers C2-SPECTR C2-PROT Campbell Collaboration Cochrane Collaboration Systematic Reviews C2-RIPE EPPI Center What Works Clearinghouse

4. Preliminary Searches Supports beginning steps: Definition of key concepts & research question Use of standard reference tools and broad searches for review articles and key primary studies Main Searches Identification of primary studies through searches of online databases, printed indices, Internet, branching, hand-searches Most difficult given a number of challenges Final Searches Occurs towards the end of the Review Process Refine search terms and update original searches Information Retrieval: A Continuous Process

5. Scope of search: Which fields should be searched (e.g. beyond Education)? Availability of indexing tools: Which tools do we have access to at our institution? Format of indexing tools: What format are they in (e.g. online, print, web-based)? Date: How far back does the indexing go for each tool? Language: What is the language of the material that is indexed? How can we locate non-English material? Unpublished work: How can we access dissertations, reports, & other grey literature? Main Searches: Decisions

6. Fields searched: Education, Psychology, Sociology, Social Sciences, Economics, Population, Family Geographic scope of indexing tools: Mostly North American, some UK Format of indexing tools: Online/web-based databases Coverage Date of Indices: Not noted Language: Foreign language included (when possible) Unpublished work: Search of dissertations, web sites, policy documents Example: Parent Involvement

7. Preparation of Search Strategies What are the key concepts to be searched? How are these represented in each field? What are their related terms? How are these key concepts represented in the controlled vocabulary within each indexing tool? Construction of the Search Statements What terms should be searched as descriptors or as “free text”? What boolean operators should be used? What limiting features are available to target primary studies only (e.g. use of Document Type codes)? What time period should be searched? Note. These decisions need to be made for each indexing tool used. Main Searches: Decisions

8. Descriptors: Parent Involvement See: Parent participation Related descriptors: Family involvement, Parent-school relationship, Parent-student relationship, Parent role, Parents as teachers Related keywords: parent* involvement, parent* effectiveness, parent* support, famil* support, famil* participation Descriptors: Academic PerformanceSee: Academic achievement Related descriptors: Science achievement, Reading achievement, Literacy Achievement, Writing achievement, Achievement gains Descriptors: Elementary School Children See: Elementary school students Related descriptors: Elementary education, Primary education, Kindergarten Example: Parent Involvement

9. Combining Keywords/Descriptors using Boolean & Proximity operators: (Parent participation OR Family involvement OR Parent-school relationship OR Parent-student relationship OR Parent role OR Parents as teachers) in DE Parent* NEAR2 involvement or Parent* NEAR2 effectiveness OR Parent* NEAR2 support OR Famil* NEAR2 support or Family NEAR2 participation #1 OR #2 (Academic achievement OR Science achievement OR Reading achievement OR Literacy Achievement OR Achievement gains) in DE (Elementary school students OR elementary education OR primary education OR kindergarten) in DE #3 AND #4 AND #5 Limiting commands: Research reports (DTC=143) Example: Parent Involvement

10. Precision vs. Recall (White, 1994) Precision: ratio of docs retrieved and judged relevant to all those actually retrieved Recall: ratio of relevant docs retrieved to all those in a collection that should be retrieved Regular searches have an inverse relationship - High precision vs low recall (a small set of highly relevant records) Reviewers want high precision and should want high recall (a large, comprehensive set of relevant records with minimal bias) Achieving this balance is the challenge for reviews! Use of incorrect terminology within a field may miss relevant records (low recall) Use of of incorrect or redundant terminology may yield unwanted records (low precision) Main Searches: Challenges

11. Compare the retrieval of 1 record vs 510! elearn (1 record) e near1 learn* (459 records) AU: Holstrom,-Lisa TI: Eliminating Barriers for All E-Learners. SO: Educational Technology. v43 n6 p61-62 Nov-Dec 2003 DE: *Access-to-Education; *Distance-Education DE: Feedback-; Higher-Education; Literacy-; Low-Income-Groups; Skill-Development IDM: *Barriers-to-Participation; *Electronic-Learning IDR: Educational-Media-Use; University-of-Cincinnati-OH (electronic learning) in DEM,DER,IDM,IDR (104 records) ((electronic learning) in DEM,DER,IDM,IDR) or (e near1 learn*) or (elearn) (510 records) Another Example: Distance Education

12. Other Retrieval Strategies Branching: Scanning the reference section of review articles Hand searches: Scanning the Table of Contents of key journals and conference proceedings Personal contacts: Contacting key researchers in the field Consulting C2 resources Main Searches: Decisions

13. “Shoestring-budget information retrieval is likely to introduce bias, and should be avoided.” (IR Policy Brief, 2004) Importance of information retrieval process Not a “one-shot”deal Requires expertise in the planning and implementation of searches Library personnel are important members of the team Use of bibliographic management software Reference Manager, EndNotes, & Biblioscape Ability to replicate review Documentation of entire process, including search strategies used for each database, decisions taken, etc. Summary