(2) Locating Studies

1. (2) Locating Studies

2. Overview • General Information to keep in mind: • A meta-analysis is only informative if it adequately summarizes the existing literature • HOWEVER, it does not have to be comprehensive (fail-safe n) but needs to be close to comprehensive • Techniques - database searches, ancestry approach, descendancy approach, hand searching, invisible college

3. Available approaches (from Johnson & Eagly, 2000) (1) Database searches • Which databases? • Depends on topic • Obviously PsycINFO • Ask Librarian if other databases are relevant • Which search terms? • For studies in hand, see “Descriptors” • For studies in hand, see “Identifiers” • Use wildcards: juror*

4. Available approaches (from Johnson & Eagly, 2000) (2) Ancestry approach • Search reference list of articles in hand • What to do about referenced unpublished? • What to do about referenced poster/talk? • What to do about foreign language?

5. Available approaches (from Johnson & Eagly, 2000) (3) Descendancy approach • Use “cited by” feature in PsycINFO? • What about SSCI? • What if there are differences between the two?

6. Available approaches (from Johnson & Eagly, 2000) (4) Hand searching • Scan individual journals • Provides a good cross-check • May find “hidden” articles • May find new Descriptors / Identifiers • Don’t spend too much time on this

7. Available approaches (from Johnson & Eagly, 2000) (5) Invisible college • Using the network of researchers • Email listservs • When in the process should you send the email? • The best time to send the email is: (1) after you have finished your first pass through finding ES for each study because then you will have a clear idea of what you need and how to craft the email letter appropriately, (2) but before you have your coders start coding, otherwise you will have to re-do the coding for the new studies.

8. Overarching Principles (1) Inclusion/Exclusion criteria • Locating Studies (Step 2) goes hand-in-hand with Identifying Hypothesis (Step 1) • This is partly an a-priori determination of what to investigate • This is partly an on-going dynamic process where you review the literature and decide what to investigate • This is partly an ad-hoc statement or summary of what you investigated

9. Overarching Principles (2) Explicit and open to scrutiny • Must detail in “Method” how you found sources so must cover all your bases, otherwise reviewers may argue about your methods • So read “Method” sections of other meta-analyses for information and copy best ones • Keep a record of what search terms you used, what databases you used, etc. See page 19-20 of my Quals for example

10. Overarching Principles (3) Comprehensive? • “Garbage in – Garbage out” • If cast too wide a net and need to trim down, then try setting conceptual boundaries such type of IV, type of DV, domain, paradigm, etc. • Ideally comprehensive but doesn’t have to be comprehensive (fail-safe n) but needs to be close to comprehensive

11. Overarching Principles (4) Inaccessible? • For articles listed as “unpublished” or “in press”, contact the authors • For articles in a foreign language, if there is no English translation, then report in Method section which articles were inaccessible due to language issues

12. Concrete Steps: • Create an excel file (See website “Example-DataSet1”) • Why this helps: • Have listing of each article you have found • Sort them into those with relevant data and those without (see bottom of excel file) • For those with relevant data, can start inserting effect sizes (Step 3) and moderators (Step 5) • For those without relevant data, can type in “notes” about why it is not relevant (you will need this information later to report in “Method” section your exclusion/inclusion criteria) • Since listing of each article you found, when come across a new article, can see if you already found it • Within excel file type in article reference (APA format) so can copy/paste whole thing into your papers “Reference” section

13. Concrete steps (cont.) • Things to keep in mind • In example excel file, Step 2 is only columns A, B, C (other columns are Steps 3, 4, 5) • In your excel file, you may create as many columns as you need the example excel file is a cleaned-up version of one I used but the actual one has a lot of information unique to the particular meta. See next slide for my actual excel file by double-clicking