How are research proposals to SSHRC evaluated A look inside the black box of peer-adjudicated social science

1. How are research proposals to SSHRC evaluated?A look inside the black box of peer-adjudicated social science Charles Davis RTA/FCAD Ryerson University 15 September 2005 Member, SSHRC committee 21, 2001-2004 Chair, 2002-2004

2. Basic program features A Standard Research Grant (SRG) is intended to fund a 3-year research program Up to $250K over 3 years to individual or team Maximum $100k/year 2447 SRG proposal adjudicated 40.1% funded 28.9% of requested funds approved (~ $80M) The success rate of new scholars is about 10% lower than that of established scholars

3. Classics, ancient and mediaeval studies, religious studies, classical archaeology01 History: history of science, technology and medicine02 Fine arts: history and philosophy of art, architecture, theatre, music, film, dance03 Linguistics, applied linguistics and translation05 Economics07 Sociology and demography08 Geography, urban planning and environmental studies09 Psychology10 Education 1: Arts education, bilingual education, civic education, computer assisted instruction, counselling and career guidance, early childhood, educational psychology, environmental education, geography, health sciences education, history, mathematics, moral, values and religious education, pedagogy, physical education, reading and writing, science, second language, special education and vocational education (For additional disciplines, see Committee 17) 12 Interdisciplinary and multidisciplinary studies15 Anthropology and archaeology16 Education 2: library and information science and archival science: adult, continuing and community education; comparative education, curriculum, distance education; educational administration, planning, and governance; history, philosophy & theory of education; higher education, measurement and evaluation, sociology of education, teacher education (For additional disciplines, see Committee 12)17 Literature 1: English (from the Mediaeval to the Victorian period), French; German; Slavic18 Literature 2: American, modern and contemporary literatures in English, English Canadian, First Nations, French Canadian & Qu�bec, romance, other languages & literatures19 Health studies and social work 20 Human resources management, information systems, international business, management; marketing, organizational studies; business policy, industrial relations21 Accounting, finance, management science, productions and operations management22 Law, socio-legal studies and criminology23 Political science and public administration24 Philosophy25 Communication, cultural studies and women's studies26

4. golden rules of peer-adjudicated grantmaking The process is objective. It does not matter whether you have friends or colleagues on selection committees. The process is not random. It is not a form of lottery. Winning proposals are not selected. weaker proposals are eliminated from the competition � the winners are those that remain.

5. The �rules of the game� perpetuate the Matthew Effect �Unto he that hath shall be given.From he that hath not shall be taken away�

6. Scoring formula Regular scholar Record of achievement 60% Research Program 40% New scholar Whichever is higher: Record of achievement 60%, research program 40% Record of achievement 40%, research program 60%

7. Research achievement evaluation of the record of research achievement is based primarily on contributions to research the applicant has made within the last six years if the applicant's research career has been interrupted, research achievement is evaluated based on his or her most recent period of research activity. For regular scholars, applicant's five most significant contributions are taken into account in order to accurately situate the most recent six years in the context of the applicant's overall career.

8. Research achievement refereed publications, including books, chapters of books and articles; book reviews by the applicant/co-applicant or published reviews of his/her work; research reports, papers presented at scholarly meetings or conferences, and other forms of written scholarly expression or participation in public discourse and debate which constitute a contribution to research; where appropriate, contributions to the training of future researchers, including the supervision of graduate theses and/or the involvement of students in research activities; research results from previous research grants, other awards from SSHRC or other sources; academic awards and distinctions-new scholars may include scholarships and fellowships

9. Research achievement quality and significance of published work (taking into consideration the quality of the chosen publication venues); originality of previous research and its impact on the discipline or field; quantity of research activity relative to the stage of the applicant's career; demonstrated importance of other scholarly activities and contributions; recentness of output (taking into account the nature of the applicant's career pattern and previous non-research responsibilities);

10. Research achievement importance and relevance of dissemination of research results to non-academic audiences (as appropriate); significance of any previous research supported by SSHRC or any other agency; where applicable, contribution to the training of future researchers. (The committee must make allowances for applicants who have not supervised graduate students simply because their university does not offer graduate programs.) efforts made, where appropriate, to develop research partnerships with civil society organizations and government departments.

11. Record of research achievement

12. Research program: one or more projects over 3 years explicit objectives, situated within the context of current scholarly literature; relationship of the proposed research to the individual's ongoing research or to insights gained from earlier achievements-, importance, originality and anticipated contribution of the proposed research; theoretical approach or framework; research strategies or methodologies (detailed methodology not necessary);

13. Research program plans for the communication of research results within and beyond the academic community specific roles and responsibilities of students and research assistants, including how their duties will complement their academic training; relationship of requested budget to proposed program of research.

14. Research program: evaluation criteria degree of originality and nature of expected contribution to the advancement of knowledge scholarly and intellectual as well as social and cultural significance of the research appropriateness of the theoretical approach or framework; appropriateness and expected effectiveness of the research strategies or methodologies suitability and expected effectiveness of plans to communicate research results both within and, as appropriate, beyond the academic community

15. Program of research

16. Score needed for funding

18. Some common errors Theoretical framework weak or insufficiently explained Methodology weak or insufficiently explained A project is extended over 3 years to make it look like a program Budget is padded or poorly formulated Padding of CV �me too� proposals SSHRC funded research like this last year Another research project in already worked-over area Implausible teams

19. Some common errors Failure to respect page limits (6 pages means 5.75-6 pages � 6.1 pages is no good) Include literature review or information compilation as research Grad students� roles not consistent with research program flow and objectives In a resubmission, failure to take into account the views of the committee and the external assessors Ultra cartesian or ultra baconian research designs

20. Risky storylines �I�m Too Important to Submit a Fully Worked-out Research Proposal � my record speaks for itself� Variant: �We�re a team of Very High Profile Researchers. Our collective Research Achievement is off your scale� �The fate of the world hangs on the outcome of my project�

21. Risky storylines �My colleague got a grant last year to work on hamsters, so I will work on hamsters also� �It would please God if this proposal were funded� �My research results will overturn all established theories� �The Minister mentioned that this would make a great research project� �Because of the proliferation of incommensurable discourses in late postmodernity, you cannot understand what I am saying and I cannot understand my respondents, but I will study them anyway if you pay for it�

22. A typical 3-year program Year 0: literature review completed; methods and instruments selected; preliminary hypotheses formulated Year 1: refine instruments and hypotheses through qualitative research (focus groups, grounded theory, etc.). Test instruments Year 2: apply instruments for data gathering Year 3: analysis, interpretation, modeling, dissemination of results

23. Common winning formulas for new scholars New scholar with good track record extends doctoral research via 3-year program Watch out. If the proposed research is too close to the doctoral research, it will be regarded as derivative. If it is too far away, it will be regarded as too bold. The most successful ones have published several articles (often with their PhD supervisor) before applying for a first grant

24. Common winning formulas for new scholars New scholar as PI with established scholar as co-investigator with specified roles The co-investigator brings up the �research achievement� score in proportion to his/her role in the project

25. Budgeting tricks and traps the committee may reduce your requested budget. It is good to ask for money for grad student stipends � build grad students into your program Note: It is best to use doctoral students. In regional universities it is OK to use master�s students. If you use undergrads, make sure you have a good reason.

26. Budgeting tricks and traps Do NOT request conference travel money in Year 1. Hint: OK to request modest funds for grad student travel to conferences, if they present. Do NOT inflate travel cost estimates. it is permissible to include travel costs of work with research collaborators, but not collaborators� research costs Note that research travel costs include dissemination costs, which are also calculated separately

27. Budgeting tricks and traps Do NOT request funds for computers unless computers are clearly necessary for the research and they are unavailable through the university OK to ask for laptops for field research NEVER ask for funding for less than three years

28. Budgeting tricks and traps Research Time Stipends are only available if the home university contributes one-to-one matched funding Do NOT request funds for books. SSHRC does not like to pay for books. Be CAREFUL if you request funds for consultancies � this is thin ice NOTE THAT once the budget is approved, SSHRC says that you can spend your grant however you like � but your University controllers do not necessarily know this. At any rate, you cannot pay yourself an honorarium.

29. Budgeting tricks and traps See SSHRC�s list of ineligible items. It includes: training, purchase or rental of standard office equipment, preparation of teaching materials, entertainment and hospitality costs, research leading to a degree, fees and honoraria to colleagues, indirect or overhead costs, etc. ALWAYS include a clear explanatory budgetary note

31. Thank you! Questions or comments?