Phoenix – Fostering the rebirth of social sciences and humanities in Central Asia

2. Phoenix – Fostering the rebirth of social sciences and humanities in Central Asia 2nd PHOENIX Workshop “Road to Excellence: Research Evaluation in the Social Sciences and the Humanities” Issyk-Kul (Kyrgystan), 20-22 May 2007 ”Supranational Peer Review in the Social Sciences and the Humanities: some European Experiences” Dr Rüdiger Klein European Science Foundation Dep Head Humanities SSO Research and Foresight

3. 2nd PHOENIX Workshop: “Research Evaluation” ”Supranational Peer Review in the Social Sciences and the Humanities: some European Experiences” • Presentation of institutional background (ESF) • Discussion of lead questions on peer review: - notion of peer review; - examples of peer review; - characterstics of peer review; - steps towards international peer review; - challenges of supranational peer review; • European Reference Index for the Humanities (ERIH)

4. ESF Membership 75 Member Organisations in 30 countries, also beyond the European Union • Research funding organisations (e.g. research councils) • Research performing organisations (e.g. national research institutes) • Academies Combat fragmentation; create critical mass; advance science • Partnerships with agencies outside Europe

5. ESF History and Role • Established in Strasbourg in 1974 • An independent, non-governmental organisation • Offices in Strasbourg and Brussels • Budget (2006): € 41 Mio. • Science budgets networked : € 1-3 Mrd. • Workshops and small research programmes (15.000 – ca. 500.000 €) • 30 large EUROCORES programmes (ca. 6-12 Mio. €) • EURYI young researchers award (1.25Mio.€) • Research foresight (incl. research infrastructures) • ESF Member Organisation Fora

6. ESF Mission ESF provides a common platform for its Member Organisations in order to: • advance European research • explore new directions for research at supranational level Cooperation between ESF Member Organisations, e.g.: - coordinated research programmes; - joint peer review processes and benchmarking; - research infrastructures ESF serves the needs of the European research community in a global context through collaboration with • Non-European, national research funding agencies (NSF, JSPS, CASS; ministries; academies: RAN) • UN (UNESCO etc.); ICSU; UAI; OECD; NATO; etc.

7. European COoperation in the field of Scientific and Technical research • Origin: • Started 1971: Ministerial Conference, – 19 Member States • Current Participation: • 2006: 34 COST Member States + 1 cooperating state (Israel) • COST Actions: • Concerted Actions (Networks) of nationally funded R&D projects [all fields of research]

8. ESF Scientific Standing Committees & Expert Boards Standing Committees • Humanities • Life, Earth & Environmental Sciences • Medical Sciences • Physical & Engineering Sciences • Social Sciences Expert Boards • Committee on Radio AstronomyFrequencies • European Polar Board • European Space Science Committee • Marine Board • Nuclear Physics European Collaboration Committee

9. Fields: Business & Administrative Sciences Communication Sciences Demography Economics Environmental Sciences Geography International Relations Law Pedagogy & Edu-cational research Political Sciences Psychology & Cognitive Science Social Anthropology Social Statistics & Informatics Sociology Women’s Studies Social Sciences at ESF

10. Disciplines: Anthropology Archaeology Art & art history History History & philosophy of science Languages & philologies Linguistics Literary studies Musicology Pedagogy & Edu-cational research Philosophy Psychology Religious studies & Theology Humanities at ESF

11. Humanities at ESF Fields such as: • Area studies • Classical studies • Cognitive science • Communication & media studies • Culture, develop-ment, environment • Gender studies • Heritage studies • Urban studies etc. Research questionssuch as: • Consciousness • Evolution of cooperation • Global change • Health & welfare • Human dignity • Landscape research • Migration • Security • Technology, culture and society etc. Research Infrastructures

12. Peer Review: lead questions Discussion of lead questions on peer review: - notion of peer review; - examples of peer review; - characterstics of peer review; - steps towards international peer review; - challenges of supranational peer review;

13. Peer Review: lead questions Examples of Peer review Assess track-record and potential: • Application for a research grant (individual; institutional) • Application for a position (individual) Assess product: • Submission of a manuscript (article, book etc.): assess “quality” (methods, results, presentation etc.) • Evaluation of research programmes, research institutes (universities, academies), research funding agencies: assess performance over time [benchmarking] Sub-category: • Examination of a degree candidate

14. Peer Review: lead questions Peer review as a social process Formal tradition from 17th century (at least) “market in scientific goods” (knowledge): control over access to resources for research “competitive struggle for scientific authority” (epistemological conflicts) Risk: denial and elite privilege Yet: notion of expertise often conveys idea of objective knowledge (and possibility of progress) Distortions rarely accounted for or made explicit: economic bias (“mafia”); professional, academic bias (research interests); personal bias

15. Peer Review: lead questions Peer review as a social process (cont.’d) Peers agree / disagree: which is best? Scientific merit: BUT • Conventional research (interdisciplinarity); • Institutional conservatism (status of researcher or department); Ethical Issues • Personal bias / research profile; • Sexism; • “Old Boys Networks” Institutional traditions of peer review

16. Peer Review: lead questions Suggestions for criteria for “successful” peer review “Success”: likely selection of best proposals according to scientific merit • Openness: transparency of the process (procedures, criteria, [sometimes names] known to applicants and public) • Multi-level selection (external mail review; panel) • Clear conflict of interest guidelines (published): training of panel members; written commitment of external reviewers • Knowledgeable secretariat (suggestion of peer group: grasp of science; procedures; IT) • Use of support tools where and as appropriate (e.g.: scientometrics; recognition of outcome/outreach)

17. Peer Review: lead questions Suggestions for criteria for “successful” peer review “Success”: likely selection of best proposals according to scientific merit IMPORTANT: • recognise fallibility (“struggle for capital”); • maintain flexibility (interdisciplinarity; internationality) • monitor group dynamics; • improve accountability For international peer review: - acknowledge cross-cultural differences (description of science fields; interdisciplinarity)

18. Peer Review: lead questions Forms of international peer review • international composition of juries for selection in national competitions; • international composition of research groups; • competitions of international grant-making agencies; • international coordination of competitive funding from different national grant-making agencies to support international research groups: “juste retour”: - agencies pays only successful researchers from own country; - funding partly according to ability of agency “common pot”: - all agencies contribute to joint funds; - funding exclusively according to excellence of researchers

19. Peer Review: lead questions Steps towards international peer review Specific challenges in the Humanities: • International research perceived as new phenomenon (but: academies; learned societies); • Role of national research traditions (incl. language); • Contribution of Humanities research to national identity debates (e.g. history, literature, anthropology); • Support tools lacking (indicators) Overall: Languages considered not as a barrier, but language diversity considered a cultural and intellectual asset

20. European Reference Index for the Humanities (ERIH) Starting point: - assessment: unsuitability of existing databases (e.g. ISI) for journal output of European and other non-Anglo-Saxon Humanities research - access: poor visibility of much of journal-based European scholarship in Humanities N.B.: Internationalisation of research (careers; collaborations; ERC etc.) requires comparability of quality across borders and languages Objective 1: identify and categorise good quality research journals in the Humanities - New: mapping of Humanities research in all languages located in an international context; - Encourage best practice (peer review): dialogue with publishers and editors; Objective 2: improve access to and visibility of Humanities research

21. European Reference Index for the Humanities (ERIH) Process: involve research community (different levels of peer review) - 30 national funding agencies (Europe) collect input through consultation of research community: 14.000 suggestions - 15 international expert panels [unpaid] analyse input and categorise journals (18 months); - international subject associations and specialist libraries widely consulted: 6.000 comments; - international journals from the entire world included [A&B] ; language-specific journals so far only from Europe [C] - open feedback form and continuous updates (April 2007 onward); structured interaction with publishers and editors (and other stakeholders) through workshops etc.

22. European Reference Index for the Humanities (ERIH): disciplinary coverage 2006

23. European Reference Index for the Humanities (ERIH) Next steps: - include new areas: applied ethics; communication & media; archives, museum, library studies - consider subdivision of large fields; - testing of robustness of the lists requested - ERIH National contact points: improve national feedback - consider models to include monographs, conference proceedings etc. - widen geographical consultation through partnerships with non-European agencies and initiatives (East Asia; Russia/NIS/Caucasus; Latin America; Africa) - widen discussion on use to include other perspectives: Research Infrastructure; Virtual Learning Environment (“Permanent Access to the records of science”)

24. 2nd PHOENIX Workshop: “Research Evaluation” ...thank you for your attention For further information , please visit www.esf.org, or write to: rklein@esf.org

25. www.esf.org