1 / 28

Timeliness and quality

Timeliness and quality Information products based on process-produced data – requirements on the quality assurance process Presentation at the 7 th euroCRIS Strategic Seminar Brussels, September, 14-15 th 2009. Agenda. DFG at a glance DFG‘s main source of information

dustin-boyle
Download Presentation

Timeliness and quality

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Timeliness and quality Information products based on process-produced data – requirements on the quality assurance process Presentation at the 7th euroCRIS Strategic Seminar Brussels, September, 14-15th 2009

  2. Agenda • DFG at a glance • DFG‘s main source of information • Timeliness and Quality • Scope of DFG information products

  3. Third-party funding income of German higher education institutions (HEI) by source (2006)

  4. DFG Awards by scientific discipline (2007) Research fields: • Soc.Sci.+Humanities 16% • Engineering Sciences 22% • Natural Sciences 26% • Life Sciences 36% Construction Engineering and Architecture Computer Science, Electrical and System Engineering Humanities 1% 8% 3% 10% Social and Material Science Behavioural Sciences Thermal and Process Engineering 6% 4% Soc.Sc. & Humanities Mechanical and Industrial Engineering Engineering Sciences Biology 6% 13% Geosciences 6% Life Sciences Natural Science Mathematics 3% Physics 10% Medicine 2% Chemistry 21% Agriculture and Veterinary Medicine 7%

  5. DFG programme portfolio and budget (2007) 29.3%: Individual Grants Programme 54.6%: Coordinated Programmes 06.3%: Promoting Young Researchers 01.2%: Scientific Prizes 07.2%: Scientific Instrumentation and Infrastructure 01.4%: Encouraging Scientific Contacts (and other) • Programme portfolio and budget: ~2 Billion €, more than 1.000 programmes and 20.000 projects funded

  6. Dept. directly involved in the funding process The DFG´s Head Office: An overview Executive Board Secretary General President Deputy Members:Heads of Departments I - III Strategic Planning CommitteesExecutive Office Quality Assurance and ProgrammeDevelopment InternationalAffairs Press andPublic Relations InternalAuditing Berlin Office Internal Advisory Committee Extended Executive Board Heads of Divisions and Executive-Level Offices Forum Heads of Divisions Department ICentral Administration Department IIScientific Affairs Department IIICoordinated Programmes and Infrastructure • Budget and Accounting • Human Resources and Legal Affairs • Information Technology and Infrastructure • Information Management • Humanities and Social Sciences • Life Sciences 1 • Life Sciences 2 • Physics, Mathematics, Geosciences • Chemistry and Process Engineering • Engineering Sciences • Research Centres • Research Careers • Scientific Library Services and Information Systems • Scientific Instrumentation and Information Technology

  7. The funding-process-chain Proposal • Whole chain is represented in theDFG-internal databases and documentmanagement systems Advice Submission Formal examination Reviewer Peer Review / Assessment Statutatory bodies Decision Funding Performance Evaluation Report

  8. Agenda • DFG at a glance • DFG‘s main source of information • Timeliness and Quality • Scope of DFG information products

  9. ElektrA – the data source Facts and figures • DFG internal electronic proposal processing system • in use since 2005 (data coverage from 1982) • used for all funding programmes • used by about 400 out of 750 staff members directly involved in funding processes • data source for all information products published by the DFG

  10. Agenda • DFG at a glance • DFG‘s main source of information • Timeliness and Quality • Scope of DFG information products

  11. Timeliness and quality • Data in ElektrA are very actual („day before“) • Because of the use of data for funding-process related products (e.g. letters to applicants and reviewers) most information are of high quality • But: less use in funding related products results in lower quality

  12. Organisational model for quality assurance (QA) • Quality responsibility is decentralised • staff members are trained to produce good quality data • software helps to avoid mistakes • Nevertheless there is a need for a central quality assurance unit: • development of complex quality assurance routines • occasionally very high demand for timeliness (e.g. ad hoc statistics or „the day after“ information products like reports on decisions of DFG boards) • most important: Training of data producing staff on how to avoid mistakes and – very helpful! – elucidation on the importance and use of infomation products based on these data

  13. Arguments for investing high effort in quality Only one: • Quality of data ist quality of information products is acceptance and practical use for planning purposes of these products

  14. Agenda • DFG at a glance • DFG‘s main source of information • Timeliness and Quality • Scope of DFG information products

  15. Statistical reporting based on the DFG internal databaseExample: Quarterly statistics for the statutory bodies • Statistical reporting for the statutory bodies of the DFG: • e.g. success rates by programme • e.g. number of programmesfunded/rejected • e.g. development ofsuccess rate over time

  16. DFG internal data as a source for evaluative studies Evaluative Study: „Women in the DFG“ (2008) • Topic: Gender equality has been one of the aims of the foundation since 2002 (Article 1 of its statutes) • Objective: Transparency, planning criteria (e.g. development for the modification offunding instruments) • The project team: Prof. Thomas Hinz, Ina Findeisen, Kathrin Auspurg, University of Konstanz • Target group: Statutory bodies, interested members of the public • Data: Mainly based on process-produced data; many other sources of data included • Study (german) and results in brief (engl.):www.dfg.de/dfg_im_profil/zahlen_und_fakten/statistisches_berichtswesen/gleichstellung/

  17. GEPRIS – German Project Information System Characteristics • information system on DFG-funded projects • different search functions • shows all funded projects at a university, department, or institute since 1999 (~ 60.000 projects) Contribution to QA-process • three quality checks – • first during proposal processing • second by applicant before publishing • constant email-feedback by users and applicants

  18. Research Explorer (REx): www.research.explorer.dfg.de Characteristics • bilingual database (german / english) on research institutesat German HEIs and other institutions • detailed information on nearly20,000 institutes at these institutions • links to GEPRIS Contribution • daily by proposal processing within DFG • periodically by QA-team for each university • occasionally by user-feedback

  19. The DFG Funding Ranking Planning tool for universities and other actors engaged in research planning and strategy. Helps to identify research profiles by means of: • Institutional Profiles: Information about research profiles and main research fields covered by German HEIs • Regional Profiles: Unveils regional profiles and clusters of excellence • Structural Profiles: Informs about networks in science • Disciplinary Profiles: Detailed analysis on research activities in different disciplines and fields of research (e.g. biotechnology)

  20. Objectives, approach and databases of the DFG Funding Ranking iele, iele, Indicators cover approximately 90 percent of the third-party funding income of scientific and higher education institutions in Germany. Huge set of additional non-monetary indicators (intern. attractivity, scientific expertise...) No costly and laborious data collection from the research institutions but direct processing of data concerning funding activities of central sources Subject-related Analyses Analyses by source Analyses by recipients Classification according to four scientific disciplines Specifics: DFG: 48 research fields German government: 12 funding fields EU: 8 funding fields AvH / DAAD a.o: 14 research areas Deutsche Forschungsgemeinschaft German government European Union European Research Council German Federation of Industrial Reserach Organizations Alexander von Humboldt Foundation German Academic Exchange Service dimensions of differentiation: countries, states, regions, institutions.

  21. Aspects of statistical reporting within the DFG Funding Ranking Funding rankings and profiles Influence of programmes (e.g. ExIn) Gender equality DFG Funding Ranking Participants in peer review processes International appeal Networks and cooperation structures

  22. Disciplinary profiles of universities • Example: DFG-funding in the humanities • range of information: • Volume of funding • spectrum of disciplines of each HEI • participation of HEI in „Excellence Initative“ • FU Berlin – LMU Munich: Similar vol of funding und similar disciplinary profile – different participation in ExIn

  23. Networks and cooperation structures within DFG funded programmes: humanities • Very dense coll. cluster in Berlin • = Berlin capitalizes its ressources best

  24. Ranking of countries Recipients in Germany 6th Framework Programme of the European Union Land Mio. € kum. % Deutschland 3.024,0 1,8 Deutschland 3,2 Großbritannien 2.369,6 Frankreich 4,5 2.172,3 1.457,9 Italien 5,4 Niederlande 1.107,4 6,1 Spanien 943,8 6,6 Belgien 707,9 7,1 Insgesamt 16.665,3 100,0 Mio. € Art der Einrichtung % Hochschulen 947,9 31,3 Außeruniversitäre Einrichtungen 1.173,5 38,8 5,1 Max-Planck-Gesellschaft (MPG) 154,3 Fraunhofer-Gesellschaft (FhG) 216,4 7,2 316,9 Helmholtz-Gemeinschaft (HGF) 10,5 94,5 Leibniz-Gemeinschaft (WGL) 3,1 128,1 Bundeseinrichtungen 4,2 263,2 8,7 Weitere Einrichtungen 900,9 29,8 Industrie und Wirtschaft 1,7 Nicht-institutionelle Mittelempfänger 0,1 Ingesamt 3.024,0 100,0

  25. The quality assurance chain one data source different products different users one qa-team individual Letters applicants, reviewers management reports report users ElektrA Feedback Quality assurance team funding ranking Ranking recipients GEPRIS GEPRIS users Research Explorer REx users quality assurance

  26. Conclusions • Good quality of data is costly and need personnel • Decentralization and direct use of data for business related products is the best guarantor for a good data base • Besides this it is necessary to centralize QA processes • The more products with certain specifications and certain target groups are based on the same data, the more feedback is possible to improve quality • every data problem causes trouble to a broad set of information products – so high data quality is a good reason for investing in well defined QA processes

  27. Thank you for your attention! • more information: • DFG: www.dfg.de • DFG-Funding-Ranking: www.dfg.de/ranking • DFG- funded projects: www.dfg.de/gepris • research institutes at German HEI: www.dfg.de/research_explorer

More Related