Research Measurement for Realizing Quality & Excellence in Universities

1. Deanship of Quality and Skills Development • Seminar 24 April 2012, Majmaah University, KSA قياس النشاط البحثي خطوة أساسية لتحقيق الجودة والتميز في جامعاتنا Research Measurement for Realizing Quality & Excellence in Universities Tariq A. Aldowaisan, Ph.D., CSP, CQM/OE, CQE, CQA Associate Professor, Kuwait University (KU) MD, Gulf Lead Consultants (GLC)

2. Seminar Objectives • Provide an overview of the meaning of quality; the relationship between quality and excellence; and their applications in the university context. • Recognize the importance of performance measurement in driving performance improvement. • Suggest the four-dimensional OECD performance measurement system, and give examples of its application in the research activity.

3. Agenda • The meaning of quality • Quality obligations of universities • From quality to excellence • Importance of performance measurement • Performance measurement systems: Stakeholders, BSC, & OECD. • Applications of the OECD Model in research activity • Concluding recommendations on performance measurement in research • Overcoming implementation challenges

4. What is Quality? Quality is customersatisfaction through: • Capture of customer requirements • Quality of Design • Compliance with customer requirements • Quality of Conformance Who are our customers?

5. What is Quality? Hard Standards in Production Quality is to • identify and encode customer requirements through STANDARD/SPECIFICATIONS, and then • achieve customer satisfaction by complying with STANDARD. CUSTOMER Defective Defective Diameter 12 ± 0.30 mm 11.7 mm 12.3 mm Defective Payment not to exceed 30 days REQUIREMENTS 30 days

6. What is Quality? Hard Standards in Service

7. CUSTOMER REQUIREMENTS CUSTOMER SATISFACTION meas, analysis improvement resource management What is Quality? Management Systems – Soft Standards Continual Improvement of the QMS clause 4 management responsibility info. flow clause 5 clause 8 clause 6 info. flow clause 7 product realization Input Output product value adding activities value adding activities

8. Management Standards • Examples from ISO 9001:2008 Quality Management System Standard: • 8.2.1 Customer satisfaction As one of the measurement of the performance of the quality management system, the organization shall monitor information relating to customer perception as to whether the organization has met customer requirement. • 8.2.4 Monitoring and measurement of product The organization shall monitor and measure the characteristics of the product to verify that product requirements have been met.

9. Hard & Soft Standards

10. Quality Obligations of Universities • Basic Quality Obligation graduates’ abilities are aligned with the market’s current and potential needs and expectations. • National Quality Obligation contribute to the development of their nations. Source: Aldowaisan, T. “Quality Obligations and Newly Established Universities”, Quality Beat, 1st Issue October 2011.

11. Quality Obligations of Universities Quality Controls • Alignment of the university’s strategic plan with the national plan. • Selection of a robust performance measurement system. • Choice and development of faculty members. Source: Aldowaisan, T. “Quality Obligations and Newly Established Universities”, Quality Beat, 1st Issue October 2011.

12. From Quality to Excellence Meet Requirements Exceed Requirements Satisfy Delight Accredit Rank Good operation (e.g. 3 ) Great operation (e.g. > 6) Customer Focus Organization Focus Customer Results Stakeholder Results Employers Ranking Bodies Society Faculty & Staff Research Community Accreditation Bodies Students

13. BNQP – Performance Excellence Program Education Criteria for Performance Excellence Framework Organizational Profile: Environment, Relationships, and Strategic Situation 2 Strategic Planning 5 Workforce Focus 85 85 7 Results 1 Leadership 120 450 3 Customer Focus 6 Operations Focus 85 85 4 Measurement, Analysis, and Knowledge Management 90 2011-2012 Education Criteria for Performance Excellence www.nist.gov/baldrige/publications

14. BNQP – Performance Excellence Program Importance of Performance Measurement Organizational Profile: Environment, Relationships, and Strategic Situation 2 Strategic Planning 5 Workforce Focus 85 85 7 Results 1 Leadership 120 450 3 Customer Focus 6 Operations Focus 85 85 4 Measurement, Analysis, and Knowledge Management 90 2011-2012 Education Criteria for Performance Excellence www.nist.gov/baldrige/publications

15. CUSTOMER REQUIREMENTS CUSTOMER SATISFACTION meas, analysis improvement resource management ISO 9001 QMS Importance of Performance Measurement Continual Improvement of the QMS clause 4 management responsibility info. flow clause 5 clause 8 clause 6 info. flow clause 7 product realization Input Output product value adding activities value adding activities

16. Excellence Drivers Education Criteria for Performance Excellence Framework Organizational Profile: Environment, Relationships, and Strategic Situation 2 Strategic Planning 5 Workforce Focus Improvement Team 7 Results 1 Leadership 3 Customer Focus 6 Operations Focus 4 Measurement, Analysis, and Knowledge Management

17. Determiners of Deployment Effectiveness • Leadership • a priori, • 2nd weight (120) • Results • clarity of target, • 1st weight (450) • Improvement Team • Spearhead change • Link leadership & Results Triads

18. Results – Get it Right AnEncompassingVision • Times Higher Education • QS World University Ranking • Academic Ranking of World Universities • US News World Best Universities • etc. Achieving success in our universities, www.glc-icm.com.

19. Results – Get it Right RealGoals – Outcome based Teaching Research Community Development

20. Performance Measurement Systems Stakeholders’ Perspectives PMs PMs PMs PMs PMs

21. Performance Measurement Systems Balanced Scorecard (BSC) PMs PMs PMs PMs Source: SmartDraw 2010

22. Performance Measurement Systems OECD Model – Activity Based Source: Measuring Government Activity, OECD 2009 organization structure and management provisions Context Inputs (Resources) Activities Outputs Intermediate Outcomes Final Outcomes

23. Performance Measurement Systems OECD Model National/ University Objectives Effectiveness Efficiency Inputs (Resources) Activities Outputs Intermediate Outcomes Final Outcomes Productivity Quality Customer

24. OECD Performance Measures Dimensions • Efficiency: measures how well resources are utilized to produce the desired output (resource/input-driven). • Productivity: measures the capacity to produce the maximal output possible for given resources (output-driven). • Quality: measures how well the produced outputs meet or exceed relevant quality dimensions standards (output-focus). • Effectiveness: measures how well the outputs and outcomes agree with the planned/targeted outputs and outcomes (outcome-focus) organization structure and management provisions Context Inputs (Resources) Activities Outputs Intermediate Outcomes Final Outcomes Doing Things Right Doing Right Things

25. Key University Activities • Teaching – Undergraduate • Teaching – Graduate • Research • Community Services; e.g. consultation, professional development, and testing.

26. The Research Activity Country Priorities • Projects • Papers • Patents • Spinoffs • Budget • Faculty • Facilities University Ranking Research Activity Research Community Business Community

27. Suggested Performance Measures for Research Activity • Efficiency: the cost of funded published papers (journal & conferences). The cost may be expanded to include the administration of the research activity. • Productivity: the number of published funded and non-funded papers (journal & conferences) per faculty (all faculty). Of course more specific measures of productivity could be developed for journals and conferences separately. • Quality: the percentage of papers (funded and non-funded) that are published in the top quartile of the Journal Citation Reports (JCR) • Effectiveness • the compounded annual growth rate (CAGR) of papers (journal & conference) over a certain period (Immediate outcome). • the percentage of papers addressing national priorities (Intermediate outcome). • the percentage of papers utilized by market (Final outcome).

28. Performance Measures & Objectives 10% utilized by market 2000 KD per paper 25% address priorities 25 % address Effectiveness Efficiency Priorities Research Activity Papers 30% CAGR Research Investment • Faculty members • At least 20 % published in 1 . 5 paper per faculty member top quartile of JCR productivity Quality Note : 1. Papers include conference and journal papers as well as funded and non - funded. 2. Faculty members include all faculty members (active and non - active). 3. For Research Investment Efficiency , we consider funded papers only.

29. Using the PM to Drive Improvement "You can't manage what you can't measure" __ Edwards Deming

30. Advantages of OECD Model • Process/activity based • More appropriate for non-profit institutions • Readily supports ratio performance metrics • Simple, efficient, robust • Supports doing things right and doing the right things

31. Recommendations on Performance Measures in Research • Focus more on reporting outcome-related performances and their development over the year. • Include non-funded published papers as well as conference papers to calculate faculty productivity(i.e. papers per faculty); which is an important measure of immediate outcome (i.e. output). • Include a measure of research cost to calculate research investment efficiency(i.e. KD/paper). • Include a priority effectiveness indexrelated to research relevance to the priorityareas (i.e. KD/KD or Papers/papers). • Include a marketability effectiveness index related to marketability of research output (i.e. KD/KD or Papers/papers).

32. Recommendations on Performance Measures in Research • Include commercial effectiveness indices on spinoff businesses and patents. • Include a leading indicator faculty research engagement (i.e. percentage of faculty involved in research). • Include a leading indicator on faculty collaboration index(KD of researches that involves two or more faculty of total research investment). • Include a leading indicator on interdepartmental collaboration index(i.e. % of collaboration research projects involving collaboration from different departments). • Include a leading indicator on large-scale interdisciplinary research collaboration(i.e. % of collaboration large-scale research projects involving different disciplines). • reflect on the impact of the level and quality of research on the university rank; through engaging credible ranking bodies.

33. Cross-Check for Completeness

34. Cross-Check for Completeness

35. Benefits of Cross-Checking • Ensure selected performance measures are collectively comprehensive. • Ensure that the selected performance measures are mutually exclusive. • Gain the benefits of each measurement framework.

36. Definition of a Performance Measure • Goal/Objective • Interpretation Reference • Formula • Assumptions • Base Value • Dimension • Benchmarks • Measurement Unit • Targets • Data Sources • Analyses • Beneficiaries • Frequency

37. Implementation Challenges • Leadership Commitment • Culture • Fear of accountability • Not meeting a target construed as a failure • Competency building • General awareness starting at top • Deep knowledge & skills of improvement team to cover both measurement and improvement methods and tools • Cumbersome to monitor • Automation is a must

38. كل الشكر والامتنان طارق عبد المحسن الدويسان tariq@glc-im.com tariq.aldowaisan@ku.edu.kw www.glc-im.com