Enhancing Higher Education Quality Through Performance Indicators

Quality Indicators Indikator Kualiti dalam Pendidikan Tinggi(Quality Indicators in Higher Education) After completing this topic, you should be able to: Explain the meaning of quality indicators Describe the characteristics of quality indicators Learn the importance of quality indicators in HEI Understand the application of quality indicators in HEI

Quality Indicators Performance cultures in higher education HEIs worldwide have undergone reform to improve quality HEIs have implemented systematic and formalized quality assurance processes to achieve greater efficiency and accountability Establishment of quality models and organizations to audit and review university performance Institutional and national quality models and performance indicators are vital components to raise the standard of HEIs Quantitative performance indicators are used to provide international comparisons

Quality Indicators Rationale for performance indicators To ensure education provided by HEIs equips students for employment and provide the country with a highly skilled workforce that support economic growth. To contribute to educational, social, and political values.

Quality Indicators Purposes of performance indicators in HEIs To monitor own performance for comparative purposes. To facilitate the assessment and evaluation of institutional operations. To provide information for external quality assurance audits. To provide information to the government for accountability and reporting purposes (Rowe, 2004).

Quality Indicators The use of performance indicators in HEIs Ensure accountability for public funds Improve the quality of higher education provision Stimulate competition within and between institutions Verify the quality of new institutions Assign institutional status Underwrite transfer of authority between the state and institutions Facilitate international comparisons Ref: Chalmers (2008)

Quality Indicators Defining performance indicators Simple indicators – expressed in absolute figures and are intended to provide an unbiased description of a situation or process. Performance indicators – imply a point of reference; for example, a standard, objective, assessment, or comparator, are relative rather than absolute in character. Involve value judgements. General indicators – externally driven and are not indicators in the strict sense; they are frequently opinions, survey findings or general statistics. Ref: Chalmers (2008, quoted from Hanney, Henkel & Kogan, 1997)

Quality Indicators Defining performance indicators Currently there is no common definition of performance indicators. PI cannot be considered as facts, but are goal, value and context laden, and utilized in different ways depending on the performance model being used. PI are defined as measures which give information and statistics context; permitting comparisons between fields, over time and with commonly accepted standards. They provide information about the degree to which teaching and learning quality objectives are being met within higher education sector and institutions. Ref: Chalmers (2008, p.3)

Quality Indicators Types of performance indicators Input Process Output Outcome The types can be categorized as quantitative indicators and qualitative indicators. Ref: Chalmers (2008, p.3)

Quality Indicators Quantitative Indicators define as those associated with the measurement of quantity or amount, and are expressed as numerical values. Input indicators Human, financial and physical resources in supporting institutional programmes, activities and services. Output indicators Output reflects the quantity of outcomes produced, including immediate measurable results, and direct consequences of activities implemented to produce results. Do not demonstrate quality of education, but quantities of outcomes. Ref: Chalmers (2008)

Quality Indicators Qualitative Indicators associated with observation based descriptions, rather than an exact numerical measurement or value. Relate to or involve comparisons based on qualities of non-numerical data such as policies and processes for assessing students’ learning, the experience, the content of a mission statement. Outcome Indicators Focus on the quality of educational program, activity and service benefits for all stakeholders. Insightful, meaningful and accurate since they are related to the objectives of higher education. Students are treated as customers. Ref: Chalmers (2008, p.5)

Quality Indicators Qualitative Indicators Process Indicators include the means used to deliver educational programmes, activities and services within the institutional environment. qualitative information on teaching and learning such as policies and practices. Considered as most practical, useful and appropriate measures of quality teaching and learning. Ref: Chalmers (2008)

Quality Indicators Research conducted in Australia Look at 13 process indicators Mission, Vision and Objectives Teaching and Learning Plans and Policies Teaching and Learning Indicators Internal and External Performance Funds for Teaching and Learning Organizational Unit Review Curriculum Review Assessment and Feedback Policies Graduate Attribute Statement Student experience Ref: Chalmers (2008)

Quality Indicators Research conducted in Australia Look at 13 process indicators Professional Development Appointment and Promotion Criteria Review of Academic Staff – performance Recognition of Excellence in Teaching and Enhancing Student Learning Experience Ref: Chalmers (2008)

Quality Indicators Research in Australia – Quality Teaching Look at four dimensions of teaching practice Institutional climate and systems – commitment to the enhancement, transformation and innovation of learning. Measure student experience and level of satisfaction. Diversity – relates to ethnic, cultural and socioeconomic diversity as well as students’ abilities, talents and learning approaches. Ref: Chalmers (2008)

Quality Indicators Research in Australia – Quality Teaching Look at four dimensions of teaching practice Assessment – the assessment tasks of student enrolled in the program of study. Look at the design, delivery and administration, provision of feedback, moderation, and review of assessment. Engagement and learning community – student engagement, i.e. the student commitment and engagement with their own education. Also includes staff engagement. Ref: Chalmers (2008)

Quality Indicators Higher Education in Malaysia– Challenges Peranan universiti dan ahli akademik (University’s and academics’ roles) Perkembangan kurikulum mengikut keperluan pasaran (Curruculum development according to market needs) Penyelidikan, pembangunan dan pengkormesialan dalam sistem inovasi kebangsaan (Research, development and commercialization in the national innovation system) Kaedah pengajaran dan pembelajaran (Teaching and Learning Methods) Perluasan akses dan mobiliti pengetahuan (Accessibility and knowledge mobility)

Pekeliling Kemajuan Pentadbiran Awam Bilangan 2 Tahun 2005 • Garis Panduan Bagi Mewujudkan Petunjuk-petunjuk Prestasi Utama Atau Key Performance Indicators (Kpi) Dan Melaksanakan Pengukuran Prestasi Di Agensi Kerajaan

Terminologi • Petunjuk Prestasi Utama (KPI) ialah salah satu kaedah bagi mengukur prestasi perkhidmatan agensi-agensi Kerajaan • Perkhidmatan Teras Bidang tanggungjawab agensi sejajar dengan visi (core business): dan misi agensi; • Proses Utama Fungsi-fungsi di bawah perkhidmatan teras yang (core process): perlu dilaksanakan bagi menghasilkan perkhidmatan untuk pelanggan; • Key Performance Petunjuk-petunjuk prestasi utama yang ditentukan Indicators (KPI): sebagai asas mengukur prestasi;

NKRA • Education • Crime (Public Safety) • Corruption • Low Income Households • Rural Basic Infrastructure • Urban Public Transportation

Quality Indicators Higher Education in Malaysia– Challenges Globalisasi dan piawaian melalui pemeringkatan dan penarafan (Globalization and standardization through development and accreditation) Peluang guna tenaga (Resource utilization opportunities)

Quality Indicators Malaysia - Current scenario 20 public universities 21 polytechnics 37 community colleges >400 registered private colleges 21 private universities and university colleges 11 local university branch campuses + 5 foreign university branch campus Areas for indicators: Academic staff Educational programs Student selectivity Educational resources Governance The method used will be peer review.

Quality Indicators How are universities ranked? Different ranking approaches: League table – each university is assigned a specific rank. Higher ranks indicate higher quality, lower ranks indicate lower quality. Quality criteria and indicators are used in this ranking methodology to assess universities. Each indicator such as research impact as the number of citations per faculty in the Thompson Scientific Database or teaching quality as in THES are given weight. This approach are applied to all universities assessed.

Quality Indicators How are universities ranked? Different ranking approaches: A ranking of individual disciplines or departments instead of whole institutions. A multidimensional concept of university quality instead of a ‘one-size-fits-all’ approach, taking into account the diversity of academic institutions, missions and goals as well as language and cultural specifics. A separate measurement and presentation of single indicators. A presentation of ranking results in rank groups (top, middle, bottom groups) instead of league tables.

Quality Indicators Purposes of ranking of HEI Ranking serves several purposes: Responds to demands from consumers for easily interpretable information on the standing of HEIs. Stimulates competition among universities. Provides some rationale for allocation of funds. Helps to differentiate among different types of institutions and different programs and disciplines. Contributes to the definition of ‘quality’ of HEIs within a particular country. (source: Berlin Principles of Ranking of Higher Education Institutions, 2006 http://www.che.de/downloads/Berlin_Principles_IREG_534.pdf)

Quality Indicators Berlin Principles of Ranking of Higher Education Institutions Purposes and goals of rankings: Be one of a number of diverse approaches to the assessment of higher education inputs, processes, and outputs. Be clear about their purpose and their target groups. Recognize the diversity of institutions and take the different missions and goals of institutions into account. Provide clarity about the range of information sources for ranking and the messages each source generates. Specify the linguistic, cultural, economic, and historical contexts of the educational systems being ranked – should be aware of possible biases. (source: Berlin Principles of Ranking of Higher Education Institutions, 2006)

Quality Indicators Berlin Principles of Ranking of Higher Education Institutions Design and weighting indicators: Be transparent regarding the methodology used for creating the rankings. Choose indicators according to their relevance and validity. Measure outcomes in preference to inputs whenever possible. Make the weights assigned to different indicators (if used) prominent and limit changes to them.

Quality Indicators Berlin Principles of Ranking of Higher Education Institutions Collection and processing of data: Pay due attention to ethical standards and the good practice recommendations articulated in these Principles. Use audited and verifiable data whenever possible. Include data that are collected with proper procedures for scientific data collection. Apply measures of quality assurance to ranking processes themselves. Apply organizational measures that enhance the credibility of rankings.

Quality Indicators Berlin Principles of Ranking of Higher Education Institutions Presentation of ranking results: Provide consumers with a clear understanding of all of the factors used to develop a ranking, and offer them a choice in how rankings are displayed. Be compiled in a way that eliminates or reduces errors in original data, and be organized and published in a way that errors and faults can be corrected.

Quality Indicators Purposes of Quality Indicator System Colorado State, USA: Encouraging continuous improvement by institutions in achieving high levels of performance. Measuring institutional performance and accountability. Determining funding recommendations and the funding distribution for the higher education system. Build public support for increased funding for higher education.

Quality Indicators Some Quality Indicators Baccalaureate graduation rates Achievement scores of graduating students on various comprehensive examinations, tests, and /or professional specific licensure or certification examinations Graduates employed or continuing their education Institutional support expenditures – administration expenditure, expenditures per student Undergraduate class size Faculty teaching workload

Quality Indicators Current scenario Categories of institutions APEX university Research intensive General The structure of Malaysian Qualifications Agency (MQA) One-stop center for institutions for registration and accreditation of courses

Quality Indicators Accountability and Quality • The concept of accountability and quality assessment in higher education is an international phenomenon • In America, many regions are moving toward ‘performance incentive funding’. • In Europe and Australia, the central government is directly involved in establishing quality mechanisms through: Quality control, Quality audit Quality assessment • The agencies involved are like the Higher Education Quality Council and the Higher Education Statistics Agency (HESA)

Quality Indicators Accountability and Quality • The focus of attention in performance indicators in the U.S. has been cost efficiency, scientific and technical education, management of intellectual property produced at universities, and undergraduate education. • Less attention is paid to graduate education and research. • Categories of costs for higher education: Education and general: faculty and staff salaries, equipment, libraries, administrative and basic academic computing, and certain capital or such as rent. Cost for sponsored research Costs of student living: room, board, clothing, laundry, entertainment, and etc. Cost of foregone earnings: While disengaged from the productive work force.

The Movement in Setting PerformanceIndicators in Higher Education(U.S.A.) • 1980s • Era concerned with growth in enrollments and access was over • Emerging issues include: Public accountability Quality Productivity Undergraduate education • In 1986, all 50 states and the District of Columbia had developed initiatives to improve the undergraduate education

Shift from growth funding (formula funding) toward funding outcomes, results, and performance • These efforts paralleled developments in Europe and Australia • 1990s • The development of performance indicators differs from that in 1980s From voluntary institutional improvement to a system of mandated public accountability • By 1994, 18 states had developed indicator systems • Greater centralization of authority • Issue raised: Will the federal government assume greater centralized control of higher education through areas such as accreditation and financial aids by using a set of national goals and performance standards?

The Future of Higher EducationThe White Paper 2003 • Higher education must expand to meet rising skill needs • The social gap among those entering university remains too wide • Many of our economic competitors invest more in higher education • Universities are struggling to employ the best academics • Funding per student fell 36% between 1987 and 1997 • Universities need stronger links with business and industry

Reports of Institutional Effectiveness(EOIE) Virginia’s Public Institutions of Higher Learning • Annual report to provide meaningful information on the academic quality and operational efficiency of Virginia’s public institutions • To provide evidence of institutional effectiveness – the extent to which institutions accomplish their missions and students achieve their educational goals.

Structure of the Reports(Five Points) • Institution’s mission The mission statement sets a vision for the institution and defines how it will get there. • Institutional profile In-depth views of enrollment and projections of future enrollment. • System-wide measures Include 14 performance measures focused on operational efficiency and factors associated with academic quality: Example: Classroom and laboratory space utilization, percentage of professionally accredited programs and etc.

Institution-specific measures Represent unique aspects of the mission that the college or university chose to highlight • Core competency reports Explore student general education assessments in written communication and technology/information literacy.

Performance Indicators of California Higher Education, 2001 • Describes the scope of the current set of indicators reported by the Commission, and highlights recent trends based on current information related to these indicators. • This report are divided into five categories: Population Context, Fiscal Context, Student Preparation, Student Access, and Student Outcomes

Key Performance Indicators (KPIs) • Key Performance Indicators are quantifiable measurements, agreed to beforehand, that reflect the Critical Success Factors of an organization. • They defer depending on organization • KPIs must: Reflect the organizational goals Be key to its success quantifiable

Curtin’s efficiency and effectiveness Key Performance Indicators (KPIs) Teaching and learning • Effectiveness indicators Quality of graduates Quality of teaching Student progress and achievement Input • Efficiency indicators Teaching and learning expenditure Student progress and achievement

EFFICIENCY & EFFECTIVENESS • EFFICIENCY means: saving TIME, MONEY or EFFORT • Efficiency measures the resources used to attain a certain level of output • EFFECTIVENESS means how well the the job gets done. i.e. the quality of the output. • Effectiveness measures the extent to which outcomes have been achieved

The End of Quality6th Quality in Higher Education International Seminar (Birmingham, UK) May. 2001 • Three Major themes: • Has external quality review has its day? • Has control of quality been usurped by the market and by information technology? • Does the development of mass education necessarily mean the end of quality?

Transforming Quality7th Quality in Higher Education InternationalSeminar (Melbourne) Oct. 2002 • Three main themes: • To reconceptualise how higher education engages with access, employability and funding issues • What constitutes a high quality learning process and outcomes • How might quality evaluation be transformed to help improve the quality of the experience and of the learning?

8th Quality in Higher Education InternationalSeminar (Sheffield) May. 2003 • Two major themes: • How does student feedback inform quality?To what extent do institutions need to adopt new procedures to make student feedback effective? • What does the White Paper encourage a closer link between quality and learning?

The First Session of the Regional Follow-up Committee for the World Conference on Higher Education (WCHE), 2- 3 November 2000, Kuala Lumpur Malaysia • Recommendations to Member Countries: • Need for ongoing efforts to broaden access taking into account the disadvantaged groups (women and ethnic minorities) • Provide increased support for staff development and research • More participation of women in higher education particularly in decision making level

Indicators of Research Quality in Higher Education • The vast majority of discoveries have been made in higher education environment (Dill, 1986). • Review of literature on research productivity highlighted several indicators which include: • Productivity dollars • Productivity publications • Peer evaluation

Productivity dollars The number of dollars generated by research was the most often cited measurement of success Those universities that are ranked higher, their faculty have are adept at obtaining research grants • Productivity publications The number of publications is frequently used as an indicator of quality in research The research that is published is taken as an indication of its quality • The types of publication which determine its quality: Journal articles, monographs, chapters, books Quality: reputation of publication in discipline, distribution of publication, refereed vs. non-refereed journals, invited chapters/papers

OUTPUT & IMPACT FACTOR • Output versus Impact factor of publications • Output refers to how prolific the the research is producing acceptable articles/books • Impact was determined by checking citations of the articles over a period of years

Enhancing Higher Education Quality Through Performance Indicators