STUDENT’S REPRESENTATION ON THE INFLUENCE OF INSTITUTIONAL EVALUATION ON TEACHING IMPROVEMENT

1. STUDENT’S REPRESENTATION ON THE INFLUENCE OF INSTITUTIONAL EVALUATION ON TEACHING IMPROVEMENT Denise Leite¹, Rui A.Santiago², Cecília Lórea Leite³,Cláudia S. Sarrico², A. M. Braga¹, M.E.Genro¹,C.Broilo¹ ¹Federal University of the Rio Grande do Sul, Brazil; ² University of Aveiro, Portugal; ³ Federal University of Pelotas, Brazil. CNPq – ICCTI - FAPERGS

2. This study presentation is part of a broader research programme on evaluation, auto-evaluation and management in higher education involving three public HE Institutions: one in Portugal – U. Aveiro -, and two in Brazil – UFRGS and UFPel. Its main objective is to seek to better understand how the evaluation processes may favor students and teachers participation on institutional and organizational change. As a part of the main research project, this presentation will focus on students’ representation of institutional evaluation in Brazil.

3. EVALUATION AND UNIVERSITY-STATE RELATIONS In the last decade higher education has been deeply altered by government policies specially on evaluation and accreditation politics. The State direct control over higher education institutions seems to be under a gradual replacement by supervision models or by hybrid models. The hybrid models hold characteristics that aim at the blending of the centralizing intervention by the State and the systemic principle of institutional auto-regulation.This seen to be the Brazilian case. The rhetoric of globalization, the competition and the creation of national wealth, and the qualification of the “human capital” contributed for the settlement of the supervision model and by the regulatory mechanisms of the market.

4. QUALITY AND EVALUATION BY THE STUDENTS’ POINT OF VIEW The evaluation done by the students on the curricula, teaching and academic activities constitutes an important item to the development of the universities quality assessment processes. Not translating them into measures constitute failures (Leckey and Neiel, 2001). As consumers (Green, 1994), “clients” or “users” students should be involved in the dialogue about the quality of their educational activities.

5. Research assumptions • In this research we assumed that: • It is possible to think that students are capable to reflect upon their institutional experiences and to question the universities’ evaluation pretexts and processes. • It is possible to think that the students’ have a pre-disposition to involve themselves in the institutional evaluation processes not as “clients” nor as “consumers” but as institutional actors. This can affect their choices about: i) actions to evaluate, ii) patterns to which these actions are compared, iii) the use of the evaluation results. STUDENTS AND EVALUATION

6. DIFFERENCES AMONG STUDENTS’PERCEPTIONS OF THE INSTITUTIONAL EVALUATION Different processes of evaluating universities are developed within European and Latin-American contexts. There are differences among contexts of different universities in the same continent, country or region and in the students’ perspective. There are differences between Brazil and Portugal concerning University – State relation and among quality evaluation politics.

7. COLLECTED DATA Table 1: Study Sample CENTRAL QUESTIONS Meanings given by the students to the concept of institutional evaluation (first question) and its impact on degree improvement (second question). Source: Leite, Santiago, Loréa Leite e Sarrico. Avaliação, auto-análise e gestão das universidades. Relatório CNPq, 2002.

8. RESULTS CATEGORIES

9. Evaluation and teachers The teachershave influence on students insights into the university. Assessment questionnaires of degrees and the “Provão” are “cold” and impersonal, and do not take into account students’ subjectivity or the degrees’ relational, social and cultural dimensions. “I believe the questionnaires we answer at the end of each course help in this evaluation, but they are inconclusive. They do not include the subjectivities of relationships in a classroom, which ultimately influence the results (…)” (History student) “I consider that the evaluation they are talking about is the «Provão». Well, in my opinion, this exam is very limited, since it evaluates just the theoretical side of the knowledge students learned during their degree. It does not consider features that I believe are of extreme importance, such as the personal relationships, talent for expressing ideas and individual growth as a human being during the course  opinion making, ethical and moral ideas; I believe these items are important to any degree, since the university should shape citizens and professionals with a holistic vision of the world.” (Civil Engineering student)

10. Evaluation and teachers Students show deep doubts about the positive effects of universities’ evaluation in what concerns the performance of teachers. “(…) I say this because I’ve had classes where teachers were not pedagogically able and my performance in that subject was determined by my commitment to learning things through books. To tell you the truth, we could count on the fingers of my hand the teachers who are really teachers; most of them have the knowledge, but they don’t know how to teach or have the least of didactic training”. (Mechanical Engineering student) InstitutionalEvaluation procedures does not produce positive effects in the teachers’ performance in students’ opinions. Evaluation does not recruit or test teachers and we would continue to observe a permanent reproduction of practices. Research is overvalued and pedagogy is undervalued in students opinion.

11. Constructive Evaluation: evaluation as and exercise of responsibility and accountability. Evaluation is seen as a way to improve situations and to assume the shape of a permanent process; and should be directed to the development of the university. “Evaluation is always valid because it leads to reflection and generates the ability to reflect critically about the work of universities, their know-how and procedures crystallized by tradition, inducing the redefinition of goals and the establishment of new priorities”. (Education student) Evaluation would be an exercise of social responsibility. Society or the community can evaluate universities as to the quality of their teaching and provided services. “Evaluation is always important. The question is who evaluates, what the criteria are, its dissemination and result applicability. In the case of universities, evaluation should be done by all the surrounding community; the criteria should take into consideration the two main roles of universities, which are the production and dissemination of knowledge (is the university being effective?); finally the evaluation results should be widely broadcasted and applied”. (Mathematics student) “I believe evaluation should be permanent, targeting enhancement. However, it seems to me that the government tends to minimize spending with education, using these evaluations as justification. In this sense, evaluation should be very complex and not simply a “is” or a “is not”. A series of variables should be taken into consideration. Government should invest in the most precarious areas. There should be an increase in investment, anyhow.” (History student)

12. State’s control and/ or regulation of universities In the Brazilian case, some students frequently identified institutional evaluation with the “Provão”, and they were politically sensible to one of its more notorious consequences: publication of the ranking results of degrees and institutions. “In my opinion, evaluation is the State’s form of control of what is being done in the degrees; they evaluate according to their own criteria and how it is the most convenient. I also believe they need to tell the people in charge of the country what our capacities are and to whom we can work for.” (Education student) “The current evaluation type is not the most appropriate as the “Provão” is the key to MEC’s[1] educational project, since the government has been taking upon the role of the manager of institutions, threatening to close low quality public institutions, even if the responsible for that is the government itself.” (Chemistry student) “In my opinion, the MEC’s “Provão” is valid, because at least the Education Ministry has a way to evaluate the private federal universities”. (Veterinary student) “I think this is positive in order to maintain a standard of educational quality of teaching/ learning, which has lately been forgotten by our government.” (Education student)

13. Evaluation legitimacy Evaluation brought improvements to the universities’ activities, i) Students (wider concern with students; training for life; information about degrees; promotion of success); ii) Teachers (improvement of teaching performance); iii) Curricula (curricula changes, adaptation of education to the job markets); iv) Education structuring (demand and quality, teaching methods, teaching efficiency); v) University procedures (creation of new degrees, image of the university, service improvement, influence on the decision making). Doubts are also cast on the method of developing the evaluation procedures. Even though evaluation is thought to be an important diagnosis on the quality of teaching (identifying failures, the strong and the weak points), some students mention the fact that it is not well linked with action, often not producing any practical effects: “I believe that besides students, teachers should also be evaluated, since in most cases the senior students have already job prospects and sabotage the “Provão”. I think the “Provão” is not valid. (Tourism student)

14. Institutional policies and aims of higher education Evaluation must emerge as a principle politically assumed by the universities, giving voice to the majority as it should also be a way to inject optimism in the institutions, stimulating them to invest widely in the improvement of their performance: “University evaluation should be the main goal of university policies. Politics is better when the voice of the majority is heard. Permanent improvement only exists with permanent auto-criticism, and the opposite is also true. In order to develop, we must think together! Union makes strength! Let there be more evaluations and let political decision be more the result of all the voices combined.” (Mathematics student) The universities’ social responsibilities, evaluation would be useful to level universities to societies’ aspirations and needs: “It is important [evaluation] because it is through evaluation that it will be possible to find out if universities are fulfilling their role in the face of the community.” (Biology student) “Universities should be evaluated, because only then we will know if the degrees are able to correspond to the markets.” (Architecture and Urbanism student)

15. Evaluation and institutional comparison by state examination of study diplomas Besides the fact of recognizing institutional evaluation as a way of diagnose of the real university conditions, students are also aware that it may show the discrepancies existent among their study degrees, the rankings (in Brazil) resultant from this sort of evaluation (Provão) and that supposedly, if you take the managerialist approach, would serve information goals  knowledge of the comparative situation of universities as a way of facilitating their choices  and competition  “quality” through competition stigmatizes undoubtedly the future professionals, whose degrees have had a less favourable assessment. “It is important to undertake a periodical evaluation in what concerns our education system, in order to improve it. I just do not agree with the way the broadcasting of the evaluation results is being done. The mentors of this assessment disregard the fact that if their degree is graded a C or a D they will not be able to compete in the job market with one that has had an A. What evaluators should be taking into consideration is that the dissemination of these results shoud not be done in this way, since they are prematurely excluding possible great professionals that were just unfortunate to study in a college of few resources, but that individually show great potential.” (Agronomy student)

16. Evaluation and institutional comparison by state examination of study diplomas “(…) the only good thing for the universities when they have good results in the “Provão” is that it brings status. Now this exam can end the great concepts within the university. Besides the “Provão” grade, which is how students are tested, there is also the evaluation of teachers and university dependencies. Concepts from A to E are offered and then you find an average. A university is not just dependencies; it is also its students, professors and students that learn. So, I don’t agree with this system that may have a high concept for a university that has a good structure, but without qualified teaching.” (Odontology student) “It is of high importance to show people that believe federal universities lack quality. «Provão» was created to weaken public education, and in the end it is strengthening it.” (Mechanical Engineering student)

17. Evaluation fragmentation University evaluation is fragmented, emphasizing just a few of its institutional activities and sectors. Evaluation does not show the whole, university. “(…) it is a sort of assessment that values quantity and not the quality of teaching. It is not a continuous evaluation that aims for development (…)” (Educations student) “University evaluation is only valid when a global assessment of all university’s sectors is made and not just the students.” (Chemistry student)

18. Conclusions Students’ perception on evaluation covers a vast area that can be structured with reference to various realities: government policies, the system’s operation, institutional policies, curricula and the education’s pedagogic organization. The approach on evaluation is, most of the times, polarized between positive and negative extremes, frequently assuming hybrid configurations. This phenomenon clearly demonstrates that it is not exactly a reproduction of the various “external” approaches given to the students: political, institutional and educational. It emerges with original items that integrate critical elements and true reorientation proposals to the creation and development of institutional evaluation in the universities. These perceptions may be valid as a reality of the institutional evaluation itself. It is not an objective reality, but common subjective views that emerge as a shared reality: through the evaluation of universities, students recognize and share somewhat of their own experiences and of the established institutional, educational and political conditions, but are also able to act upon them.