Peer review in further education colleges : the Merseyside experience

1. Peer review in further education colleges : the Merseyside experience A workshop by: Steve Cropper, Senior Librarian, Wirral Metropolitan College and Andrew Eynon, Library Resource Manager, Coleg Llandrillo Andrew: Opening slide, welcome participants, introduce ourselvesAndrew: Opening slide, welcome participants, introduce ourselves

2. The Circle of Merseyside College Librarians: Founded 27 years ago Members from 6 major FE providers on Merseyside and two 6th form colleges Strong links with North Wales, Lancashire and Cheshire Meetings each term Annual Benchmarking of key performance indicators Andrew: A brief bit of background about the Merseyside Circle, which we are representing here today. The circle is a very active group that was founded 27 years ago (Click) by Jeff Cooper. It represents all the major FE providers on Merseyside and welcomes colleagues in the 6th form college sector (click) It has excellent relations with colleagues in North Wales, Lancashire and Cheshire and colleagues from colleges outside the immediate area have all attended meetings and participated in our activities. (click) It has well-attended and lively meetings every term at a different venue each time (click) and has tried to ensure that it is not merely a talking shop. To this end: (click) We conduct an annual benchmarking exercise on 19 key performance indicators to provide us with comparators relevant to our own area of operations. And, the reason we are here today, (click to next slide) Andrew: A brief bit of background about the Merseyside Circle, which we are representing here today. The circle is a very active group that was founded 27 years ago (Click) by Jeff Cooper. It represents all the major FE providers on Merseyside and welcomes colleagues in the 6th form college sector (click) It has excellent relations with colleagues in North Wales, Lancashire and Cheshire and colleagues from colleges outside the immediate area have all attended meetings and participated in our activities. (click) It has well-attended and lively meetings every term at a different venue each time (click) and has tried to ensure that it is not merely a talking shop. To this end: (click) We conduct an annual benchmarking exercise on 19 key performance indicators to provide us with comparators relevant to our own area of operations. And, the reason we are here today, (click to next slide)

3. The Peer Evaluation Scheme Has carried out 20 evaluations to date In Lancashire: Blackpool and the Fylde College and Runshaw College In Cheshire: Widnes 6th Form College and West Cheshire College In North Wales: Denbigh and Rhyl Community Colleges and Coleg Llandrillo Lancashire and North Wales have provided Evaluators for the scheme Andrew: We run a well-established peer evaluation scheme which has (click) carried out twenty such evaluations to date. We would like to emphasise that although the scheme is run under the auspices of the Merseyside Circle, we are not constrained by geography. To this end, we have carried out peer evaluations in (click) Lancashire (click) Cheshire and (click) North Wales. Not only have these colleges invited us in for peer evaluations, they have also (click) provided evaluators. This type of reciprocal professional assistance, without charge, is one of the key factors underpinning what we do. Andrew: We run a well-established peer evaluation scheme which has (click) carried out twenty such evaluations to date. We would like to emphasise that although the scheme is run under the auspices of the Merseyside Circle, we are not constrained by geography. To this end, we have carried out peer evaluations in (click) Lancashire (click) Cheshire and (click) North Wales. Not only have these colleges invited us in for peer evaluations, they have also (click) provided evaluators. This type of reciprocal professional assistance, without charge, is one of the key factors underpinning what we do.

4. What this session is about: We are not here to sell you anything This is a workshop, not a lecture We intend to challenge you to bring forward your own ideas The learning objective is that you go away from here sufficiently interested in peer evaluation to want to try it out for yourselves Andrew: Some things we would like to emphasise (click) We are emphatically not here to sell you anything (click) This is a workshop, not a lecture (click) We want you to think and give of your own ideas (click) And all we hope for out of today is that you will think about what we have said, and go away and try it for yourselves Andrew: Some things we would like to emphasise (click) We are emphatically not here to sell you anything (click) This is a workshop, not a lecture (click) We want you to think and give of your own ideas (click) And all we hope for out of today is that you will think about what we have said, and go away and try it for yourselves

5. How we got started: A brief introduction to how peer evaluation began on Merseyside Steve: OK, a brief introduction, just to set the context and let you know where we are coming fromSteve: OK, a brief introduction, just to set the context and let you know where we are coming from

6. The climate that gave rise to peer evaluation: Then, as now, many college libraries struggling with inadequate resources and management indifference An inspection regime for which we had high hopes delivered very little Unqualified inspectors passing judgement on libraries Process too easy to subvert Steve: A bit of historical context, so that any of you not around in the 1990s can gat an impression of the climate in which our ideas were formed. (click) We would not wish to over-emphasise the negatives. There have always been FE providers with well-resourced library services, but as the recent CoFHE survey proves, there are many other library services out there struggling with inadequate staffing, budgets and accommodation, with indifferent managements who choose to ignore the contribution they make to the learning process. We were both working at that time in a college where the library was widely predicted to disappear altogether in favour of IT. On the final year before the new management team came in, one of our site libraries actually made more in income than it received from the college budget. That is how bad it actually was. (click) We all had high hopes of the FEFC inspection regime when it began, but when we saw the first reports, in which libraries were usually relegated to one ill-informed, vague paragraph, and heard accounts from colleagues of how they had been inspected, we realised our hopes were not going to be realised. (click) The FEFC at that time had only one qualified librarian on the whole national inspection team. When pressed on this, their response was that their inspectorate was made up of ex-principals and local authority education managers with wide experience of these matters. One wonders if the government had decided to inspect the teaching profession using former tax inspectors, what the reaction of the teaching profession would have been! (click) And, from our own experience, we quickly learned how very easy it was for college managers to subvert the inspection process to their own ends, by editing documentation and ensuring that inspectors were well-chaperoned, and kept away from Librarians.Steve: A bit of historical context, so that any of you not around in the 1990s can gat an impression of the climate in which our ideas were formed. (click) We would not wish to over-emphasise the negatives. There have always been FE providers with well-resourced library services, but as the recent CoFHE survey proves, there are many other library services out there struggling with inadequate staffing, budgets and accommodation, with indifferent managements who choose to ignore the contribution they make to the learning process. We were both working at that time in a college where the library was widely predicted to disappear altogether in favour of IT. On the final year before the new management team came in, one of our site libraries actually made more in income than it received from the college budget. That is how bad it actually was. (click) We all had high hopes of the FEFC inspection regime when it began, but when we saw the first reports, in which libraries were usually relegated to one ill-informed, vague paragraph, and heard accounts from colleagues of how they had been inspected, we realised our hopes were not going to be realised. (click) The FEFC at that time had only one qualified librarian on the whole national inspection team. When pressed on this, their response was that their inspectorate was made up of ex-principals and local authority education managers with wide experience of these matters. One wonders if the government had decided to inspect the teaching profession using former tax inspectors, what the reaction of the teaching profession would have been! (click) And, from our own experience, we quickly learned how very easy it was for college managers to subvert the inspection process to their own ends, by editing documentation and ensuring that inspectors were well-chaperoned, and kept away from Librarians.

7. What we sought to achieve: To put libraries back on the inspection agenda, as central to the learning process To counterbalance ill-informed judgements with qualified, professional assessment To present inspectors and managers with independent, knowledgeable reports from well-qualified volunteer inspectors that would be hard to dispute To place those reports into the public domain Steve: This is what we set out to do (click) To get libraries back on the inspection agenda as far as we could (click) To bring tour collective reserves of knowledge and experience in library work into the process, and challenge ill-informed, unqualified opinion and cursory inspections of library facilities. (click) By putting detailed, impartial reports into the inspection documentation base, we knew that it was probable that some inspectors would welcome these as an aid to the inspection, especially if they had little knowledge of libraries. The fact that those reports had been prepared by people with a demonstrable track record in managing college libraries, for which they had received no payment, made them hard to dispute. (click) The public domain element was essential to ensure transparency and fairness, and had the added bonus of concentrating the minds our managers. Steve: This is what we set out to do (click) To get libraries back on the inspection agenda as far as we could (click) To bring tour collective reserves of knowledge and experience in library work into the process, and challenge ill-informed, unqualified opinion and cursory inspections of library facilities. (click) By putting detailed, impartial reports into the inspection documentation base, we knew that it was probable that some inspectors would welcome these as an aid to the inspection, especially if they had little knowledge of libraries. The fact that those reports had been prepared by people with a demonstrable track record in managing college libraries, for which they had received no payment, made them hard to dispute. (click) The public domain element was essential to ensure transparency and fairness, and had the added bonus of concentrating the minds our managers.

8. How we put the ideas into practice: Small working group set up consisting of representation from 4 member FE colleges and one 6th form college Draft framework prepared Emphasis on practical, real-world approach Keeping it free kept the scheme impartial, and achievable for any library Written up for CoFHE Bulletin Steve: (click) we were keen to secure the widest possible input into the design of the scheme, including the view from the 6th form colleges. (click) In practice, the draft framework came together very quickly. (click) From the beginning we sought a practical, real-world approach that focussed on the nitty-gritty issues that matter to college librarians (click) we wanted a framework that would be achievable for colleagues working in hard-pressed services on tiny budgets, and free from any suggestion of “buying” consultants’ opinions we wanted to hear. Keeping the scheme free contributes to its credibility, and allows anyone to use it. (click) We publicised the scheme, and subsequent updates, in the three articles we produced for the Bulletin that are in your bundles. Steve: (click) we were keen to secure the widest possible input into the design of the scheme, including the view from the 6th form colleges. (click) In practice, the draft framework came together very quickly. (click) From the beginning we sought a practical, real-world approach that focussed on the nitty-gritty issues that matter to college librarians (click) we wanted a framework that would be achievable for colleagues working in hard-pressed services on tiny budgets, and free from any suggestion of “buying” consultants’ opinions we wanted to hear. Keeping the scheme free contributes to its credibility, and allows anyone to use it. (click) We publicised the scheme, and subsequent updates, in the three articles we produced for the Bulletin that are in your bundles.

9. Refining the ideas further: Evaluation cycle brought in line with FEFC / ALI / OFSTED four-yearly inspection cycle Evaluation framework continually revised to reflect changes in inspection regime Future evaluations will include benchmarking against the new CoFHE guidelines Steve: The scheme has not stood still. We recognise that the climate it operates in is continually evolving, and we have continually refined it to reflect this. We have: (click) Brought it into synch with the ALI/OFSTED inspection cycle. It is now usual for members aware of a forthcoming inspection to ask for a Peer Evaluation to be done as part of their inspection preparation/ (click) The framework itself has been continually revised to stay abreast of changes in the inspection process (see: 2nd Bulletin article your bundle) (click) We are now looking beyond the framework to include some benchmarking against the new CoFHE guidelines, both to bring these to the notice of college managers, and to support the standards ourselves.Steve: The scheme has not stood still. We recognise that the climate it operates in is continually evolving, and we have continually refined it to reflect this. We have: (click) Brought it into synch with the ALI/OFSTED inspection cycle. It is now usual for members aware of a forthcoming inspection to ask for a Peer Evaluation to be done as part of their inspection preparation/ (click) The framework itself has been continually revised to stay abreast of changes in the inspection process (see: 2nd Bulletin article your bundle) (click) We are now looking beyond the framework to include some benchmarking against the new CoFHE guidelines, both to bring these to the notice of college managers, and to support the standards ourselves.

10. First group activity: You are designing a peer evaluation framework of your own. What elements would you include in it, and (briefly) how would you assess each of them? Andrew: Right, we said this was a workshop, so now it is your turn. You have ten minutes to produce a list of the elements you would focus on in an evaluation scheme of your own design, and we would like you to give a brief idea, in note form only, of what you would look at under each element. Good luck!Andrew: Right, we said this was a workshop, so now it is your turn. You have ten minutes to produce a list of the elements you would focus on in an evaluation scheme of your own design, and we would like you to give a brief idea, in note form only, of what you would look at under each element. Good luck!

11. The nitty gritty bit….. Having produced a framework, how do we actually go about conducting a peer evaluation? Andrew: Let us now see how we would put your framework into practice on the ground. In this section we will ask some key questions, invite your thoughts, and then tell you how we did it. This by no means assumes that we know all the right answers!Andrew: Let us now see how we would put your framework into practice on the ground. In this section we will ask some key questions, invite your thoughts, and then tell you how we did it. This by no means assumes that we know all the right answers!

12. First: select your evaluators Evaluations are initiated by a formal request to a Circle Meeting Two volunteers are requested from among the membership who have no connection with the college to be inspected. We also call upon colleagues from outside the immediate area with whom we have reciprocal links Andrew: How would you go about selecting your evaluators to conduct a peer evaluation? Answers from floor, then click through each of the criteria.Andrew: How would you go about selecting your evaluators to conduct a peer evaluation? Answers from floor, then click through each of the criteria.

13. What to do in advance of the visit: Request advance documentation eg. SAR, Business plan Draft questions arising out of the advance documentation Set a realistic timetable for the day Arrive at a division of labour Make necessary travel arrangements Andrew: What would you think it important to do before the actual visit? Answers from the floor then (click): Request relevant advance documentation, notably SAR, Business Plan and other core documents. (Ensure you are not “mail bombed” with unnecessary documents). (click) Study these, noting inconsistencies and questions you would like to ask to clarify points (click) Set a realistic timetable for the day, particularly if looking at multi-site operations (click) Divide the evaluation schedule between you, in a fair division of labour that recognises any areas of special expertise each of you may have (click) Make proper travel arrangements so that there are no unnecessary delays on what will be, we promise you, a busy day!Andrew: What would you think it important to do before the actual visit? Answers from the floor then (click): Request relevant advance documentation, notably SAR, Business Plan and other core documents. (Ensure you are not “mail bombed” with unnecessary documents). (click) Study these, noting inconsistencies and questions you would like to ask to clarify points (click) Set a realistic timetable for the day, particularly if looking at multi-site operations (click) Divide the evaluation schedule between you, in a fair division of labour that recognises any areas of special expertise each of you may have (click) Make proper travel arrangements so that there are no unnecessary delays on what will be, we promise you, a busy day!

14. On the day: Ensure you speak to a range of staff and to students Strike a balance between interviews and legwork. Go and see for yourself and be hands-on Stick to your part of the brief and ask the right questions Link to benchmarks Andrew: What will you actually do when you get there? Responses from the floor and then: (click) Ensure you speak not only to managers, but also front-line staff and students (Click) Don’t spend the whole day in interviews being given the librarian’s perspective. Get out on to the library floor, use your eyes, handle stock, bring your experience to bear. (click) Resist the temptation to stray from your brief unless you spot something unusual. Ask probing questions based on a thorough study of the documentation and pay attention to how claimed figures have been calculated. (click) Link your findings to national and local benchmarks. Andrew: What will you actually do when you get there? Responses from the floor and then: (click) Ensure you speak not only to managers, but also front-line staff and students (Click) Don’t spend the whole day in interviews being given the librarian’s perspective. Get out on to the library floor, use your eyes, handle stock, bring your experience to bear. (click) Resist the temptation to stray from your brief unless you spot something unusual. Ask probing questions based on a thorough study of the documentation and pay attention to how claimed figures have been calculated. (click) Link your findings to national and local benchmarks.

15. Writing Up: This is time-consuming! Allow adequate time Be prepared to re-check points of fact Be helpful to the Librarian, but always tell the truth Andrew: What is it important for you to do when writing up the draft report? Answers from the floor then: (click) Writing up the draft report can be very time-consuming. You will be weighing your words with care, drafting and re-drafting until you achieve the right emphasis. Allow adequate time for this activity. (click) Do not worry about ringing the Librarian or any other member of the team back to check points of fact. They would rather be mildly inconvenienced than see factual inaccuracies in the finished report. (click) Probably the most difficult bit: striking the right balance between being helpful and supportive to the Librarian, but always tell the truth. For the scheme to have integrity, it cannot be allowed to become a resource-bidding exercise or a vehicle for personal hobby-horses. Remain objective, and let the facts speak for themselves. At the same time, do not be afraid to be critical, and praise good practice. This means a lot to the front-line staff.Andrew: What is it important for you to do when writing up the draft report? Answers from the floor then: (click) Writing up the draft report can be very time-consuming. You will be weighing your words with care, drafting and re-drafting until you achieve the right emphasis. Allow adequate time for this activity. (click) Do not worry about ringing the Librarian or any other member of the team back to check points of fact. They would rather be mildly inconvenienced than see factual inaccuracies in the finished report. (click) Probably the most difficult bit: striking the right balance between being helpful and supportive to the Librarian, but always tell the truth. For the scheme to have integrity, it cannot be allowed to become a resource-bidding exercise or a vehicle for personal hobby-horses. Remain objective, and let the facts speak for themselves. At the same time, do not be afraid to be critical, and praise good practice. This means a lot to the front-line staff.

16. FEEDBACK: Completed draft to the Librarian Opportunity to correct matters of fact Andrew: What would you have in place as a final checking and feedback process? Answers from floor then click through the aboveAndrew: What would you have in place as a final checking and feedback process? Answers from floor then click through the above

17. Final Report: Remember, it is a public domain document Remember the importance of an executive summary Copies to: The Librarian The Quality manager College Principal Central Archive Andrew: What is important to remember about the final version of the report, and who do you send it to?Andrew: What is important to remember about the final version of the report, and who do you send it to?

18. What those who have been through the process think: “The peer evaluation provides an assessment of your service by fellow professionals greater in scope than other reviews or inspections, providing a substantial report useful for QA, external validations and service reviews” (Patrick Cox, Learning Resources Manager, Liverpool Community College) “ A valuable and encouraging experience, with confidence that the assessors and the process are informed and supportive” (Catriona Martin, Librarian, West Cheshire College) “The library has been inspected by those qualified to pass judgement and I welcomed the valuable report which has led to improvements” (Cathy Bennett, Librarian, St Helens College) The Peer Evaluation report is a quality document produced is a quality document by fellow professionals whose judgement I respect (Sue Haydock, Librarian, Southport College) Steve: Here are some comments from those who have been through peer evaluation under the scheme.Steve: Here are some comments from those who have been through peer evaluation under the scheme.

19. In conclusion, peer evaluation: Was a response to in inadequate inspection regime that did little to drive up quality in hard-pressed libraries in the FE sector. Nothing much has changed, and it is as valid today as it was then. Seeks to re-assert the role of qualified library and information professionals and challenge inadequate inspections An affordable, practical and realistic means of assessing the strengths and weaknesses of college library services Is not a resource-bidding exercise Steve: The perorationSteve: The peroration

20. Two final thoughts… This conference is about justifying your library services. Peer evaluation is about being honest enough to open up your service to a searching, impartial review process, which gives the end product added credibility. Steve: The link to conference themeSteve: The link to conference theme

21. Further reading/questions Peer evaluation : redressing the balance? CoFHE Bulletin, Issue 75, Summer 1995 Peer group evaluation revisited, CoFHE Bulletin, Issue 85, Autumn 1998 Peer Evaluation : the Merseyside Experience. CoFHE Bulletin, Issue 99, Spring 2003 Resources and Services Supporting Learning : a Service Quality Development Toolkit. SFEFC, November 2002, full text available at:  http://www.slainte.org.uk/Files/pdf/FEnet/toolkit03.pdf This presentation is available in full at: http://www.wmc.ac.uk/2004/wmc_libraries/papers.html Steve: Thank the audience, leave this slide up for further reading and offer to take questions. Remember to invite delegates to look at the sample reports, and make sure they do not remove them from the room.Steve: Thank the audience, leave this slide up for further reading and offer to take questions. Remember to invite delegates to look at the sample reports, and make sure they do not remove them from the room.