User-centred evaluation of digital repositories: reUSE experiences

1. User-centred evaluation of digital repositories: reUSE experiences LIDA Conference, Dubrovnik 30. May – 3. June 2005 by Mateja Šmid, Darko Majcenovič, Špela Zupanc, Alenka Kavčič Čolić

2. Project reUSE: reuse digital master files of printed materials! Financial source: EU eContent (nr. 11173) Partners: 9 partners University and national libraries from 4 different countries: Austria, Estonia, Germany, and Slovenia. Duration of the project: June 2004 - June 2006 Introduction INTRODUCTION

3. Introduction REPOSITORY IMPLEMENTERS • UBER – Humboldt University Berlin repository • ONB – Austrian national library • ALO – Austrian literature online • NLE – national library of Estonia Each implementers will establish its own digital trusted repository with its own restrictions, services and policies.

4. Objectives TRUSTED DIGITAL REPOSITORIES • Repository = organization that intends to maintain information for access and use. • Trusted repository = has the mission to provide reliable long-term access of managed digital resources to its designated community, now and in the future.

5. To: build useful systems systems optimatization improve services to users measure progress enhance the role of digital repositories OVERALL TO MAKE USER-CENTRED REPOSITORIES PURPOSE OF reUSE EVALUATION Evaluation

6. Evaluation approach Methodology: multi-atribute system and SWOT analysis focused on: Organizational aspects Technical aspects User perspective We will use interviews and questionnaires to gain a quantitative and qualitative comprehensive study. EVALUATION FRAMEWORK Evaluation

7. External and internalcontext in which the repositories were developed and their main functions in it. It emphasizes the concept of trustfulness through: The organizational strategy which defines the goals and mission of the repository, The stakeholders interests in its maintaining and financing, Favourable organizational culture to its development, The long-term preservation strategy and policy. ORGANIZATIONAL ASPECTS Evaluation

8. Technical specifications which can show us to what extent the repositories satisfy users requirements and needs. Several subgroups will be evaluated: General technical information OAIS compliance Technical and procedural suitability, security and procedural accountabillity User friendliness and services TECHNICAL ASPECTS OF THE REPOSITORIES Evaluation

9. Digital libraries are useful when are in harmony (Bishop, Van House, Butterfield: Digital library use, 2003) Evaluation The documents The users The DLs are seen as: • sociotechnical system • networks of technology • information artifacts • people and practices interacting with the larger world of work and society The system

10. User survey Happy user is the main goal. The fact that the user predetermine the collection should be highly considered. It is imporant WHO WILL ACCESS and for WHAT PURPOSE.

11. Purpose: to evaluate repositories from user perspective Form: questionnaire is in national language one sheet in paper form Based on: ISO 9241 Usability standard ISO 13407Human-centred design processes for interactive systems some specific questiones for reUSE repositories … User survey USER PERSPECTIVE • ADDITIONALY • other methods such as simple observation and logs and statistics will be carried out (depands on certain institution)

12. User survey End-user questionnaire consists of 2 parts:1. general (demographic) information2. questiones

13. DIFFERENT SET OF QUESTIONES: content user’s satisfaction with material in the digital repository recognition understandability of the interface performance and efficiency efficiency of the repository personal and subjective perception personal view and satisfaction with the repository error tolerance possible effects on further search User survey

14. 2 periods of end user’s evaluation. June 2005 after repositories are set up September 2005 after strong dissemination will be done To get stable and representative sampling  around 100 questionnaires per repository will be needed. Every 10th visitor will be asked:Whether he/she has ever used the repository? If yes, he/she is pleased to fill in the questionnaire for end-user. If no, the person is pleased to fill in short questionnaire for non-user. Perception/opinion of non-users is also important. User survey Conclusion REALIZATION OF THE SURVEY

15. Different set of questiones will give us several answers. Important feedback from end-users and non-users as well. implementers might redesign certain part(s) of repository if it is necessary clarification of what kind of content is expected by users Based on those answers we will be able to compose a broader picture about user’s perspective on certain repository. Usage habits, user’s wish, a need for training,… answers on such questiones will be given. … Cultural differencess might also surface (different backgrounds of national/university libraries, different participating countries…) Usersurvey EXPECTED RESULTS

16. At the end of the evaluation a comprehensive SWOT analysis for each implemented repository will be made. The evaluation should have the retroactive effect on the awareness and further dissemination valuable input for the implementers to improve the repositories if necessary to improve services to users CONCLUSION Conclusion

17. THANK YOU FOR ATTENTION! Mateja Šmid (msmid@ikpir.fgg.uni-lj.si) Darko Majcenović (darko.majcenovic@nuk.uni-lj.si)