Cristian Cechinel, Salvador Sánchez-Alonso, and Miguel-Ángel Sicilia

1. Exploratory analysis of the correlations between peer-reviewers and users ratings on MERLOT repository Cristian Cechinel, Salvador Sánchez-Alonso, and Miguel-Ángel Sicilia

2. Objectives • Analyzing the existence of associations between the ratings given by these peer-reviewers and users in MERLOT • Discovering whether or not they diverge about the quality assessment • Initially exploring the usefulness of these two complementary evaluations towards the assurance of quality inside the repository

3. Introduction • LORs are searching for mechanisms to evaluate their catalogued/stored materials • Most of the existing LORs harness the features of such social environments through the adoption of strategies for the establishment of quality that rely on the impressions of usage and evaluations given by regular users and experts that are members of the repository community • Distinct LORs use distinct solutions regarding this subject

4. RepositoriesSolutions • In E-Lera users can create reviews using LORI, and can add resources to their personal bookmarks. Materials can be searched by their ratings, and by their popularity • In Connexions resources are arranged by a system called lenses according to evaluations provided by individuals and organizations. Materials can be searched by ratings given by users, and by their number of access over the time • In MERLOT resources are evaluated by users and peer-reviewers. Users can add resources to their Personal Collections

5. Peculiarities in the MERLOT case • Existing of two well defined and different groups of people (public and experts) which possibly come from distinct backgrounds and may have divergent opinions with respect to quality. • Complementary Approach

6. Differences between Peer Reviewing and Public Reviewing Table on the next slide.

9. Reviews and Ratings in MERLOT • Editorial boards of MERLOT decide on the process of selecting materials that are worth of reviewing, and the assigned materials are then independently peer-reviewed by their members according to three main criteria: 1) Quality of Content, 2) Potential Effective as a Teaching Tool, and 3) Ease of use.

10. Reviews and Ratings in MERLOT • After peer-reviewers report their evaluations, the editorial board chief-editor composes a one single report and publishes it in the repository with the authorization of the authors

11. Reviews and Ratings in MERLOT • In addition to peer-review evaluations, MERLOT also allows the registered members of the community to provide comments and ratings about the materials, complementing its strategy of evaluation with an alternative and more informal mechanism

12. Reviews and Ratings in MERLOT • The ratings of both (users and peer-reviewers) range from 1 to 5 (with 5 as the best rating). • The use of the same rating scales for both kinds of evaluations allows for direct contrast of the groups in order to evaluate possible correlations and the existence or not of disagreement between them.

13. Data Sample and Method • Data from a total of 20.506 learning objects was gathered (September 2009) through a web crawler developed ad hoc for that purpose. • Most of the resources did not have any peer-review or user rating, and from the total amount of collected data, only 3,38% presented at least one peer-reviewer and one user rating at the same time

14. Data Sample and Method • PRR – Peer Reviewed • UR – User Reviewed

15. Results and Discussion • A non-parametric analysis was performed using the Spearman’s rank correlation (rs) to evaluate whether or not there is association between the ratings of the two groups • In order to observe potential differences in ratings according the background of the evaluators, we split the samples in categories of disciplines and also performed the same analysis for each one of them

16. Results and Discussion • The disciplines of Arts, Business, and Mathematics and Statistics did not present any association between the ratings given by users and peer-reviewers • The ratings are associated for the overall sample, as well as for the disciplines of Education, Humanities, Science and Technology and Social Sciences.

17. Results and Discussion

18. Results and Discussion • Even though these associations exist, they are not too strong, as their coefficients of correlation are relatively small. • A strong correlation between the ratings could be suggested by a formation of a diagonal line, or the agglomeration of dots in some region of the matrix, for instance.

19. Results and Discussion

20. Conclusions • Both communities of evaluators in MERLOT are communicating different views regarding the quality of the learning objects refereed in the repository. • Peer-review and public-review approaches can be adopted in learning objects repositories as complementary strategies of evaluation

21. Conclusions • As the community of members and their ratings in MERLOT are naturally growing much more than the community of peer-reviewers and their evaluations, it becomes necessary to invest attention in exploring the inherent potentialities of this expanding community

22. Acknowledgments • The results presented in this paper have been supported by the Spanish Ministry of Science and Innovation through project MAPSEL, code TIN2009-14164-C04-01.

23. Contacts • Cristian Cechinel • contato@cristiancechinel.pro.br • Salvador Sánchez-Alonso • salvador.sanchez@uah.es • Miguel-Ángel Sicilia • msicilia@uah.es