Design and Evaluation of an Adaptive Incentive Mechanism for Sustained Educational Communities

1. Design and Evaluation of an Adaptive Incentive Mechanism for Sustained Educational Communities Ran Cheng and Julita Vassileva User Modeling & User-Adapted Interaction Presented by Rosta Farzan PAWS Group Meeting April 13, 2007

2. Problem Insufficient user participation in online communities Provide incentives to encourage participation Too much participation especially of low-quality

3. Goal Proposing an incentive mechanism regulating quantity and quality of user contribution and ensuring a sustainable user participation

4. Outline • Related work • Collaborative quality evaluation mechanism • Brief introduction to Comtella • Proposed mechanism • Implementation of proposed mechanism in Comtella • Evaluation • Discussion

5. Collaborative Quality Evaluation Mechanism • Active and sustained user participation while avinding information overload requires quality control • Decentralized moderation • Real world example • Measuring the quality of journals or papers by counting the time they were cited • Online communities • Counting the number of clicks on each item • Problem • Not always a click means a positive attitude • Explicit user rating • E.g. rating process in Slashdot • Problem – “rich get richer” • Unfair score for items with insufficient attention • Lower initial rating • Contributed late in discussion

6. Requirement of Incentives • Time based community need • New contributions in the early period of discussion • Ratings of the contribution when many contributions are collected • Different users have different contribution patterns • Encourage higher participation for users who contributed few high-quality resources • Inhibit contributions from users who contributed many low-quality resources • Overall community need

7. Comtella • Developed at University of Saskatchewan • Online community for sharing URLs of class-related web-resources (bookmarks)

8. Proposed Incentive Mechanism • Mechanism encouraging users to rate resources • Adaptive reward mechanism

9. Encouraging Users to Rate • All users can rate others’ contribution • Each user receives a limited number of rating points to give out • Users with higher level membership receives more points • Initial rating of zero independent of level of resource provider • Ensuring all contributions have equal chance to be read and rated • Summative rating • Incentive: Virtual currency (c-points) • Limited initial c-points • Awarding users for rating resources • Depending on user’s reputation • Can be invested to promote the initial visibility

10. Sorted by c-points Summative Ratings Report duplicate, broken, or special permission links

11. Adaptive Reward Mechanism • Hierarchical membership • Adapt rewards for different forms of participation • Quality of users’ participation so far • Community need • Personalized motivational message • Stating specific performance goals • Calculating adaptive rewards • Community Model • Individual Model

12. Community Model • Expected # of total contributions • Set by community admin • Community Reward Factor • Usefulness of new resources

13. Individual Model • Contribution reputation • Sharing • Average summative rating of all shared resources • Rating • Quality ~ Difference between the specific rating and the average of all ratings the resource gets eventually • Smaller difference  Higher quality • Current membership level

14. Individual Model • Expected # of resource contribution • Reward Factor

15. Adaptive Reward Mechanism

16. Implementation of proposed mechanism in Comtella

17. Evaluation • Questions to be answered • Will the users in the test group rate articles more actively? • How well will the summative ratings reflect the real quality of the articles? • Will the users tend to share resources earlier in the week? • Will the actual number of contributions be close to the desired one? • Will the users share the number of articles that is expected from them? • Will the users contribute a higher percentage of high-quality articles? • Will there be information overload?

18. Participant • Class on Ethics and Information Technology • Share web-articles related to the topic of each week • 31 4th year undergraduate students • Test group: 15 • Control group: 16 • Randomly assigned while controlling gender and nationality

19. Result • Users in the test group were more active in rating articles • The articles with higher ratings were more likely to be chosen by users to summarize • The users in the test group were more satisfied with the summative ratings received by their articles • The users in the test group tended to share resources earlier in the week • No big difference between total number of shared articles across the two groups • In both groups, the users’ attitude towards the quality of the articles were generally neutral • No information overload problem – the overall contribution did not exceed the community need • The users in the test group were more active in terms of logging on the system and reading articles – sustainability

20. Discussion/Limitations • Choosing the Parameters for the mechanism • Expected sum of contributions for each topic • Threshold for reputation values • Community reward function • Narrow scale of rating (+1 or -1) • Less cognitive load • No option to rate different aspect of a paper • Popularity of topic, originality, interestingness • Measure of quality • Average rating means average taste rules • In educational context tends to be superficial, easy to read articles

21. Discussion/Connection to Our Work • Trying personalized motivational message in CourseAgent or CoPE • Good evaluation questions • “Some people are easy to be motivated by glory and recognition” • Is there any cognitive tool to measure this? • Compare Comtella with CoPE • CoPE has the option to write summary and read summary written by others • More personal benefit