How to remove an o ut layer tester

1. How to remove an outlayer tester Lucjan Janowski Faculty of Electrical Engineering, Automatics, Computer Science and ElectronicsDepartment of Telecommunications

2. Agenda • Can a tester be an out layer? • The detecting philosophy • Latent variables • Rasch model • WinSteps • The final decision • Conclusion

3. Can a tester be an out layer?

4. What would we like to model? • Whydowe use testers? • A tester represents human perception that is difficult to model • People are different and so are our users/clients. Our goal is to take suchdifferenceinto account • Some of us are critical and others are uncritical • A tester can be tired or not focused enough and therefore his/her answer can be random

5. A tired tester problem • A user can be tired too. Should we remove all tiredtesters? • Can a tester score randomly? What are the consequences? • Note that detecting that a tester scores a picture differently than the average scoredoes not mean that it is a random tester • We have to be very careful with testers removal since our goal is to build a model of the average user not the proper user

6. Why are some scores different? • Different effects can affect tester’s judgement differently (e.g. motion intensity, color, etc.) • Testers have different experience (e.g. watching mainly youtube or films on a DVD set) • Each of us is more or less critic to anything that he/she judges • The words describing the opinion scale can be understood differently (in Poland OKis good in England OKis fair)

7. What can we do? • We have to detect random scores • A tester that scores randomly often should be removed from the model building • An answer that differs from the average score is not necessarily a random one therefore we have to consider the average score but corrected by a tester individualism • We need a mathematic model of a user behavior that takes into account those properties

8. Latent variable

9. Latent variable

10. Latent variable manifestation

11. An example

12. Non extreme values testers

13. Wide range for 10 and 1

14. Critical tester

15. Are the answers random?

16. Rasch model • We assume that a latent variable is the variable that is really scored by testers • We assume that the opinion score probability is a logit function of the model parameters • The function has parameters describing: • a tester “criticism” factor • a film/picture/… quality • an average threshold value for particular score

17. Rasch model equation • n the tester number • i the object number (what is scored) • x the opinion score value (1-5, 0-10, …)

20. Rasch model • We assume that Rasch model is correct and the data that do not fit this model are incorrect [sic] • Note that without any assumption we are not able to detect randomly scoring testers

21. OMS (Outfit Mean Square) • Knowing the model probability and the user answer we can estimate howfar is a tester from the model • A tester’s accuracy or quality is based on the OMS (Outfit Mean Square) • Rasch model can be computed by WinSteps software (http://www.winsteps.com/) • The OMS can be interpreted on the basis of heuristically obtained ranges

22. Results interpretation

23. An example results

24. Rasch model disadvantages • It is more accurate for more data. It is difficult to have lots of results since the tests are expensive • Not all type of correct testers’ behavior can be modeled • The algorithms are not implemented in Matlab therefore it is difficult to implement it in an automatic analysis made in Matlab

25. Conclusion • A tester’s answers make it possible to model human perception but not all his/her answers are correct • Out layers should be removed • Rasch model helps to detect not relevant testers • The final decision should be checked since not all correct behaviors can be modeledby Rasch model