Writing & Reading Workshop

2. Writing & Reading Workshop

3. Today´s programme • Workshop on: • writing experiment reports • how to read research papers • – expect to be challenged ... • Will not go through pensum in detail – focus on utilizing your acquired knowledge • Ask questions as we get to each section of the reports

4. Today´s programme • Exercise: • Writing a review of Law et al.: ”Understanding, Scoping and Defining User Experience: A Survey Approach” • Following the guidelines in Philip Fong: ”How to Read a CS Research Paper?” • We will do the first parts in steps throughout the day, the latter parts you will complete at home and hand in next week • Some points may be to answer because they require knowledge of the field of user experience – take your best shot or skip them.

5. Practical information • We have been approved (horray!) with breaking from 11.45 to 12.45 – same exercise room (4A58) • Re. the 28/9 – heard from 2 students – weren´t there 3??? • Today´s test if you read the curriculum: Handout by Graham Hole is exact copy of chapter 9 in the book – how many discoverd this? • Handouts for exercises on the course website

6. The game lecture series • Center for Computer Games Research Game Lecture Series • Runs on Thursdays at 15.00 around every 2 weeks in the semester • Talks of interest to you: • 24th of September: Janus Sørensen from IO Interactive on user testing • 19th November: Lennart Nacke on bio-physiological measures of user experience

7. Paper review - assignment • 1-2 page review following the recipe of Philip Fong • 1) Summary of the paper in your own words • 2) Evaluation of the work • Is the research problem significant? • Are the contributions significant? • Are the claims valid? • 3) Synthesis: summarise your thoughts on the paper

8. Writing scientific reports

9. Writing scientific reports • Reports are based on the idealised scientific journal article • Reports are divided into sections which provide a specific piece of information each • This standard format is used in virtually all the sciences (not humanities) and with minor modification, the industry • Being able to write reports is a key skill

10. Writing scientific reports • Purpose: • To communicate research findings to others in a clear, systematic and standardised way. • Clear: So we get it ... • Systematic and standardized: So we can quickly read the paper, know where things are

11. Sections • Sections of a report: • Title • Abstract (summary) • Introduction • Method: • Participants • Design • Procedure • Results • Discussion/Conclusion • References • Appendices 10-20% 50-70% 15-25%

12. Aims of a report • 1. "Why?". Why did I do this particular experiment? What did I expect to find out by doing it? This question is dealt with in the Introduction. • 2. "How?". How did I actually carry it out - what procedures and apparatus did I use? • This question is covered in the Method section. • 3."What?". What did I find? What were my results? This information is provided in the results section. • 4."So What?" What does it all mean? How do my results relate to previous research on the same topic, and what are their theoretical implications? What are my conclusions? These issues are all dealt with in the Discussion and Conclusion sections.

13. Other components • Title • Provides quick idea what the report is about • Abstract • Summary of the contents and main conclusions • Should keep the attention of the reader • References • Documents arguments in report and provides means for locating further knowledge

14. Title • Ask yourself: Do you want anyone to actuallyread the report? • If yes, use a title that catches people´s attention • If no, use a really long and boring title • Informative but succinct (10-12 words maximum) - e.g.: • “Gender differences in attitudes to "Bambi" amongst Navaho Indians” • “The effects of nationality and age on sun-bed claiming behaviour”

15. Report titles ...

16. Abstract • An abstract is a brief summary of the report • Typically 120-150 words Give reader: • (a) what you did: Quick overview of contents • (b) what you found: The main results • (c) what it means: Theoretical/practical implications • Typically easiest to write the last

17. Keywords • Typically 3-5 keywords of the report • Nominally provides the reader another quick idea about the contents + is used as search words in databases/browsers Consider • One thing is what the paper is about, another is what the current buzzwords in your field is • Ensure you know which keywords to use to make people find/not find your report in library/company databases

18. Executive Review • Used business reports and white papers, or experiment reports in the industry • The same thing as an abstract, but tuned to the specific audience/stakeholder • Design department • Management • Shareholders • Etc.

19. Introduction • Introduces the reader to the overall problem area first, then focuses on the specific problem under investigation • General problem introduction and statement (why?), e.g.: • - gaps in previous knowledge; • - methodological flaws in previous work. • Review of current work in the field • Discussion of any relevant literature and theories • Overview of the report contents • Brief outline of our experiment, and its possible outcomes and their implications. • Why are we bothering to do our study?

20. The METHOD section (dun-dun!)

21. Method • Describes how the experiments were run • Describes what data that were collected and how they were analyzed • Argues for why the approach is correct • Introduction + hypotheses • Design • Participants • Apparatus • Procedure

22. Method Provide enough detail that experiment can be replicated, but not any irrelevant information

23. Hypotheses • In HCI research papers, there will normally be specific hypotheses and research questions that are sought answered • These are normally – not always - described in the method section • In commercial HCI reports, often there are no hypotheses, but specific purposes with the test/experiment (sort of the same thing...)

24. Participants • Who took part in the experiment? • What were their characteristics? (demographics – age, gender, nationality, experience) • Volunteers or paid? • Randomly selected or not? • How were they allocated to the different conditions of the experiment?

25. Experiment conditions • Typically we have a control group and an experiment group • The experiment group we do something with – ”Will this pill turn people purple?” • The control group we do NOT submit to an experiment – they permit us to measure the effect of the experiment reliably

26. Materials (apparatus, equipment...) • Also called ”setup” in many HCI situations • Basically describes the equipment of the experiment + measurement instruments • Equipment: computers, watches, cameras, software etc. • Instruments: surveys, pulse watches etc. • Diagrams and photographs are often used

27. Procedure • Explain how the experiment was carried out • Usually pretty detailed with exact description of what was done to the participants, and what they had to do • Order of the tests being administered • Temporal frame: How long? When? Etc.

28. Results • An objective walkthrough of what results that have come out of the experiment • Do not interpret numbers here – just report them • Only critical raw data and summary statistics are included in the report • ”76% of people turned purple...” • The rest is in appendices

29. Results • Two overall parts in the results • 1) Descriptive statistics • These describe datasets • ”35 samples, mean weight 2 kgs, SD=0.2 kgs” • 2) Inferential statistics • These allow us to answer research questions • ”An ANOVA shoved that the iron meteors had statistically different weights than ice meteors”

30. Results • Descriptive statistics • Start by providing details for each group or experiment condition, e.g. Means • Follow up with inferential statistics (the results of statistical tests), to decide whether there was any difference between the groups or conditions

31. Results • inferential statistics ... • State which test you used • The value of the ”test statistic” • Number of degrees of freedom (where appropriate) • Significance level We will get back to these – just remember that you need this stuff in the results!

32. Results • Example: • ”An independent-means t-test was performed. This showed that participants who had received 40kg of the drug ‘Coffee' recalled significantly fewer words than those who had consumed 20 mg (t (29) = 3.65, p < . 001)" Statistical test Test statistic value Significance level

33. Results • If you have many results, summarise them in tables and graphs • Do not dublicate information (in both tables and graphs) • All graphs and tables should be clearly labeled with self-explanatory titles and legends (i.e. no need to read the main text to understand what the figure shows)

34. Results Title • Figure X: An initial analysis of the causes of death in Fragile Alliance as function of time. The figure charts the percentage distribution of kills for AI-controlled autonomous agents and players. Clear data presentation Axis labels + measurement units Graph label Figure text

35. Discussion • Interpretation of the results • What do the results mean? • Explain what the statistics mean • Relation to other works • What have other people found? • Impact • What is the importance? • Critical reflection • Criticize your own work, indicate possible weaknesses, design flaws, limits of application of results? • Research agenda (what now?)

36. Discussion: Research agenda • Worthwhile suggestions for future research? • Extension to other situations, to see if the results are generalisable[applicable to entire population we are interested in] • Replication with a better-matched set of participants. • Explain briefly whythese studies would be worth doing.

37. Conclusion • Can be sub-sumed under discussion • Quick summary of the results, their meaning and importance, and setting of future research agenda. • If these are in conclusion, remove from discussion

38. Acknowledgements • This section should give thanks to the major people (supervisors, associates) and organizations (sponsoring agencies, funders) that helped you. • Mostly in research articles • For example: “The authors thank Spiderman, who provided constructive feedback on earlier drafts of this report. Also a heartfelt thanks to Captain Kirk, who financed the project”.

39. ?

40. Task 1: Research problem • You get 1 hour to give the paper a quick read-through (”skimming”) • Use this time to orient yourself in the paper (hint: start with the abstract), and discuss in the group if there are components you do not understand • After 45 minutes, discuss in the group: What is the research problem/-s the paper attempts to address? (use Fong, section 2 point 1 as a guide)

41. Sections • Sections of a report: • Title • Abstract (summary) • Introduction • Method: • Participants • Design • Procedure • Results • Discussion/Conclusion • References • Appendices 10-20% 50-70% 15-25%

42. References • ALL papers, reports, artwork or similar was cited in the report, are listed here in full detail • In alphabeticalorder, very standardizedformat • E.g.: • Buonaparte, N. and Nelson, H. (1805, March 21). An analysis of sun-bed claiming behavior in Western Europe. Seafaring Weekly, 75, 46-49.

43. References • References must provide enough informtion for the reader to be able to track it down • (unless you don´t want to of course) • This allows the reader to: • Seek further knowledge on the subject • Evaluate if you have used the reference correctly (science is all about mistrust, as Graham and Andy puts it)

44. References • There are many referencing formats • Journals, conferences and companies have different standards • Some common standards, like APA, CHI, Science, Chicago, ACM, Harvard etc. • Here we use the APA format – described in detail in our textbook • You must use this in all reports for this course

45. References • In-text references usually something along the lines of (single-author): • (Spock, 2005) • Studies by Spock (2005) indicate that ... • Studies by Spock (2005, 2007) indicate that ... • It has been claimed that aliens are illogical (Spock, 2005) • The results of Spock (2005) are at odds with those of Kirk (2004) on the matter of aliens

46. References • If multiple authors • ”Spock and Kirk (2006) agreed that aliens are indeed weird” • If more than 2 authors • ”Spock et al. (2007) argue that aliens are green” Figure 1: Are aliens weird?

47. References • Multiple references ... • ”Aliens have been claimed to be alien (Kirk, 2004; Spock, 2005; Spock et al., 2007)”

48. References • Primary and secondary references • Primary: Those you have read (ideally, ALL your references • Secondary: • Those you quote because it makes you look smart ... • Because someone else referenced them and you should probably do it too ... (looks important) • Because the primary is in Swahili and you don´t read Swahili • To avoid getting boggeddown with detail – if someone else has done the work, quote that -

49. References People say you should be honest about it if you use a secondary reference: • "It has been claimed that aliens can be cured by killing them (Kirk, 1995, as cited in Spock, 2000)". • “Kirk (1995, as cited in Spock, 2000) claimed that aliens can be cured by killing them". • I say: Adopt and adapt ...

50. References LEARN TO DO PROPER REFERENCING!