Evaluation of Student's Papers & Presentations in Italian Network of PhD Programs in Food Science, Technology, and Biote

1. The evaluationofstudent’spapers and presentations in the Italian Network ofPhDprograms in Food Science, Technology and Biotechnology: a primary trait analysisexercise Eugenio Parente Università degli Studi della Basilicata

2. In thispresentation • the need for a consensus system for PhD student evaluation • a word on plagiarism • what is primary trait analysis? • the “ideal” paper and presentation • the rubric: • Material organization, accuracy, language, discussion of results, references • Research and methodology, novelty, statistical analysis, data-supported conclusions • Quality of the presentation, clarity, language property, mastery of the topic • Interaction with the referees • An overview of the evaluation at the XV Workshop

3. The Network ofItalianPhDprograms in Food Science, Technology and Biotechnology • Estabilished in 1995 • Currentlyincludes 24 PhDprograms (DoctorateSchools, Doctoratecourses, curricula) • promotesanannual meeting • promotesstudentsexchanges, co-operationamongPhDcourses, fostersexcellence in scientificachievements in Food Science, Technology and Biotechnology

4. The workshop • first workshop in Viterbo, now in the XVI edition at the PTP Lodi • attendancehasincreasedovertime:122 studentsattended the XV Workshop in Naples • designedtoencourageexchageofideas and experiencesamongstudents, tutors and coordinators • intendedtotrainstudents in presenting and defendingtheir work • first yearstudents: PhDDissertation project and miniposter • secondyearstudents: short communication and poster • thirdyearstudents: short paper and oralcommunication • students are evaluated on a pentenary scale and ECTS credits are assignedas a resultof the evaluation • the Proceedings are publishedas hard copy and on DVD-ROM

5. The evaluationprocess • the students are required to submit a paper within the end of June • the responsibility of ensuring the quality of the paper and the compliance with the presentation guidelines remains with the student, the tutor and the PhD coordinator • papers are published as they are in the Proceedings • at the workshop: • first and second year students present a poster, which must be discussed with 2 reviewers, who grade the work • third year students are required to give an oral presentation and to discuss it with the session chairman and/or additional reviewers • The reviewers turn in the evaluation sheets to the Organizing Committee, and a certificate is issued to the students

6. The needfor a consensus system • students and reviewers come from different areas (Food Science, Food Microbiology, Food Chemistry, etc.) in which different standards may be applied • there is a need to: • make the process transparent (i.e. the students must be informed of what is expected of them) • make the process fair (i.e. the referees should use a similar evaluation scale) • The process in use has evolved in several years of discussions and debate

7. A word on plagiarism • plagiarism is a plague in science (and teaching) and students do not seem to be fully aware of the consequences • plagiarism is "taking over the ideas, methods, or written words of another, without acknowledgment and with the intention that they be taken as the work of the deceiver." American Association of University Professors (September/October, 1989). • ethical writing principles are necessary • students must be aware that republishing material published elsewhere is something to be avoided

8. The design of the evaluation score sheet: a primary trait analysisexercise • “Primary Trait Analysis (PTA) is a way to take what we already do - record grades - and translate that process into an assessment device”. B. Walvoord and L.P. McCarthy, 1990 • the PTA procedure (barebones): • learning goals are clearly identified • a test/assignment is designed to evaluate how the learning goals have been achieved • the assignment (the paper, poster, presentation) is split in components (the primary traits) • a rubric (in the form of a table or a list) is built to identify different levels of achievement of the learning goals and a score is assigned to each level • each cell should contain an “anchor”: a verbal description of the level of performance corresponding to the cell. As a result, different reviewers should assign approximately similar grades to similar performances • the rubric is made public in advance • if correctly carried out the PTA procedure has several advantages • it is clear • it is transparent • it is fair • it assists both the teacher and the student in assessing how well the learning goals have been achieved

9. The rubric • Manuscriptquality: MQ, 30 points. Material organization, accuracy, language, discussionofresults, references • Noveltyofcontents: NC, 20 points, research and methodology, novelty, statisticalanalysis, data-supportedconclusions • Presentationquality: PQ, 20 points. qualityof the presentation, clarity, languageproperty, masteryof the topic • Interactionwith the referee: AR, 30 points, Accurate replytoreferee'squestions

10. The rubric

11. The rubric

12. The “idealmanuscript”: manuscriptquality • material organization (tests the abilityof the studenttowrite a scientificpaper in the format requiredby a given journal): • the text mustbedrafted in accordanceto the guidelinesforauthors; • all the requiredsections are included; • the objectives are clearlystated; • allfigures and tablesarereferredto in the text and providedwithclearlegends/headings, • the presentationofresultsisclear and concise; • the referencelistwasdraftedaccordingto the instructions

13. The “idealmanuscript”: manuscriptquality • Accuracy (evaluates the abilitytopresentmaterials and methods, figures and tables): • the material and methodsmustbeclearlyillustrated, • the resultssectionmustbe complete, • Tables and Figuresmust show the resultswith the necessaryaccuracy (significantdigits, readability). • Language (evalutes the abilitytousesceintific English): • the text mustbecorrect in termsof spelling, grammar, English usage.

14. The “idealmanuscript”: manuscriptquality • Discussionofresults (evalutes the abilitytodiscussratherthanmerelypresent, the results): • the resultsmustbediscussed in viewof the objectives and ofcurrentliterature and notmerelydescribed; • the authorsmustpresent data supportedhypotheses; the speculationmustbeavoided in the discussion. • References (evaluates the abilitytoselect and useanadequate set ofreferences): • the referencesmustbesufficientto illustrate the state of art (asrelatedto the subjectof the article) • mustbesufficienttocorrectlyusedtodiscussanycorrespondence or disagreementwith the author'sfindings.

15. The “idealmanuscript”: noveltyofcontents • Researchmethodology (evaluates the abilitytousean appropriate methodologicalapproach): • the researchmethodologydescribed in the text published in the Book ofProceedingsshouldbe appropriate for the topic at hand (given the resourcesavailable); • Novelty (evaluates the abilitytoaddressnewideas/subject/approaches): • the work mustbenovel in termsofapproach, methodology, results, conclusions • plagiarism (includingself-plagiarism) isunacceptable

16. The “idealmanuscript”: noveltyofcontents • Statisticalanalysis (evaluatesheabilityto design experiments and analyze the data with the appropriate method): (notapplicabletominiposters) • the statisticalmethodsmustbe appropriate for the topic at hand • the resultsof the analysismustbeclearlypresented, withoutredundancy and unnecessarydetails. • thereis no excuseforlackofstatistical treatment of the data. • Data supportedconclusions (evaluates the abilitytocriticallyanalyze the results): (notapplicabletominiposters) • the conclusionsmustbefullysupportedby the data and anydiscrepancywith the objective, hypotheses, literaturehastobeaddressed and/or explained.

17. The “idealpresentation”: presentationquality • Qualityof the presentation (evaluates the abilityto set up and organizeanoral or poster presentation): • the oral, poster or miniposter communicationmustbeclearlyorganized (sequence, graphics, organization) • mustnotexceed the allottedtime (15 min for the oralcommunication or 5 min for the poster and miniposter) • mustbeclearlyreadable/visibleby the audience. • Clarity (evaluates the contentof the presentation/poster): • the introductory material, the methods and the resultsmustbeclearlypresented, • the resultsmustbediscussed and comparedwith the literature and with the objectiveof the work

18. The “idealpresentation”: presentationquality • Masteryof the topic (evaluates the knowledgeof the problem at hand): • the author (not the tutor) mustbecurrentwith the pertinentscientificliterature, master the methods and understand the results and theirimplications. • Languageusage (testsfluency in spoken English): • the spokenlanguagemustbecorrect in termsofgrammar, English usage, and pronunciation. If the presentationis in Italian the score isautomaticallynil.

19. The “idealpresentation”: interactionwith the referees • we espect that our students are able to defend successfully their work in a scientific meeting • their answers must be pertinent and complete • if the referees points out a problem (maliciously or not) that problem must be addressed in a satisfactory way, not eluded

20. The evaluationasanassessmentexercise • the purpose of a PTA excercise is two-fold: • grading i.e. assigning a grade (A to E) in a fair and transparent way • assessment: i.e. evaluating the teaching, learning, and assessment process itself to introduce corrections is needed

21. A bit ofstats: the evaluation at the XV workshop. Q1. Do the studentsimproveovertime? Standardized score

22. A bit ofstats: the evaluation at the XV workshop. Q2. Is the distributionofscoresnormal? Data for 3° year students

23. A bit ofstats: the evaluation at the XV workshop. Q3. IsthereanydifferenceamongstudentsfromdifferentPhDcourses? Standardized total score

24. A bit ofstats: the evaluation at the XV workshop. Q4. Can webuild a syntheticindex (using PCA) and compare students (rawscores are used)? total score PQ, AR circles: 1st year triangles: 2nd year squares: 3° year MQ, NC

25. Whatnow?