200 likes | 331 Views
Helping Program Assessment Using Automated Text Processing. Anne Gilman SoTL Brown Bag 25 Sept 2013. Finding questions we hadn't thought to ask. ..in what students say. Assesment Tools. The Law of the Instrument:
E N D
Helping Program Assessment Using Automated Text Processing Anne Gilman SoTL Brown Bag 25 Sept 2013
Finding questions we hadn't thought to ask... ..in what students say.
Assesment Tools • The Law of the Instrument: “Give a small boy a hammer, and he will find that everything he encounters needs pounding.” -Abraham Kaplan
Assesment Tools • Are there tools that can help us • ..to articulate goals we take for granted? • ..to point in new directions?
Data • LiM participant essays • 57 from 2007-2012 • Holistic ratings • 23 Director’s Cut • Totals: • 9 Excellent • 11 Good • 3 Just barely makin’ it
LiM Assignment In a paper of at least two pages, discuss what the Language in Motion experience has meant for YOU. Be as specific as possible and provide examples where relevant. Consider the following: A. Why did you choose to participate in Language in Motion? What personal goals, both short and long term, did you have for this experience? Were they met? If so, how well? If not, why not? B. In what areas have you improved your skills and/or knowledge? Consider what you learned about the following: • Your second language, if any, • The topic(s) of your presentations, • Your home culture, • Your second culture, • American education, • Presentation skills: Organization, ability to give clear directions, adjustment of content to audience (age and knowledge level), etc. • Yourself: Communication or cross-cultural communication abilities, time management, self-confidence, flexibility, leadership, etc. C. Was there something you wanted to learn through this experience that you didn’t? If so, what was it and why did you not learn it? Is there something we should do differently?
Next Steps: • Analyze by paragraph • Consider value of enjoyment • Look under the hood
Are these more familiar? • Hoping the XPS viewer works in our room!
What's a trigram? • Demo
Results: Cluster-to-Category Match • Better: • Talking less about your own country • zzt, zto, top • TBD: • Less: tha • More: ent, eth
Next Steps • Get more ratings • Add more trigrams • Compare to LiM, ILAC goals • Consider POEs
Many Many Thanks • LiM Director: Dr. Deb Roney • Gilman Lab Assistants • LIWC project: Helen Hu, Aric Koestler, Olivia Moody, Sungouk Park, Seth Weil • Trigram project: Tori Rehr, Tori Buser • Dr. Kim Roth • The Juniata Department of Psychology • The SoTL Leadership & the Lakso Fund
Gory Details • > aggregate(limtrunc[-1], by=list(cluster=fittrunc.km$cluster), mean) • cluster age all and ati den • 1 1 0.002255296 0.002815929 0.007260874 0.006236264 0.002169403 • 2 2 0.004112582 0.002038445 0.006445021 0.007287239 0.003207497 • 3 3 0.001498536 0.002247922 0.007557460 0.006659985 0.004155337 • ent ere erseseest • 1 0.007423606 0.003938757 0.002885631 0.003315418 0.003273650 • 2 0.010172528 0.003758447 0.001438849 0.005238279 0.003749762 • 3 0.010626493 0.004926549 0.002321635 0.004856809 0.003323920 • eth eve for. hat her • 1 0.003958753 0.002694240 0.003256768 0.005716957 0.003714643 • 2 0.002908776 0.002441420 0.002277172 0.004412824 0.002856234 • 3 0.003351811 0.002440801 0.003763736 0.002332821 0.003069785 • ing ion ztozztver • 1 0.008014828 0.005659334 0.001344568 0.001344568 0.003019872 • 2 0.008223715 0.009158820 0.001807482 0.001807482 0.001872784 • 3 0.005631146 0.006514531 0.004178863 0.004178863 0.003330204 • tertha the thitio • 1 0.002892176 0.005949131 0.01666473 0.003739603 0.005432872 • 2 0.002505902 0.004485792 0.01535570 0.003083640 0.008802554 • 3 0.002784424 0.003212007 0.01513814 0.004036677 0.006142843 • top tthntsonsopi • 1 0.002193457 0.003513021 0.002240106 0.002399789 0.001670029 • 2 0.002822616 0.002456230 0.002990221 0.002544836 0.002204109 • 3 0.005067310 0.002195498 0.003912161 0.002463628 0.004443421 • ore pia pre rea res • 1 0.002220215 0.001284905 0.003154103 0.003108067 0.004400721 • 2 0.001488311 0.001586731 0.005186367 0.002198381 0.005775334 • 3 0.002816267 0.004117230 0.004174962 0.003289315 0.005404926 • sth • 1 0.002374196 • 2 0.002569780 • 3 0.002009747