Assessment and Technology Research

2. Assessment and Technology Research Maj Marie Revak Director of Academic Assessment Center for Educational Excellence US Air Force Academy

3. Context of the Project • Project: USAFA Faculty Notebook Computer Study • Nov 98 – Dec 99 • 85 faculty members turned in desktop computers for notebooks • 100 faculty members received new desktop computers • Groups equal demographically

4. Purpose of the Study • Are notebooks suitable replacements for desktops? • How do notebook computers affect teaching, research, and service? • Which features and software packages are used? • Should we conduct a similar study with cadets? Under what criteria? • What relevant information can we provide to others: Air Force, DoD, and Higher Education

5. Multiple Uses for Assessment Data • For identifying needs • For evaluating programs • For improvement • For accountability

6. Methods and Measurements • Goal was to collect data from multiple sources • 6 surveys: 1 initial, 4 intermediate, 1 exit • Maintenance logs, network logs, training rosters • Anecdotal data • Data • Quantitative and qualitative • Process and product

7. Findings and Results The Notebook computer group: • Reported higher satisfaction with their overall computer experience, computer speed, response time, and ergonomic design • Provided a higher proportion of positive comments on the surveys • Spent more time (per person) dialing in • Took their computers home an average of 2-3 times per week • Used their notebook computers 93% of the time • Reported more required repairs • Added more hardware and software • Suffered no losses due to theft or accidents

8. Findings and Results • The Desktop Computer Group: • Used their desktop computers 80-85% of the time (they relied on other computers more) • Both Groups: • Reported ergonomic problems • Used computers in class about one-third of the time

9. Findings and Results • Biggest Positives for Notebooks: • Work ubiquity • Increased productivity • Biggest Negatives for Notebooks: • Mouse • Keyboard • Two computers not necessary

10. Decisions • For faculty, notebook computers are a valuable addition to the computing “mix” at USAFA • Proceed with a cadet study

11. Lessons Learned • Use rewards (or hammers) to encourage participation • Don’t rely too much on technology (to assess technology use) • Think about data analysis and reporting from the start • Don’t collect more data than you can use • Report results quickly • Must be willing to provide support while collecting data • Anecdotal data is powerful!

12. Assessment of the Impact of Ubiquitous Computing on Learning Ross A. Griffith Wake Forest University Ubiquitous Computing Conference Seton Hall University South Orange, NJ January 4-6, 2001

14. Wake Forest UniversityFall 2000

15. Overview of Presentation • Major Elements of the Strategic Plan • Assessment Structure • The Computing Initiative • Changes by Students and Faculty • Academic Outcomes • Summary

16. Major Elements of the Strategic Plan • IBM laptop computers provided to all entering freshmen and faculty effective with the fall semester 1996 • A new first-year seminar, ensuring each freshman an in-depth intellectual encounter effective with the fall semester 1996

17. Major Elements of the Strategic Plan (continued) • 40 new tenure-track faculty members representing a 15 percent increase • Scholarships for 175 students to study abroad • Fellowships for 150 students to perform joint research with faculty members

18. Assessment Structure Evaluation Committee consisting of faculty and administration formed to evaluate effectiveness of the strategic plan. Evaluation Committee sanctioned: • College Student Experiences Questionnaire (CSEQ) • Cooperative Institutional Research Program (CIRP) Freshman Survey • Freshman Essay

19. Evaluation Committee sanctioned:(continued) • Higher Education Data Sharing (HEDS) Consortium Alumni/ae Survey • HEDS Senior Survey • Higher Education Research Institute (HERI) Faculty Survey • In-house Faculty Computer Survey • In-house Faculty Survey

20. Evaluation Committee sanctioned:(continued) • In-house Student Computer Survey • Wake Forest Fact Book • Wake Forest Key Measures of Quality Evaluation Committee • Reviews Fact Book and Key Measures • Analyzes survey data • Communicates results • Conducts follow-up

21. The Computing Initiative • IBM laptop computers provided to all entering freshmen and faculty • Standing Faculty Committee on Information Technology formed • Entire campus wired • Created new position of Academic Computing Specialist (ACS)

22. The Computing Initiative(continued) • Information Systems Support Center (ISSC) revamped • Student and faculty training provided by library • Computer-Enhanced Learning Initiative (CELI) formed by faculty • Student Technology AdvisoRS (STARS) created

23. Changes by Students and FacultyCollege Student Experiences Questionnaire (CSEQ) • The survey directed by George Kuh of the University of Indiana was administered to a random sample of one-half of the freshmen, sophomores and juniors in March of 1996, 1997 and 1998 and all freshmen, sophomores and juniors on the web in spring 2000. • The main purpose of the survey is to determine the activities of students with respect to how they spend their time.

24. Items for 2000 CSEQ with Significantly Higher Mean Scores than 1998 CSEQ Time spent: • Used computer to prepare reports or papers • Used e-mail to communicate with instructor/other students • Used computer tutorial to learn material for a course • Searched the WWW or Internet for course material • Used computer to retrieve material from outside library • Used computer to produce visual displays • Used computer to analyze data • Developed a Web page/multimedia presentation

25. Items for 2000 CSEQ with Significantly Higher Mean Scores than 1998 CSEQ (continued) Institutional Emphasis: • Information literacy skills (using computers) Made gains: • Use Computers and other information technology • 28 other “intellectually developing” items

26. Items for 2000 CSEQ with Significantly Lower Mean Scores than 1998 CSEQ Time spent: • Participate in class discussion via electronic medium • Use dictionary or thesaurus • Use campus learning lab to improve study skills • Use campus recreational facilities • Had discussions w/students of different values Opinion: • Overall opinion of college • Attend same institutional again

27. CSEQ - Quality of Effort: Computer and Information Technology Scale

28. HEDS Senior Survey • The HEDS Senior Survey was administered to the Classes of 1993 through 1998 and the Class of 2000 in the spring of their senior year. • The HEDS Senior Survey is an excellent survey for indicating the degree of educational enhancement and satisfaction in a number of areas. • The Class of 2000 is the first class to graduate under the Plan for the Class of 2000 while the Senior Survey results are compared to the Class of 1998 as well as a College Group and a University Group of peer institutions.

29. HEDS Senior SurveyUse of Quantitative Tools - Enhancement

30. HEDS Senior SurveyUse of Technology – Enhancement

31. HEDS Senior Survey - 2000 Wake Forest Items Significantly Above & Below 1998 Wake Forest, 2000 College and 2000 University Above Enhancement: • Use Quantitative Tools • Read or Speak Foreign Language Below Satisfaction: • Relive college experience at same institution • Social life on campus • Ethnic/Racial diversity • Climate for minority students on campus

32. In-house Faculty Survey • The Faculty Survey was developed by the evaluation committee to assess the effectiveness of all portions of the Wake Forest strategic plan. • The survey was administered to the undergraduate faculty in December 1995 and February 1998.

33. Computers in teaching Computers in communication Computers in individual instruction Computers for presentations Computers with information gathering Computers for modeling/simulation Computer skill Computer training & assistance In-house Faculty Survey ResultsMean Scores Significantly Higher: 1998 vs. 1995

34. Students proficient with computers Technology changed effectiveness of teaching Effect of computers on communication Effect of computers on resource material Effect of computers on presentations Use of technology in teaching In-house Faculty Survey Results Mean Scores Significantly Higher1998 vs. 1995 (continued)

35. Intellectual climate among students Religious development of students Prepare students for graduate/advanced education Prepare students for employment after college End of course student evaluations Enroll more graduate students In-house Faculty Survey ResultsMean Scores Significantly Higher: 1998 vs. 1995(continued)

36. In-house Faculty Survey ResultsMean Scores Significantly Lower: 1998 vs. 1995 • Number of papers or compositions published since previous fall • Number of professional meetings attended since previous fall

37. 1998 HERI Faculty Survey • Administered by UCLA in Fall 1998 to faculty nationally • Several questions contained items regarding computer use and opinions • Wake Forest full-time undergraduate faculty results compared with peer group of nine private institutions

38. Use of Computers1998 HERI Faculty Survey

39. Sources of Stress1998 HERI Faculty Survey

40. Opinion in General1998 HERI Faculty Survey

41. Academic Outcomes First-time Freshmen RetentionFollowing Year Year Entered

42. Average GPAEnd of Freshman Year

43. Student & faculty computer usage has increased significantly since implementation of ubiquitous computing at Wake Forest Student & faculty computer usage is higher than peer group institutions Students are more engaged intellectually but are less satisfied with their overall college experience Freshmen retention rate and average GPA have increased slightly overall Summary

44. Evaluating (and planning and implementing) Ubiquitous Computing:Hypothetical Example of Flashlight Methods Stephen C. Ehrmann, Ph.D.

45. Thanks • 150+ institutional subscribers to TLT Group services • TLT Group Founding Sponsors • Blackboard, Compaq, Microsoft, SCT, WebCT • TLT Group Program Funders • FIPSE, Mellon Foundation, National Science Foundation

46. Apology • For past examples of real studies, see http://www.tltgroup.org • Click on "resources" • Click on Flashlight Case Studies • Or subscribe to F-LIGHT (free) (directions on the Web site)

47. The Challenge • Your institution is about to make an expensive upgrade in computers, connectivity • Goals of the study: • Document whether the IT helps improve educational outcomes • Increase those educational gains • Control costs, reduce stress

48. An Important Digression • Three ways of thinking about technology, two of which are usually wrong

49. Monadic Thinking • Monad = one thing: just think about technology • “Computers for all” = GREAT (or AWFUL). It’s obvious. Don’t waste money on evaluation. Technology!

50. Technology! Outcome Dyadic Thinking • “Computers for all: kids will learn calculus 20% better!” • For example, if we invest in ubiquitous computing but our study shows calculus scores did not improve, • Then either we need better machines or the investment was a mistake.