Institutional Research and Its Role in USA

1. Institutional Research and Its Role in USA Dr. Faxian Yang University of South Carolina The United States faxian@mailbox.sc.edu September 25, 2019 South East Asian Association Institutional Research (SEAAIR) Taipei, Taiwan (ROC)

2. Contents • History Review of Institutional Research. • What is Institutional Research doing? • Staffing. • Team work across campus. • Six major roles of Institutional Research. • Internal report. • Integrated Postsecondary Education Data System (IPEDS). • Case Study: Graduation rate, STEM majors. • Critique: College wouldn’t cost so much. • Trends of Institutional Research.

3. History Review of Institutional Research • University of Minnesota and University of Illinois had their IR Office in 1957. • The National Institutional Research Forum (NIRF) meeting was “conceived” in 1960 that led to the formation of AIR in later years after a few annual NIRF meetings. • Association for Institutional Research (AIR) was formally established was in 1965. • IPEDS replaced the Higher Education General Information Survey (HEGIS) in 1986. HEGIS collected data from 1966 to 1986 from a limited universe of approximately 3,400 institutions accredited at the college level by an association recognized by the Secretary of the U.S. Department of Education. • The transition to the IPEDS program expanded the universe to include all institutions whose primary purpose was the provision of postsecondary education. In 2016–17, a total of 6,760 Title IV institutions and 74 administrative offices (central or system offices) in the United States and other U.S jurisdictions participated in data collection.1 1: https://nces.ed.gov/statprog/handbook/pdf/ipeds.pdf

4. History Review of Institutional Research • From approximately 1985 the assessment movement examined major events and forces influencing assessment’s evolution as a “scholarship,” including demands for curricular and pedagogical reform, shifting patterns of accountability, and changes in instructional delivery.1The faculty teaching load was no longer the only means to evaluate faculty performance. The students’ learning experience has also used to evaluate faculty performance. • Perhaps the most influential element in accreditation’s response to changes in society and in higher education has been the shift from a focus on “inputs” (institutional resources such as library collections, student/faculty ratios, and laboratories) to “outputs” (graduation rates, graduate earnings, and, most desirably, the accomplishment of learning outcomes). • Around early 1990’s, when specialized accreditations (professional programs: business, education, engineering, law, nursing, etc.) starting to push for outcome-based assessment using “student-learning outcomes” as the target in addition to the typical reporting/accounting of inputs/outputs at colleges and universities. https://catalogimages.wiley.com/images/db/pdf/0787959456.01.pdf

5. What Is Institutional Research Doing? • Institutional Research is not a program or major offered at most universities of the United States though its unique Classification of Instructional Programs (CIP) Code is 13.0608. • Institutional Research is more than questionnaire designing, collecting, and analyzing. • Institutional research is a broad category of work done at schools, colleges and universities to inform campus decision-making and planning in areas such as admissions, financial aid, curriculum assessment, program review, enrollment management, staffing, student life, finance, facilities, athletics, and alumni relations. • Institutional researchers collect, analyze, report, and warehouse quantitative and qualitative data about their institution's students, faculty, staff, curriculum, course offerings, and learning outcomes. They are involved in collecting and reporting information to government bodies, to the public, and various college guide publishers.

7. Six Major Roles of Institutional Research • Report to Board of Trustee, President/Chancellor, Provost for the university/college’s strategic planning, as well as any other topics of interest. • State University: State government mandatory reports. Private University: COFHE, etc. • Federal Government mandatory reports: 12 IPEDS reports. • Campus wide data request. • Supply data to IE for its assessment, program review, accreditation, effectiveness. • Public and Various College Guide Publishers: Common Data Set, US. News & World Report, College Board, QS, etc.

8. Topic of Internal Report: Descriptive/Correlates • FTE Staff • Student Credit Hours (SCH) per FTE Staff • Faculty Load • Student Credit load • Unit Cost • Student Demographics: gender, race/ethnicity, first generation, geographic origin • Cost Deviations • Classroom Utilization • Learning Outcomes by Pace of Curriculum Innovation • Curriculum Development Correlation to Enrollments Wycoff, J. (2017) Outsourcing student success. Historia| Research Press, Washington, DC. P. 42.

9. Topic of Internal Report: Systemic/Causal • Enrollment Trend Projections • Inter-rater Reliability and Validity of Grading Practices • Reasons for Attendance/Choice of College • Applicant Behavior based on Application Fee Amount • Characteristics and Causes of Retention/Attrition • Level of Occupation by Gender and Degree Status • Pre- and Post- Testing Learning Outcomes in General Education • Change in Attitudes toward Psychiatry Associated with Program Intervention Wycoff, J. (2017) Outsourcing student success. Historia| Research Press, Washington, DC. P. 42.

10. Topic of Internal Report: Mechanism/Process • Enrollments Forecasting with Variable Parameters • Effect of Relaxing General Requirements on Course-Taking Behaviors • Predicted Academic Success • Predicting Medical School Success to Plan Admissions • Learning Outcomes by Physics Students by Lab Experiments vs. Lecture Demonstrations with Semi-Randomized Control Group • Pre- and Post- Testing Learning Outcomes in Independent Studies with Control Groups • Predicted vs. Actual First-Year GPAs to Determine Inequities in Grading by Academic Departments. • Wycoff, J. (2017) Outsourcing student success. Historia| Research Press, Washington, DC. P. 42.

11. The Integrated Postsecondary Education Data System( IPEDS) • Institutional Characteristics • Completions • 12-month Enrollment • Student Financial Aid • Graduation Rates • 200% Graduation Rates • Admissions • Outcome Measures • Fall Enrollment • Finance • Human Resources • Academic Libraries

13. Staffing: Basic Requirement of IR Employee • Integrity and honesty. • Attention to detail. • Multi-task orientated. • Team work, learn and cooperate with colleagues inside/outside office, as well as across campus. • Cross-checking, try to be as accurate as possible.

14. Staffing: Staff at IR Office • Bachelor degree required, and master degree preferred. • Can work to detail of content by retrieving, reporting, and analyzing data. • Willing to learn new knowledge and skills. • Have good enough written and oral communication, as well as people skills. Know how to present the data and analytics results to people who may not be good with data. • Familiar with the database table structure, which are commonly used by IR, of PeopleSoft/Banner, etc. • Familiar with at least one of following computer languages, SAS, SPSS, SQL, R. • Understand descriptive statistics, and statistical modeling skills preferred.

15. Staffing: Director of IR Office • Master degree required, and doctor degree preferred. • Have good enough written and oral communication, as well as people skills. Know how to present the data and analysis results to people who may not be good with data. Get along with colleagues inside and outside campus. • Familiar with at least one of following computer languages, SAS, SPSS, SQL. • Can complete all mandatory reports independently. • Apply statistical modeling to data, such as enrollment projection by multi factor-regression, and finding out key point of student dropping by using survival analysis. • Not only have good knowledge of program review, assessment, effectiveness, accreditation, but know how to support them. • Not only know major issues or challenges faced by higher education, but understand the reasons causing the issues, can demonstrate the issues by using data of the university, such as students’ success, faculty productivity, underrepresented minority students/faculty in STEM majors, etc. • Fully understand higher education trends, as well as the impact to the university.

16. Team Work across Campus Collaboration between Institution Research and Assessment Institution Research (IR) Assessment (IE) Questionnaire designing, collecting. Course evaluation. Quality Enhancement Plan (QEP). Faculty/student related topics. Program Review. Accreditation. • Supply aggerated data, information, analytics to support Program Review, Institutional Effectiveness, Assessment, Accreditation. • Questionnaire analyzing.

17. Team Work across Campus Collaboration between IR and IT Offices Institution Research (IR) Information Technology (IT) Expert of all database structure. Expert of query data. Have basic knowledge of business rule to higher education. The end products are data. • Familiar with database relationships. • Query the database to retrieve data. • Knowledge of all business rule related to higher education, especially related to student, course, degree award, admission, human resources, test score, financial aid, facility use. • The end products of IR are information to support decision making.

18. Team Work across Campus Collaboration to Admission, Registrar, and Financial Aid, Human Resources • Admission Cross-checking the accuracy of students in cohort, and SAT/ACT scores. • Registrar Cross-checking students enrollment and degree awarded status. • Financial Aid Cross-checking students award status and award amount. • Human Resources Verify employees’ information, especially faculty’s tenure, employment, terminal degree status.

19. Team Work across Campus Collaboration to College/School and Department • Supply data/information for their program review, accountability, assessment, as well as specialized accreditations. • Supply data/information for the faculty proposal application. • Supply data/information for their strategic planning.

20. CritiqueCollege Wouldn’t Cost So Much If Students and Faculty Worked Harder • By Richard Vedder • Source: A17 The Wall Street Journal • Thursday, April 11, 2019 • He earned his bachelor's in economics in 1962 from Northwestern University in 1962 and his Ph.D in economics from the University of Illinois in 1965. • He has since studied U.S. economic history, particularly as it relates to public policy. Some of his research has involved American immigration, economic issues in American education, and the interrelationship between labor and capital markets.

21. CritiqueMain Ideas of the Article: Faculty Reasons • I assign far less reading, demand less writing, and give higher grade than I did two generations ago. • I taught three courses a week for nine hours in 1965; my colleagues today teach only two courses for six hours. At some top-flight research universities, senior professors may teach only one course. Results • The volume of research is immense-but little of it is often cited or even read. • Why should professors reduce their teaching load to write papers for the Journal of Last Resort? Diminishing returns have long since set in.

22. Critique Main Ideas of the Article: Students • Reasons • Students do not study much, professors teach little, few people read most of the obscure papers the professor write. • Students study about 27 hours a week. The typical student takes classes only 32 weeks a year. So he/s spends fewer than 900 hours per year. • Results • Collegiate Learning Assessment indicates the typical college senior has only marginally better critical reasoning and writing skills than a freshman. • A civic literacy test administered by the Intercollegiate Studies Institute Shows appalling gaps in knowledge, with senior knowing little more than freshmen.

24. Letter to the Editor, The Wall Street Journal April 19, 2019 Student Faculty Student evaluations provide a deterrent against more demanding and critical review of their work, especially for untenured professors. Tom O’Hare, Boston College In the last 30 years college bureaucracies have become bloated with highly-paid nonteaching bureaucrats. According to Johns Hopkins University political science professor, Benjamin Ginsberg, the number of “administrators rocketed by 85% and their attendant staff by a whopping 240% from 1985 to 2005.” Few of these bureaucrats have real practical work to do. The salaries of top bureaucrats have increased exponentially over the past 20 years and the trend rapidly escalated after the Bipartisan Student Loan Certainty Act passed into law in 2013. Eric Allan Sibul Ph.D., Wilbur Wright College, Chicago If Prof. Vedder signed up for his own school’s 100 level pre-med courses, like biology, chemistry, or physics, I’m confident he’d reach a different conclusion. The only students in those courses who don’t work hard are the 50%-60% who withdraw before the midterm. Richard Crane MD, Chicago As a college freshman in 1950-51, at Regis College (Weston, Mass.), I carried a required 22 hours a semester while my then friend, now my husband, carried 21 hours at Boston College. We felt fortunate to have one evening a week free to socialize as the remaining six were spent in hard study. Before examinations we often devoted more time to study than that. Why is the average degree today not worth what it has been historically, and at so very much greater a cost? Patricia Caroline Cruise, Cincinnati

25. Case Study I: Student Successful Rate • Student academic performance, especially their graduation rates, is a key indicator of the student success. • Nationwide, the six-year graduation rate at four-year institutions is around 60% based on the latest information from National Center for Education Statistics. • The rate was 59% at state institutions, 66% at private nonprofit institutions, and 26% at private for-profit institutions. • U.S. Department of Education (2018).

26. Case Study I: The College Dropout Scandal • The contention that college is the engine of social mobility is false advertising for the 34 million Americans over twenty-five, who have some college credits but dropped out before receiving a diploma. • When it comes to the “dropout scandal” there is plenty of blame to go around. • One can start with high schools that don’t teach the skills children need to get a good job but instead encourage all students to go to college even if they don’t have the ability or inclination. • There is the federally funded financial-aid system that encourages colleges to keep raising prices, making higher education ever more out of reach for poor or working-class kids. • And there are the colleges themselves, particularly the larger public schools: They don’t offer students much guidance about which courses to take or how to finish a degree in a reasonable amount of time. Kirp, D. (2019) The college dropout scandal. Oxford, UK: Oxford Press.

27. Case Study II: STEM Majors • International students make up the large majority of full-time students in many graduate science- and engineering-related programs. • Their numbers have been rising much faster than the number of domestic students, according to a new report from the National Foundation for American Policy, a research organization focused on immigration and the economy. • Redden E. (2017) Inside higher education.  October 11, 2017

28. Case Study II: Doctorates Awarded in Science and Engineering FieldsNational Science Foundation (2017) 2015 Doctorate recipients from U.S. universities national center for science and engineering statistics directorate for social, behavioral and economic sciences.

29. Case Study II: STEM MajorsRatio of International Graduate StudentsRedden E. (2017) Inside higher education.  October 11, 2017

30. Case Study II: STEM Majors Six STEM Courses Taken more than Once

31. Trends of IR • Big data is applied in some study of IR, such as longitudinal study, which tracks data of the students from K-12 system to employment. • Business Intelligence (BI) becomes more and more popular, especially Tableau software. • Vendor companies are playing more of a role in supporting the traditional function of IR. They help university complete some analysis which cannot done easily by IR office, such as report faculty’s publications and citation by Academic Analytics. • IPEDS has more and more rigorous requirements of its 12 reports. It crosschecks not only same information among different IPEDS reports in same year, but compare the information with previous year to see whether it is consistent. This makes the IR task more challenging. • It is very challenges for IR to recruit new qualified employees.

32. Questions and Comments Thanks!!!