1 / 46

THE QUANTITATIVE/ QUALITATIVE IMPACT STUDY

THE QUANTITATIVE/ QUALITATIVE IMPACT STUDY. BENEFITS AND CHALLENGES Dr. Gloria Pryor James Dr. Brenda R. Shaw Mr. Samuel T. Rhoades, J. D., President Emeritus National Association of HBCU Title III Administrators, Inc. Technical Assistance Workshop June 23, 2011. WHAT IS AN IMPACT STUDY?.

kedma
Download Presentation

THE QUANTITATIVE/ QUALITATIVE IMPACT STUDY

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. THE QUANTITATIVE/QUALITATIVE IMPACT STUDY BENEFITS AND CHALLENGES Dr. Gloria Pryor James Dr. Brenda R. Shaw Mr. Samuel T. Rhoades, J. D., President Emeritus National Association of HBCU Title III Administrators, Inc. Technical Assistance Workshop June 23, 2011

  2. WHAT IS AN IMPACT STUDY? TheProgram Impact Studyis one of the most important processes that identifies over time the impact of the Institution’s programmatic outputs and inputs and the impact that these components had on the target population and the University, as a whole.

  3. SOME BENEFITS OF THE QUANTITATIVE/QUALITATIVE IMPACT STUDY Specific outcomes can be compared to resources expended; and successful programs can be highlighted;

  4. SOME BENEFITS (CONTINUED) Programs that have not accomplished their intended outcomes can be identified; The extent to which the planned outcomes were met can be documented;

  5. SOME BENEFITS (CONTINUED) The adequacy of the program’s outcomes as related to the problem that the program was developed and implemented to address can be documented;

  6. SOME BENEFITS (CONTINUED) Strengths and weaknesses of each activity can be identified that affected theEffectivenessandAdequacyof the programmatic inputs ($$$, human resources, space, etc.) in meeting the activity’s planned outcomes.

  7. SOME BENEFITS (CONTINUED) The institution has the ability to determine the impact of the Title III funds awarded to various discrete activities and programs.

  8. SOME BENEFITS (CONTINUED) • The process determines not only the success or failure of the program, but also provides explanations for the actual outcomes.

  9. PRIMARY BENEFIT A COMPREHENSIVE picture of the overall impact of the programs being evaluated is the major outcome.

  10. WHY CONDUCT EVALUATIONS AND IMPACT STUDIES As a result of theMANDATE of funding agencies and institutions to demonstrate programmatic accountability, one of the services requested most by institutions of higher education is programASSESSMENT/EVALUATION.

  11. ABOUT NORTH CAROLINA CENTRAL UNIVERSITY • NCCU, chartered in 1909, opened its doors to students in 1910 as the National Religious Training School and Chautauqua.

  12. ABOUT NORTH CAROLINA CENTRAL UNIVERSITY • North Carolina Central University is a part of the University of North Carolina System, made up of 16 individual campuses (17 when we count the School of the Arts) and headed by a single president and governed by the University of North Carolina Board of Governors.

  13. ABOUT NORTH CAROLINA CENTRAL UNIVERSITY • North Carolina Central University is a comprehensive university offering bachelor’s degrees in more than 100 disciplines, master’s degrees in more than 40 programs and three professional degrees in law.

  14. ABOUT NORTH CAROLINA CENTRAL UNIVERSITY • Enrollment at North Carolina Central University increased from 5,753 in Fall 2001 to 8, 619 for Fall 2006, an increase over the six year period of 33%. We currently enroll 8,500 students (2010-2011). • Major emphasis for the current planning period (2007-2012) is “Enhancing Retention, Graduation, and Placement Rates through Increased Student Learning.”

  15. ABOUT NORTH CAROLINA CENTRAL UNIVERSITY Our goals: • to provide programs, activities, and services that will assist students in becoming successful during their tenure at the University so that they persist and graduate and are successful in the career markets of the future; • to put in place those systems designed to make the matriculation more seamless and easier to negotiate;

  16. ABOUT NORTH CAROLINA CENTRAL UNIVERSITY • to provide the training and upgrading of the skills of our faculty to work with those students whom we enroll that have been underserved and are from diverse backgrounds; and • to promote a more efficient, effective, and accountable environment as relates to student learning and student success.

  17. INCEPTION OF THE IMPACT STUDY: INTERNAL MOTIVATION • Impetus for conducting an Impact Study – Chancellor Charlie Nelms; • A directive he gave me as Title III Director, which I passed on to the Activity Coordinators in our planning meeting; • His absolute need to have concrete documentation of the impact that our Title III dollars had had over the years.

  18. CONTRACT OUT OR CONDUCT INTERNALLY North Carolina Central University (NCCU) contracted with Associates for Institutional Development, Inc. (AID, Inc.) to conduct an Impact Study of select Title III activities encompassing the grant years 2004-2009 to document the extent to which each activity had accomplished the outcomes for which the activity was funded.

  19. THE KEY IS COLLABORATION THE INSTITUTION MUST: THE RESEARCH TEAM MUST: WORK CLOSELY WITH THE TITLE III STAFF TO COLLECT THE DATA • PROVIDE ACCESS TO ALL RECORDS PERTINENT TO EACH ACTIVITY

  20. ASSESSMENT GOALS OF THE IMPACT STUDY A data collection processMUST be implemented that includes: • Onsite visits; • Interviews with key staff; • Participant self-assessments;

  21. ASSESSMENT GOALS OF THE IMPACT STUDY • Interviews with faculty, students, and external stakeholders; • The administration of surveys and questionnaires, as needed.

  22. ASSESSMENT GOALS OF THE IMPACT STUDY To guarantee objectivity, the process relies upon multiple sources of data encompassing multiple points of view. Finally, to be adaptable, the assessment’s design must meet the assessment needs of the institution.

  23. INFORMATION NEEDED FOR THE STUDY • PROGRAM PLANS (RATIONALE, TARGET AUDIENCE, BASELINE DATA; • MEASURABLE OBJECTIVES; • MEASURABLE AND TIME-SPECIFIC PERFORMANCE INDICATORS; • IMPLEMENTATION STRATEGIES IN CHRONOLOGICAL ORDER

  24. INFORMATION NEEDED FOR THE STUDY • DOCUMENTATION ON ACTUAL PROGRAMMATIC OUTCOMES (DESCRIPTIVE AND STATISTICAL) • EXTERNAL EVALUATION REPORTS • PROGRESS REPORTS (MONTHLY OR QUARTERLY)

  25. INFORMATION NEEDED FOR THE STUDY • PROGRAMMATIC OUTCOMES (QUANTIFIABLE AND DESCRIPTIVE)

  26. INFORMATION NEEDED FOR THE STUDY • ANNUAL REPORTS • RESULTS OF SURVEYS AND QUESTIONNAIRES • FUNDS AWARDED EACH ACTIVITY DURING THE YEARS INCLUDED IN THE STUDY 

  27. INFORMATION NEEDED FOR THE STUDY • ACTIVITY BUDGET BURN RATES • ANEDOTAL INFORMATION THAT WILL ADD VALUE TO THE OUTCOMES

  28. CRITICAL CHALLENGES TO CONDUCTING AN IMPACT STUDY • OBJECTIVES THAT ARE NOT MEASURABLE • TOO MANY TASKS MASQUERADING AS OBJECTIVES • PERFORMANCE INDICATORS THAT ARE NOT MEASURABLE AND NOT TIME-SPECIFIC

  29. CRITICAL CHALLENGES TO CONDUCTING AN IMPACT STUDY • THE ABSENCE OF BASELINE DATA FOR THE FIRST YEAR OF IMPLEMENTATION AND EACH YEAR OF FUNDING (WHERE APPROPRIATE) • LACK OF ANNUAL EVALUATION REPORTS OR SOME OTHER TYPE OF ASSESSMENT TO VERIFY THE ATTAINMENT OF THE OBJECTIVES EACH YEAR OF FUNDING

  30. CRITICAL CHALLENGES TO CONDUCTING AN IMPACT STUDY • ACTIVITY DIRECTORS NOT HAVING A CLEAR VISION OF THE PRIMARY ANTICIPATED/PLANNED OUTCOMES OF THE ACTIVITY AND HOW THE ACTIVITY WILL STRENGTHEN THE INSTITUTION

  31. IMPACT STUDY PROCESS USED AT NCCU • Identified the over-arching goal or planned outcome for each program; • Identified the actual outcome based on data;

  32. IMPACT STUDY PROCESS USED AT NCCU • Analyzed the gap between the desired outcome and the actual outcome by utilizing the “Effectiveness Estimate (EE)” technique:

  33. “EFFECTIVENESS ESTIMATE (EE)” TECHNIQUE: • DOCUMENTS: THE EFFECTIVENESS OF THE PROGRAM IN AMELIORATING OR STRENGTHENING THE ENTITY FOR WHICH IT WAS FUNDED

  34. ELEMENTS OF THE EFFECTIVENESS ESTIMATE • (R) Actual Results • (C) Results Without Program • (P) Planned Impact

  35. THE EFFECTIVENESS ESTIMATE EFFECTIVENESS ESTIMATE = (R) Actual Results - (C) Results Without Program Divided By (P) Planned Impact - (C) Results Without Program

  36. ADEQUACY RATIO (AR) • QUANTIFIES: PROGRAM ACCOMPLISHMENT BY CONCEPTUALIZING THE “PROPORTION OF THE PROBLEM” ELIMINATED BY THE PROGRAM

  37. ELEMENTS OF THE ADEQUACY RATIO • (R) Actual Results • (C) Results Without Program

  38. ADEQUACY RATIO ADEQUACY RATIO = (R) Actual Results 1 - (C) Results Without Program Divided By

  39. THE IMPACT STUDY REPORT INCLUDES: • Institutional enhancements as a result of implementing the program; • Enhancements to student and faculty performance, where applicable;

  40. THE IMPACT STUDY REPORT INCLUDES: 3. Affect on administrative personnel effectiveness, where applicable; 4. Overall academic and/or administrative program effectiveness enhancements; • Problems encountered that adversely affected the implementation of the program during the specified time- period;

  41. THE IMPACT STUDY REPORT INCLUDES: • Recommendations to strengthen the continued implementation of each programregardless of funding status;

  42. THE IMPACT STUDY REPORT INCLUDES • Results of the Effectiveness Estimateanalysis to determine the level of program effectiveness;

  43. THE IMPACT STUDY REPORT INCLUDES • Results of the Adequacy Ratioanalysis documenting the extent to which the program addressed the problem it was designed and implemented to address and the extent to which extramural funding had on program effectiveness.

  44. CONCLUSION Information garnered through the impact study process, in combination with other sources of data, will provide an opportunity for the institutional leadership to incorporate the findings into programmatic and budgetary decision-making.

  45. CONTACTS • Dr. Gloria P. James: gpryorjames@comcast.net • Dr. Brenda R. Shaw: bshaw@nccu.edu • Mr. Samuel T. Rhoades: strhoades@aol.com

More Related