Dr. Jen Janecek-Hartman, United Tribes Technical College, Bismarck, North Dakota

1. The Lesson and Legacy of TCUP: The Importance of Evaluation for Building a TCUP Legacy: Evaluation Model for the UTNT4E TCUP Project Dr. Jen Janecek-Hartman, United Tribes Technical College, Bismarck, North Dakota

2. TCU ROI Model Highlights • Model designed using participatory research methodology • Allows for local control and American Indian value systems to be honored as well as drive the indicators of success. • Builds upon Indigenous Evaluation Framework by Nichols & LaFrance

3. TCU ROI Model Highlights • TCU ROI Model based on Phillips’ methodology which is used worldwide in academics, business and government • The principal difference between the Phillips return on investment model and the CREATION model is the embedded and overarching importance of traditional and contemporary American Indian cultural values. • TCU-ROI model also adds two more guiding principles • Must be participatory in nature • Must take into account the local cultural value system

4. Data Collected in Eight Categories

5. Counts • The first, and most basic, category of assessment is student count • Collects information about how many students access the program as well as persistence, retention, and completion rates. • Data collected provides indicators of program relevance and program effectiveness.

6. Reaction and Student Satisfaction • The second category of assessment is student reaction and satisfaction, which can be predictors of student success. • Student perspectives about the effectiveness of particular educational and support techniques are critical to assessing those techniques. • Students complete a midterm questionnaire and an end of term survey • Regarding how well the program meets their expectations and needs, the relevance of the program, their impression of the program, and how they intend to use the information they learned that semester. • Short term training course participants complete an evaluation survey at the conclusion of each training course answering similar questions.

7. Student Learning • The third category is student learning. • Information is collected through grades and learner outcomes assessments. • Grades are very important but provide only a snapshot of how well a particular student is learning. Grades can be impacted by absences (e.g. sick children), test anxiety, or any other event compounded by various risk factors. • Learner outcomes assessments provide another perspective about how well students are learning particular concepts. • They provide data about the instructional quality and program effectiveness.

8. Contemporary and Traditional American Indian Values • The fourth category is one of the aspects sets this model apart from many of the other evaluation models, that of traditional and contemporary American Indian knowledge. • Analysis of these data provides project management with indicators of how well the project is meeting the college’s mission and vision by helping students understand issues important to Indian Country and provides information regarding the acculturation levels of its students. • Curriculum guides and syllabi for the courses indicate how well each course addresses the American Indian Learner outcome. • Department assesses how well students understand what it means to be American Indian, how they feel about being American Indian (self-esteem), how well they understand the tribal-federal relationship, and how they see themselves fitting in and thriving in a diverse world by including activities around the above 4 areas.

9. Experiential Learning • The fifth category is experiential learning and internships. • Data collected in this category provides project personnel with key information regarding program effectiveness in terms of how well the students can apply the skills they have learned to the workplace. • Data collection tools for this category include supervisor observations, self-reporting, and action plans.

10. Business Impact/Impact on Indian Country • The sixth category is impact on Indian Country, which is essential to the college’s core mission and a core goal of the TCUP program. • Data collection for this category include data on student graduation, matriculation, and placement rates, employer and graduate surveys, and per-student costs calculations. • Each successful graduate, especially American Indian graduates, means success for Indian Country. • Costs calculations provide an indicator as to how efficiently the project is using its resources and what it will take for the college to sustain the program after the project funding expires.

11. Return on Investment • The seventh category is Return on Investment (ROI) and is the definitive category in the evaluation models. • Calculating a full ROI allows UTTC to clarify the costs benefit of the UT.NT4E program. • This becomes an important tool in sharing information about the costs, benefits, and impacts of the program in concrete financial terms. • All of the previous categories of assessment, and intangible data are used in calculating a full ROI of the program.

12. Intangibles • The eighth and final category of assessment in this evaluation plan includes those results which may not be quantified and are called intangibles (Phillips, 2005). • Examples may be stories about a first generation college student, stories about how community service or internships have impacted students’ educational experiences, or other nuggets of information (Jody Chase) that help round out the full story of the project’s impact. • The information gathered for this category can be used as the personal faces on the numbers born out in the other categories. • Data for this category are collected through course evaluation comments as well as monthly, quarterly, and annual reports provided by instructors, counselors, and project personnel.

13. Data Collection Table

14. Questions? Contact information • Jen Janecek-Hartman • Email: jjhartman@uttc.edu • Phone:701-255-3285 x 1396