What You Should Expect from your evaluator: promising practical practices

1. What You Should Expect from your evaluator: promising practical practices July 28, 2011 Hi-TEC Conference, San Francisco

2. Three perspectives • Experienced ATE Program manager: Lara Smith • Experienced community college professional and researcher: Dr. Jerry Somerville • Experienced ATE evaluator and researcher: Dr. Norena Norton Badway

3. Who is in attendance? • Community College • University • High School • Workforce Development Advocate • Trade Organization • Industry Professional • Technician • ATE Personnel • ATE Project or Center Director • Program Evaluator • Researcher

4. What is Evaluation? • It is a form of research • The results are to be used • Its purpose is to educate • Assist in making informed decisions

5. Information Needs of ATE Programs • Using a Delphi Process (an elaborate survey) we asked ATE leaders about the most important information needs. • Purpose: to identify research topics that researchers could study that would produce meaningful results for practitioners. • We hoped to overcome the issue that results from research/ evaluation are not used in practice.

6. The first thing we learned…

7. A Few of Our Findings • Program directors desire meaningful and usable results for program improvement • Practitioners feel that from their experience they know what works and what doesn't • They don’t have the time or expertise to conduct research • They want someone else to collect, analyze and synthesize the data and then disseminate the results in a meaningful form

8. Our Belief • Practitioners are the primary users of the information generated from evaluation/research. • If it is to be useful, practitioners need to be actively involved in working with evaluators/researchers.

9. Dissemination:Suggestions by ATE Leaders • Offer sessions at ATE/HI-TEC types of conferences • Make more use of Webinars • Publish a periodical that digests both promising practices and unsuccessful methods and presents them in briefs. • Create a listserv for ATE project and center directors so they can communicate with each other.

10. Dissemination Beyond ATE: Community College Related Community College Journal of Research and Practice Serves researchers and practitioners in higher education and the behavioral and social sciences. Journal for Applied Research in Community Colleges Serves institutional research and planning professionals in 2-year, postsecondary educational institutions The Community College Review Deals with all aspects of community college administration, education, and policy.

11. Dissemination Beyond ATE: STEM Related Journals Journal of STEM Education Emphasizes real-world case studies that focus on issues that are relevant and important to STEM practitioners Research in Science & Technology Education Publishes original research from throughout the world dealing with science education and/or technological education

12. Dissemination Beyond ATE: CTE Related Journals Journal of Career and Technical Education Serves as a forum for discussion of philosophy, theory, practice, and issues in career and technical education. Career and Technical Education Research Publishes articles that examine research in vocational/career and technical education, career development, human resource development, career issues in grades K-12, postsecondary education, adult and lifelong learning, and workforce education.

13. WHY do Evaluation? • ATE requires it • Acts as a “mirror” so you can see what you are doing • Lends legitimacy/ evidence to your claims of achievement or challenges • Is a tool for professional development for staff and stakeholders [measuring impact on student learning] • Offers collaboration for dissemination

14. Who Should You Select as Evaluator? • Match personally and professionally • Experienced with ATE and CCs • Applies evidence-based/ research theoretic framework • Moves beyond “body counts” • Analyzes, rather than audits • Partners, rather than judges • Provides sample report(s) • Evalua|t|ewebsite

15. WHEN Should You Engage Evaluator?

16. Measuring impact • Impact on student learning • Impact on faculty learning  • Impact on faculty technical knowledge  • Impact on pedagogy  • Impact on curriculum design  • Impact on organizational learning • Structure of program/ schedule of courses  • Expanding access and success 

17. Realities for an ATE Program • Establishing the evaluator relationship • Time • Budget • Time

18. Establish Scope of Work (SOW) • Accompanies contract • Defines activities; deliverables • Allows evaluator to invoice • Allows project to pay

19. Cost Proposal • Start with Scope of Work • Decide on hourly or fixed payment • Establish number of site visits and/or other travel • Establish number and type of reports

20. Budget Example Project Budget $900,000 ÷ 3 $300,000 per year Evaluation expenses $50,000 fixed fee + $9,965 expenses $59,965 6.7%7 Where are the negotiation points?

21. For more information • Norena Norton Badway nbadway@sfsu.edu • Jerry Somerville jasomerville@comcast.net • Lara Smith lara.smith@domail.maricopa.edu