380 likes | 392 Views
Evaluating and Reporting Program Impacts: Contentious Public Issues. Loretta Singletary Professor & Extension Educator University of Nevada Cooperative Extension. Staci Emm University of Nevada Cooperative Extension
E N D
Evaluating and Reporting Program Impacts: Contentious Public Issues Loretta Singletary Professor & Extension Educator University of Nevada Cooperative Extension
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Common Assumptions • Can’t measure impacts of “process” programs because… • Handing out evaluations is awkward • Participants don’t like to complete forms • I don’t need to measure the impacts because I know when it is successful • I don’t want to report results of little to no impacts or negative impacts
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Common Assumptions • Can’t measure impacts of “process” programs because… • These programs are just too complex • There is too much controversy • Some participants arrive angry, stay angry and leave angry • My role is limited to that of facilitator only • My role is that of educator only • My role is fact-finder and researcher only
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Evaluation Dilemma • All affected stakeholders do not see collaboration as a better alternative • Stakeholders do not participate equally • Participants have different skills, commitment, motives
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Evaluation Dilemma • Playing field is uneven • Stakeholders lack sufficient time and receive varied “dosage” • External forces are rushing the process • Funding and time constraints are problems
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Evaluation Required? • LOGIC model is a systematic method for program planning that requires (handout): • Situational Analyses • Priority Setting • Action Plan • Evaluation
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Benefits of Evaluation • Serve to strengthen programs over the short and long term • Evaluations determine strategies and methods that work best, under which conditions and for which purposes
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Benefits of Evaluation • Enable comparisons with traditional decision-making processes to build public understanding of consensus-based processes
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Evaluating Impacts of Programs that Address Contentious Public Issues • WHAT? • WHO? • WHEN? • HOW?
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 What to Evaluate? • Learning objectives = desired changes • Programs that address contentious public issues try to accomplish some basics: • Educate • Communicate • Explore options • Seek mutually satisfying agreements
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 What are Desired Changes? • Contentious issue efficiently addressed • Multiple options developed • Multiple options considered • Climate becomes less contentious • Climate supports public discourse • Conflict is resolved for mutual satisfaction
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Primary Impacts(Gray, 1989) • Two objective criteria to measure impacts of a collaborative effort: • Negotiated agreement is reached • Agreement has been implemented
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Secondary Impacts • Subjective criteria show impacts of collaboration • Alter individuals’ skills to participate • Alter perceptions of others • Alter perceptions of issue • Alter attitudes • Alter behavior • Develop criteria to measure these changes
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Secondary Impact Measures • Improved… • Communication skills • Networking skills • Relationship building skills • Increased hope for resolving the issue
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Secondary Impact Measures • Quality of Interaction • Volunteers learn how to share power • Treat one another fairly and with respect • Operational Details • Meetings follow timeline • Meetings accomplish goals • Participants stay informed (Gray 1989)
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Secondary Impacts… • First order vs. second order effects • Second order effects… • Increased knowledge about issues • Increased awareness of issues • Increased awareness of diverse views • New personal and working relationships • Stakeholders accept and understand scientific analyses • Respond to future conflict civilly (Innes, 1999) • Best when retrospectively measured
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 When to Evaluate? • Midcourse – partway through to assess progress and improvements • End-of-process – at conclusion to assess learning, satisfaction and first order effects • Retrospective – 12 to 18 months after process to identify value and stability of process and second order effects
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Midcourse Evaluation • After a few months of process program to assess “how it is going”… • Are process criteria being met? • Are participants satisfied? • Are representatives communicating with their stakeholder groups? • Do all participants feel well informed? • Do all participants feel equally empowered? • Are participants engaged and interested?
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 End-of-Process Evaluation • Within month or two of final meeting • Experience still “fresh” in everyone’s mind • Objectively assess accomplishments • Participants feel time well spent? • Participants plan to stay involved with implementing agreement? • Participants would try this type of process again?
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Retrospective Evaluation • Of single process takes place at least one year after process completed • Examines immediate and longer term effects of program • Traces over time program effects on participants and their relationships to one another
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Who to Evaluate? • Program objectives determines WHO • Joint fact finding • Policy dialogue • Public-private partnerships • Regulatory negotiation • Advisory Committees • Public Forums • Seeking negotiated agreement
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Who to Evaluate? • Program objectives determines WHO • Process participants (stakeholders—those directly involved) • Agencies not directly involved • Citizens not directly involved • Funding sources • Program speakers, researchers, facilitators
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? • Focus Groups (least expensive) • Face-to-face interviews • Postal surveys • Internet surveys
Principles for Writing Questions • Simple wording, yet complete sentences • Avoid vague quantifiers • Avoid specificity (how many) • Equal scalar (positive and negative) • Provide “Don’t Know” category for each question * • State both ends of scale equally (Poor / Excellent) • Eliminate “check all that apply” • Avoid open ended questions • Offer ordered responses (scale) Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate?
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? Close-ended questions with ordered responses provide participants with a response scale; Select one answer from a scale or fixed range of choices.
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? All citizens have a responsibility to conserve water on a daily basis. Strongly disagree Somewhat disagree Neither agree nor disagree Somewhat agree Strongly agree
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? • My ability to listen to others’ views has: • Improved very little • Improved little • Stayed about the same • Improved much • Improved very much
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? • My understanding of the effect of salinity on the Lahonton Cutthroat Trout fishery has: • Improved very little • Improved little • Stayed about the same • Improved much • Improved very much
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? Close-ended questions with unordered responses present answers in no particular order and participants must pick the one best response.
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? Whose responsibility is it to see to it that watershed water quality improves? (check only one): Area farmers Environmental Protection Agency My Regional Watershed Planning Group Local elected officials Environmental interest groups Every citizen living in the watershed area
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? • An example of a partially closed-ended question with unordered response is: • How do you prefer to receive ABC Program information? (check all that apply) • Printed material mailed to me • Small group meetings • Large public forums • Newspaper articles • Other (please describe) ________________________________
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Reporting QuantitativeImpacts • Ranked means using Likert scale • Percentages collapsing highs/low Likert scale • Regression analyses using participant characteristics to find impact patterns • Several other tests for relationships
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Reporting QualitativeImpacts • Frequency count by key words • Verbs • Nouns • Direct objects • Categorize comments by class • Liked most • Liked least
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Reporting Impacts • Citizen participants • Participating agencies • Your administrators • Land-grant universities (academic body) • Funding sources
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Reporting Impacts • 4-page fact sheets • In-depth Extension publications • Power point presentations to citizens • Professional conferences (ANREP, NACAA, NACDEP, ESP, Galaxy) • Refereed journal articles (JOE) • News articles • Radio and television interviews
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Bibliography • Bickman, L., Rog, D.J. & Hedrick, T.E. 1998. Applied Research Design: A Practical Approach in L. Bickman and D.J. Rog (eds.) Handbook of Applied Social Research Methods. Thousand Oaks, CA: Sage Publications. • Dillman, D. A. 2000. Mail and Internet Surveys: The Tailored Design Method, 2nd edition. New York: John Wiley and Sons. • Dillman, D. A. 1978. Mail and Telephone Surveys: The Total Design Method. New York, NY: John Wiley and Sons. • Gray, B. 1989. Collaborating: Finding Common Ground for Multiparty Problems
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Bibliography • Heron, J. 1996. Co-operative Inquiry: Research into the Human Condition. Thousand Oaks, CA: Sage Publications. • Innes, J. E. 1999. Evaluating Consensus Building. Thousand Oaks, CA: Sage Publications. • Patton, M.Q. 1978. Utilization-Focused Evaluation. Beverly Hills, CA: Sage Publications. • Salant, P. & Dillman, D.A. 1994. How to Conduct Your Own Survey. New York: John Wiley and Sons. • Singletary, L. 2004. Mail Surveys, in L. Singletary (Ed.) Conducting Community Situational Analysis: A Field Guide to Dynamic Extension Programming. Reno, NV: UNCE EB-04-02,available online at www.unce.unr.edu.
Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary, Ph.D.Professor and Extension Educator University of Nevada Cooperative Extension Phone: (775) 463-6541 FAX: (775) 463-6545 Email: singletaryl@unce.unr.edu