American Evaluation Association/ Canadian Evaluation Association joint meeting October 28, 2005

1. Creating Uniform Standards for Research and Development EvaluationPanel: Cooperative Efforts in International Science and Technology Evaluation American Evaluation Association/ Canadian Evaluation Association joint meeting October 28, 2005 Cheryl J. Oros Director, Planning and Accountability Cooperative State Research, Education, and Extension Service

2. Evaluation Goals – Aid Decision Makers • Evaluation is applied research in service to decision makers • Provide performance and evaluation results which will enable managers and decision makers to implement, manage, refine, and show the impact of the highest quality programs. • Meet external accountability requirements

3. Evaluation/ManagementCycle Planning/Decision Making Evaluation Feedback • Identify needs, problems, solutions, refinements • Conceptualization of Program • Formulation of Evaluation Questions and Design • Feedback of Evaluation Findings to Managers • Refinement of Program • Continued Funding Implementation • Actualize the Program Plan • Collection of Evaluation Data • Analysis of Data

4. Standardization Recommendations • Use of Strategic Planning, with alignment of research portfolios, programs, and projects • Use of OMB/ NAS Research and Development Criteria • Use of logic models to develop relevant evaluation questions, indicators, and approaches

5. Table 2: Common Evaluation Approaches For Assessing Program Effectiveness

6. New U.S. federal encouragements to standardize evaluation approaches • PART • BPI • Budget Justifications

7. OMB PART • OMB’s Program Assessment Rating Tool • Program Purpose & Design • Strategic Planning • Program Management • Program Results

8. Federal Budget & Performance Integration • Budget submissions need alignment with goals & objectives, with funds distributed accordingly, along with quantifiable performance measures

9. Explanatory Notes/ Budget Justifications • Introduction: national problem, why important & how CSREES plans to solve problem • How CSREES plans to spend funds • What would happen if funding not received

10. Budget justifications (cont.) • Draw on past accomplishments • Identify performance measurements and evaluation plans • Acknowledge and deal with environmental complexities

11. Example of fully aligned Strategic Goals: 1. Enhance economic opportunities for agricultural producers 2. Support increased economic opportunities and improved quality of life in rural America 3. Enhance protection and safety of the nation’s agriculture and food supply 4. Improve the nation’s nutrition and health 5. Protect and enhance the nation’s natural resource base and environment

12. Cascading Alignment Mission . Goal Strategic Objective Portfolio Topic areas Programs/projects

13. Alignment Example Goal 3: Enhance Protection & Safety of Food Supply Strategic Objective 3.2 b: Animal Protection Animal Protection Portfolio • Knowledge Areas • (KA 311) Improve the management of animal diseases • (KA 312) Increase knowledge & improve management of pests & external parasites • (KA 313) Increase knowledge to control internal parasites • (KA 314) Reduce losses in livestock, poultry, and fish • (KA 315) Develop and implement effective animal care

14. Evaluating Research & Education Efforts: Portfolio Approach • OMB PART/BPI led to development of new portfolio assessment tool and measures • Portfolio analysis (meta-analysis) used to assess progress toward goals; guide future research plans • Uses OMB (NAS) R&D criteria (relevance, quality, performance)

15. “Portfolio” concept • Allows numerous complex interrelated programs, funding lines, and research projects to be described as they address strategic objectives and goals

16. Portfolio Review Expert Panel (PREP) Process • Focus on outcomes rather than processes • Level of analysis is a portfolio identified via research topic areas in databases • Scores portfolio progress & provides recommendations for Agency

17. Evidentiary materials • Tracks papers, citations, patents, products, educational efforts, adoption of products/ practices • Entire evaluation studies and special analyses • Budget tables to show portfolio priorities and emphases

18. 5 Dimensions of Relevance • Scope • Focus on critical needs • Identification of emerging issues • Integration of CSREES programs • Interdisciplinary integration

19. 4 Dimensions of Quality • Significance of findings & outputs • Stakeholder assessment • Alignment of portfolio with current science • Methodological rigor

20. 5 Dimensions of Performance • Portfolio productivity • Portfolio completeness • Portfolio timeliness • Agency guidance relevant to portfolio • Portfolio accountability

21. Inputs Outputs Activities- What we do Outcomes-Impact Short term Medium Term Long Term • What we invest • Investments $ • Collaborative Ventures • Researchers • Labs & Facilities • Research: • Conduct studies • Design new techniques/ technologies • Develop models • Dissemination: • Publications • Citations • Patents • Networking What the short term results are New Knowledge, technologies, techniques, products developed and commercialized What the medium term results are Adoption/ use of new knowledge, technologies, techniques, products • What the ultimate impact(s) is • Needs met • Problems solved • Opportunitiesseized Priorities: Consider: Mission Vision Values Mandates Resources Local Dynamics Collaborators Competitors Intended Outcomes Situation Needs, Problems, and Opportunities Assumptions External Factors Evaluation Focus - Collect Data – Analyze and Interpret - Report Generic Research Logic Model Worksheet

22. CSREES Marine Shrimp Virus Research Logic Model Situation Inputs Activities Outputs Outcomes Short Medium Long • Legislation: Marine Shrimp Farming Program goal is to reduce American multi-billion dollar trade deficit by developing U.S. shrimp farming industry • Problem: New and virulent viral diseases threaten the US & world shrimp farming industry • Research needed to mitigate viral threats • Research needed on high-priority opportunities to develop a sustainable U.S. commercial shrimp farming industry. • US Marine Shrimp • Farming Program • (USMSP) funds • $4 million per year • $20 million over • 5 year-period • Genetic research for: • Identification of shrimp diseases • Disease prevention • Disease resistance • Marker- • assisted • Selection for genetic shrimp stock • Develop • diagnostic • field kits for viruses • Disseminate information • via website • Breeding strategies for SPF stocks and TSV resistance • Use biosecurity • and best • management • practices • Reduce • water usage • rates • shrimp • produced • Reduce • Levels of contaminants (TSS, NH3 • and CBDO) to • lbs of shrimps • produced • New lines • Created due to • research • Reduction of disease transfer • Produce and • Distribute • Selected • stocks • Improved • economic • opportunity • for producers • & communities Basic Research • Human Capital: • Faculty • Researchers • Industry • Labs Expanded Knowledge base re Marine Shrimp Viruses Applied research Developed biosecurity protocol methods & diagnostic tests • Production: • New pedigree lines for disease resistance • New products • Marketing • Developed Marketing • Strategies for shrimp Train producers in best management practices Extension • Dissemination: • Publications, citations, patents, • Networking; industry briefs Assumptions- Joint efforts are likely to find better solutions; without the federal lead, research in this areas will not move forward; basic research in genetics is needed and will pay off in applications for combating disease; marketing research needed to complete commercialization External Factors – Shrimp prices, global trade in shrimp broodstock, new shrimp virus outbreak, intensive vs. Extensive management/production facilities, environmental activists’ reactions, transportation issues

23. Management of Forest and Rangelands Logic Model Situation Inputs Activities Outputs Outcomes Short Medium Long-term • In the 21st Century, the nation’s forest and grasslands face four threats: • Fire and fuels • Invasive Species • Loss of open space • Unmanaged recreation • Rehabilitate and restore treatment priorities where risks are the greatest; prevent and contain the spread of invasive species; slow the loss of open space due to development; and manage the use of national forest for outdoor activities to increase the sustainability of our national forests and benefits to the American people • A well balanced ecosystem is an imperative backbone for the survival of nation’s natural resources and environmental base. • Federal dollars • State Appropriations • Determine Influence of fungal associations on fitness of sibling beetles • Determine if mycangial fungal associates change in nutrient quality of tissue • Identify genes for Dutch Elm disease resistance in American Elm. • Identify inhibitors that will reduce growth of branches and apical shoots. Extension Oregon State will assist ranchers in solving problems related to rangeland ownership and management Assist in revitalization in cooperation with utility companies and encourage the planting of trees around homes and buildings Train producers in use of best management practices Conduct workshops including homeowners, community leaders, fire professionals, insurance company representatives, real estate professionals and others Basic Research • Increase knowledge of rangelands for land owners and students • Invasive species • Change beliefs that pesticides must consist of invasive or abrasive agents • Identifiy genes for Dutch Elm disease resistance • Discover • gibberellin synthesis inhibitors • Paclobutrazol and flurprimidol will significantly reduce elongation growth • New community partnerships will be developed to increase fire awareness • Implement new ways to extend rangeland and environmental issues beyond a political framework • Invasive Species • Utilize new species to ward off pests • Changes in technologies for controlling pests and diseases of urban trees, development of “designer” trees that can adapt to a wide range of urban environment • Increase • overall individual and community fire awareness and action • Habitat iwill be restored at low cost while biodiversity will rebound • Recreation & wilderness • Lawsuits will be reduced which in turn will save money • A healthy, livable and sustainable urban forest ecosystem • Increased surveillance to guard against invasive species • Identify • promising technologies in fire protection • Increase economic opportunities for natural resources • Human Capital: • Faculty • Researchers • Industry • Labs Applied research Invasive Species Develop a novel pesticide which uses herbivore and preferential grazing habits of sheep and cattle to offset the damage of leafy spurge Develop semiochemical-based monitoring and protective techniques to enhance integrated control strategy for pine tip moth Calculate changes in regional biogeochemistry due to urban expansion during the past several decades Assumptions – Basic research is needed in identified areas to allow applied research to occur; extensive extension will be needed to alert targeted groups to new technologies and techniques External Factors – Bio terrorism, invasive species, environmental disasters/concerns, high energy consumption by competitors and global trade of wood products

24. Logic model benefits • 1. Improves management • Requires that everyone articulate and agree on achievable outcomes • Helps provide roadmap for implementation • Allows everyone to recognize specific data needed for performance assessment and proof of accomplishments

25. Logic model benefits (cont.) • 2. Facilitates evaluation • Facilitates the creation of evaluation designs • Facilitates the identification and selection of appropriate data elements • Facilitates appropriate data collection • Helps in the provision of measurable impacts

26. Logic model benefits (cont.) • 3. Facilitates accountability reporting and funding decisions • Establishes links between problems areas and strategic plan • Extremely useful in justifying additional funds • Takes fuzziness out of the program • Helps in demonstrating attributional links in a chain of activities

27. Use of Honeycomb Graphics • Key players; roles and coordination • Address questions of overlap and duplication across agencies/depts. • Show topic area (Portfolio and PAs) in context of world, U.S. partnersips, federal entities, USDA, and within CSREES

28. PA 215 PA 214 PA 213 PA 216 PA 212 PA 211 Accomplishments Plant Protection Portfolio Areas of Science: PA 215 PA 214 PA 213 Ecology PA 212 Detection/ Diagnosis Epidemiology PA 211 Management Biology Prevention Economics and Safety Insects, Mites and Other Arthropods Affecting Plants PA 211 Needs

29. Plant Protection Portfolio PA 215 PA 214 PA 213 PA 216 PA 212 PA 211 Ecology Detection and Diagnosis Epidemiology Biology Management Prevention Economics and Safety • Disease free seed and stock for some diseases • Breeding resistant plants • Novel types of resistance genes • Marker assisted selection Prevention • Ability to identify some unculturable organisms • Characterizing take-all suppressive soils • Ecology of vectors • Genomic sequencing • Molecular communication between plants and microbes • Programmed cell death Biology and Ecology • Pathogen’s genetic fingerprint used for rapid diagnosis • Pathogenicity sequences identified • Digital diagnosis / NPDN • Seed propagation certification programs • Quarantine programs • Traditional detection methods Accomplishments Detection & Diagnosis • Understanding dynamics of spread • Understanding mechanisms of spread • Forecasting of disease based on knowledge of environmental parameters Epidemiology • Biological control –e.g., A. radiobacter, Trichoderma, nematodes • Chemical control (pathogen and/or vector) • Cultural practices, IPM Management • Disease loss estimates • Trade embargoes – international and interstate • Safer pesticides, reduced use through IPM Economic Sustainability and Safety • Isolate resistance genes, create resistance genes • Interfere with mechanisms of signaling, pathogen’s virulence systems • Durable resistance (understanding) Prevention • Functional genomics, proteomics, bioinformatics • Non-host resistance • Publicly accessible databases for genome-enabled biology Biology and Ecology • Culture collection and characterization, and specific DNA probes for identification • Other pathogens genetic fingerprint for rapid diagnosis • Other pathogenicity sequences • Ability to detect individuals within a microbial population • Rapid / high-throughput methods of detection (user friendly/economical) • Genomic reclassification of microbial taxonomy Detection & Diagnosis PA 212 Pathogens and Nematodes Affecting Plants • Influence of global change on pathogen spread and disease establishment • Accurate determination of disease origin Epidemiology • BC agents – establishment, mechanisms, compatibility • Alternatives to methyl bromide • Post-harvest disease • Chemical induction of resistance • Chemical resistance management Management Needs • Better understanding of impacts of diseases • Development of sustainable production practices • Development of trace-back Economic Sustainability and Safety

30. Use of Research Evaluation as Leadership Tool • The PART and BPI have raised consciousness and focused attention on post-project management and use of evaluation results for planning

31. Summary • Use of Logic Models and Portfolio graphics provide clear and concise maps of work • Meaningful outcome indicators for OMB PART and BPI provided via scores from panel • Panelists provided useful recommendations • Managers have used PREP recommendations in planning, writing RFAs, budget, etc.

32. Summary (Cont.) • PREP process provided panelists with new insights in planning and accountability to spread around US • PREP improved panelists’ understanding and willingness to integrate their activities into the overall federal goals identified

33. Contact Information • Telephone: 202/ 720-5623 • Fax: 202/ 720-7714 • CSREES Web site: www.csrees.usda.gov • Email: coros@csrees.usda.gov